[Pythonmac-SIG] Performance on the Mac - A suggested fix

[Gary Bringhurst]

Recently, Jack Jansen <Jack.Jansen@oratrix.com> said:
> - Your slowdown comes from PyMac_DoYield calls not from the mainloop
> but from elsewhere. The only place I can think of is the GUSI I/O idle
> loop: the I/O wait-for-completion routine eventually ends up in
> PyMac_DoYield. If that is the case you've sped up some cases,
> probably, but seriously slowed down others (talking to a local
> webserver comes to mind).

Very good questions Jack.  I think you've nailed it with that last one.

For the record I'm using the PythonInterpreter for my purposes, not the IDE.
I don't know if that matters.  When I randomly break into the interpreter
while it is executing a tight loop reading 256K bytes from one file and
writing them to another I see a call chain that includes calls to
GUSIMacFileSocket which in turn calls GUSIContext::Yield,
GUSIProcess::Yield, PyMac_GUSISpin and finally PyMac_DoYield.

MacPython registers a GUSISpinHook of PyMac_GUSISpin, and every call to
Yield in GUSIMacFile passes the parameter kGUSIBlock which results in a true
wait argument to PyMac_GUSISpin, so we get a maxsleep parameter of 6 in
PyMac_DoYield for every Yield during file I/O.  Because of the other default
settings in the interpreter the net result is that WaitNextEvent is called
everytime the GUSISpinHook is called.

You could easily argue that GUSI should not be using kGUSIBlock in those
calls but rather kGUSIYield, which seems a better semantic match.  But even
that would help in this case, since we don't currently do anything but
change the maxsleep parameter based on this, and any call to WaitNextEvent
in the current MacOS is a license for the OS to abuse you.

On the assumption that any yields via the GUSISpinHook are *nonessential* we
could ignore the wait parameter and always impose a minimum time between
calls to PyMac_DoYield.  This seems to help performance quite a bit.

static void
PyMac_GUSISpin(bool wait)
{
    static Boolean    inForeground = true;
    int        maxsleep = 6;    /* 6 ticks is "normal" sleeptime */

    if (PyMac_ConsoleIsDead) return;

    if ( !wait )
        maxsleep = 0;

#if qStrataHacks
    static unsigned long lastTimeInHere;
    unsigned long thisTimeInHere = LMGetTicks();
    const unsigned long maxCheckFreq = 30;
    if (/* !wait && */ thisTimeInHere - lastTimeInHere < maxCheckFreq)
    {
        return;
    }
    lastTimeInHere = thisTimeInHere;
#endif

    PyMac_DoYield(maxsleep, 0); /* XXXX or is it safe to call python here?
*/
}

Now for the numbers.  Here are the times to execute our Python scripts using
the various builds of the Python interpreter:

Python 2.1 Mac G4/450:          ~19 minutes
Python 2.1 Mac G4/450 w/hack:   ~2 minutes
Python 2.1 Win P3/600:          ~1 minute

I keep being retaught the lesson that simple assumptions and changes are
very dangerous.  It seems that I'd learn.

Hoping this helps,

Gary Bringhurst
garyb@strata.com