[Pythonmac-SIG] Extensions for static versus framework MachoPython

[Jack Jansen]

Folks,
I need a bit of extra brainpower for a solution to the following 
problem.

MachoPython can be built in 2 variants: as a static binary (like 
it usually is on Unix systems) or with the whole interpreter in 
a dynamic framework with the python executable being a tiny main 
program linked to that.

Now, there's a potential problem with dynamically loadable 
plugin modules. If you build a plugin module for one variant and 
then try to use it in the other variant this is potentially 
unsafe. And I don't know how to fix it. Actually, I don't even 
know that it is unsafe.

If you build an extension module for a static Python the module 
will be linked with "-bundle_loader python.exe" and then 
everything is fine: all Python symbols it needs it will try to 
satisfy from the program that loads it, and this will fail if 
you load it into a framework python (because the symbols are now 
defined in the framework, not in the  main program, which is 
where the extension module looks for them).

If you build an extension module for a framework Python it will 
not use the -bundle_loader option, but in stead link with 
"-framework Python". If you import this module into a framework 
Python it'll notice that the Python.framework is incore already, 
link against that and everything is fine.

If you try to load this module into a static Python, however, it 
will simply pull in a fresh copy of Python.framework. Even this 
shouldn't be a problem, because once the module tries to 
initialize itself it'll call PyModule_Initialize4(), which 
should notice that the module lives in a fresh, uninitialized 
interpreter and it should complain bitterly.

However: it doesn't complain. Which means that it must somehow 
call PyModule_Initialize4() from the already loaded static 
python (or, at least share its datastructures). But it *does* 
load Python.framework (checked by turning on DYLD debugging). 
This makes me very uneasy, is it may be using some stuff from 
the static Python and some (uninitialized) stuff from the 
framework it pulled in. That would definitely spell disaster, 
especially if the two Pythons have been compiled with different 
options (GC or debugging, for instance).

I'm not sure why it doesn't complain, it may have something to 
do with Python.framework being linked with -flat_namespace, 
which it needs because it needs access to _environ (which 
frameworks or dynamic libraries don't get, sigh, only main 
programs).

I tried specifying Python.framework as the -bundle_loader, but 
that doesn't work.

I'm out of good ideas now.

And it may be that I'm seeing a problem where there really isn't 
any: if 100% of the symbols are satisfied from the static Python 
then everything should be fine. Major differences between 
versions will be caught by the PY_API_VERSION mechanism (which 
is the general mechanism that stops you from loading an 
extension module into a Python that is too old or too new), so 
it's really only "little differences".

Should I stop worrying and just think of this as an advantage 
("If you build your extension module with a framework Python you 
can also use it with a static Python")?
--
- Jack Jansen        <Jack.Jansen@oratrix.com>        
http://www.cwi.nl/~jack -
- If I can't dance I don't want to be part of your revolution -- 
Emma Goldman -