★ wanayoo — archive 1999 https://github.com/Technologicat/unpythonic/issues/55Nouvelle recherche | Portail wanayoo
Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Investigate pickling of singletons #55

Closed
Technologicat opened this issue Dec 5, 2019 · 8 comments
Closed

Investigate pickling of singletons #55

Technologicat opened this issue Dec 5, 2019 · 8 comments
Assignees
Labels
bug
Milestone

Comments

@Technologicat
Copy link
Owner

@Technologicat Technologicat commented Dec 5, 2019 •

Currently (0.14.2-pre) the empty frozendict is freshly created for each Python session.

This can be an issue if fd = frozendict(), we then pickle fd, and later unpickle it in a new Python session. Then, in this new session, fd is frozendict() is False, because the singleton empty frozendict has been freshly created.

This needs some thinking. It's a global, not an attribute, so it's not easily updated in a custom __setstate__ like our cons does for the nil singleton.

Maybe __getnewargs_ex__?


Hmm, wait, I may be interpreting the docs incorrectly. Looking at Pickling Class Instances, it may actually happen that only one frozendict instance (which is supposed to be the empty one) is ever created upon unpickling. If so, this is a serious bug that needs to be fixed ASAP. (EDIT: yes, that's exactly what happened. Fixed.)

We should also check how nil behaves. We can probably make the old and new id(nil) agree after unpickling, without the __setstate__ hack currently in cons, if we use the low-level approach to create the singleton: make Nil.__new__ create one global Nil instance and always return it. (Assuming that __new__ is called at unpickling time even if no __getnewargs__ is present. Need to test this.) (EDIT: Yes, seems to be so. Done. The hack is gone, and it works.)

Keep in mind only the class name and the instance data are actually stored in the pickle file. When unpickling, __init__ is skipped, but __new__ runs (by default with no args other than the mandatory cls).

EDIT: At unpickle time, after __new__, then __setstate__ runs if defined; else the pickled state must be a dictionary, and it is dumped to __dict__.


It seems this isn't documented explicitly, but from what I can gather, it seems that __getnewargs__ (and its _ex variant) will be called at pickling time. The return value will be stored, and later used when __new__ is called to create the uninitialized object (for populating the state) at unpickling time. See Flyweighting in Python by Simon Engledew (2009). EDIT: Yes, that seems to be exactly how it works.

Usually __getnewargs__ isn't needed nowadays, but in cases where the behavior of a custom __new__ depends on the arguments passed to the constructor call, it is needed to support pickling properly.


Edit 22 February 2020: Added some notes above based on investigation results.

@Technologicat Technologicat added the bug label Dec 5, 2019
@Technologicat Technologicat modified the milestones: ∞, 0.14.2 Dec 5, 2019
@Technologicat Technologicat changed the title Support pickling of empty frozendict Investigate pickling of singletons Dec 5, 2019
@Technologicat Technologicat self-assigned this Dec 5, 2019
@Technologicat
Copy link
Owner Author

@Technologicat Technologicat commented Feb 21, 2020 •

Yep, our handling of frozendict is broken.

Let's add something like this to the unit tests for unpythonic.collections:

from unpythonic import frozendict
import pickle
fd1 = frozendict({'a': 1, 'b': 2})
fd2 = frozendict({'c': 3, 'd': 4})
fd3 = frozendict()
s = pickle.dumps([fd1, fd2, fd3])
o = pickle.loads(s)
print(o)  # [frozendict({}), frozendict({}), frozendict({})]   # as of 0.14.2-pre

EDIT: 4e32777 now includes the new test case. The important part was to have both the empty frozendict as well as some non-empty frozendict instances in the data.

Technologicat added a commit that referenced this issue Feb 21, 2020
@Technologicat
Copy link
Owner Author

@Technologicat Technologicat commented Feb 21, 2020

Hacky handling of nil (linked list terminator) singleton exorcised in 0bc0dc4, for great justice.

@Technologicat
Copy link
Owner Author

@Technologicat Technologicat commented Feb 21, 2020 •

The empty frozendict and nil are the only singletons in unpythonic that may end up in a pickle file in typical use. As of 0bc0dc4, the code to handle those has been fixed.

The other three, namely dyn, getvalue and runpipe most likely won't.

But perhaps think a bit more before closing this. It's not that hard to fix those too.

@Technologicat
Copy link
Owner Author

@Technologicat Technologicat commented Mar 1, 2020

Fixed getvalue and runpipe in 761be89.

@Technologicat
Copy link
Owner Author

@Technologicat Technologicat commented Mar 1, 2020

Fixed dyn in cc1623d.

Note, pickling and unpickling dyn makes no sense, because the whole point is that it tracks dynamic state - in a manner of speaking, dyn has one foot on the call stack. But still, it shouldn't hurt to make it a proper singleton, using the pattern that actually ensures singleton semantics in Python.

@Technologicat
Copy link
Owner Author

@Technologicat Technologicat commented Mar 1, 2020

Upon closer investigation, we can't prevent pickling in dyn, because MacroPy's hygienic quasiquotes (hq[]) build on pickle.

If dyn fails to pickle, some macros in unpythonic.syntax (notably autoref and lazify) crash, because they need both dyn and hq[].

Fortunately, no state is saved in the dyn singleton instance itself, so it doesn't matter that the default __setstate__ clobbers the __dict__ of the singleton instance at unpickle time.

5f87e18 adds a comment documenting this.

I think this is now finally fully done. Closing.

@Technologicat
Copy link
Owner Author

@Technologicat Technologicat commented Mar 4, 2020 •

For documentation: as of v0.14.2, unpythonic has four distinct singleton objects:

  • nil: the linked list terminator marker,
  • getvalue, runpipe: the pipe terminator marker. These are two names for the same object, to make the name fit the eager/lazy semantics as appropriate.
  • frozendict(): the empty frozendict,
  • dyn: the controller of the dynamic assignment machinery.

These use cases split to three categories:

  1. A process-wide unique marker value, possibly with a nice __repr__, its own docstring, and arbitrary extra functionality (e.g. nil is iterable, yielding the empty sequence).
  2. An empty immutable collection.
  3. A service that must be instantiated at most once per process.

In many cases, for use case 1, a local marker = object() is sufficient and the simpler solution. However, sometimes it's important that only one instance of the marker (e.g. the linked list terminator nil) exists in the same process. This single-instance restriction must hold even if we unpickle something that has a copy of the marker - the saved and live copies should then be made to point to the same instance. This allows the marker to behave like the built-in constants True/False/None do.

If we don't need to support a docstring and arbitrary extra features, then a symbol, in the Lisp sense, could be another possibility. Python has sys.intern; maybe a symbol abstraction could be built on that.

Specifically, nil and getvalue/runpipe are markers, which have been made distinct from None. Although null is the billion dollar mistake, multiple nulls are "a programmer's worst nightmare", and we don't even provide a Maybe monad, nevertheless nil was chosen to be distinct, to allow the pythonic None as a data value also in the cdr position. (Maybe this decision should be revisited.)

getvalue/runpipe wouldn't have to be distinct from None, since None is not callable, but piping into getvalue or runpipe is more descriptive than piping into None - or even giving None an alternative name, which would confusingly look exactly like the pythonic idiom of setting a placeholder value.

The empty frozendict() is an empty immutable collection, use case 2. Since no items can be added to it after constructing the instance, there's no point in creating multiple copies of it. This follows the semantics of tuple and frozenset.

The last singleton, dyn, is an example of use case 3. Only one controller for dynamic assignment is needed, so whenever someone wants to refer to it, they should get the same instance.

However, the implementation is slightly misleading in that dyn actually has no instance data - all the dynamic variables live in the thread-local stacks, which are stored at the module top level. Having no state is what makes dyn safe for pickling. Although doing so makes no sense, we must allow picklling dyn for technical reasons, so that our macro code can use both dyn and MacroPy's hq[].

See also #22.

@Technologicat
Copy link
Owner Author

@Technologicat Technologicat commented Mar 4, 2020

The empty frozendict() needs some special-casing, so it can't use Singleton as-is. Since the code is just a few lines, leaving it as-is for now.

(The current frozendict() code is also nice in that it doesn't need a metaclass to guard its constructor against multiple invocations, so it's possible to multiple-inherit from both frozendict and some other class that has a custom metaclass, in case user code needs this ability.)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
1 participant
You can’t perform that action at this time.