fix: cache flush_only_netcdf_file class to fix pickling (GH#11323)#11414
Closed
C1-BA-B1-F3 wants to merge 1 commit into
Closed
fix: cache flush_only_netcdf_file class to fix pickling (GH#11323)#11414C1-BA-B1-F3 wants to merge 1 commit into
C1-BA-B1-F3 wants to merge 1 commit into
Conversation
The _PickleWorkaround pattern created a new flush_only_netcdf_file class on each call to _open_scipy_netcdf(). This broke pickling because subsequent calls overwrote _PickleWorkaround.flush_only_netcdf_file with a new class object, making previously-created instances unpicklable. Fix: Cache the class in a module-level variable _flush_only_class so it's only created once. This preserves the lazy scipy import while ensuring class identity is stable across multiple opens. Closes pydata#11323
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this fixes
Fixes #11323 - Opening multiple scipy-backed datasets from file-like objects breaks pickling of previously-opened datasets.
Root cause
The
_PickleWorkaroundpattern inscipy_.pycreated a newflush_only_netcdf_fileclass on each call to_open_scipy_netcdf(). When a second dataset was opened:C2was created inside_open_scipy_netcdf()_PickleWorkaround.add_cls()storedC2, overwriting the previousC1ds1(created withC1) failed because pickle foundC2via qualname lookup, notC1Fix
Cache the
flush_only_netcdf_fileclass in a module-level variable so it's created once and reused. This preserves the lazy scipy import while ensuring class identity is stable.Testing
Added regression test
test_scipy_pickle_after_multiple_opensthat reproduces the exact scenario from the issue. All existing scipy backend tests pass.Related
_PickleWorkaroundpattern)