[yt-users] unpickle / load problems
Matthew Turk
matthewturk at gmail.com
Fri Mar 15 14:41:57 PDT 2013
On Fri, Mar 15, 2013 at 5:37 PM, Elizabeth Tasker
<tasker at astro1.sci.hokudai.ac.jp> wrote:
> Hi Matt,
>
> Great -- that now works for me and I've approved the pull (but only tested it on new pickles).
Awesome!
>
> For a testing script, is there a specific format? If so (and I should know about it!) could I be pointed to the appropriate thread / example for writing one?
Yup, here's an example of the type of thing, as well as the docs:
(don't let the length of the file daunt you, the pickle tests can be
much shorter)
https://bitbucket.org/yt_analysis/yt/src/ed10fc84bafcefebfdfd813ca59caa245f8a5ebd/yt/visualization/tests/test_plotwindow.py?at=yt
http://yt-project.org/docs/2.5/advanced/testing.html
I'm happy to help out, too. Thanks!
-Matt
>
> Elizabeth
>
> On Mar 15, 2013, at 1:01 PM, Matthew Turk <matthewturk at gmail.com> wrote:
>
>> Hi Elizabeth,
>>
>> Okay, I have a fix. I think. Unfortunately, pickle files are not
>> currently tested -- they were, once upon a time, but are not now.
>> Would you be willing to help out and convert this script to a testing
>> script? I think it draws out some important things.
>>
>> The code I came up with is not terribly pretty. Unfortunately,
>> keeping backwards compatibility for pickle files means making a few
>> compromises in the niceness of the code. So hopefully this will work,
>> and not break a bunch of pre-existing pickles. If it does (paging
>> David Collins) then it's a bug and we cannot accept the PR.
>>
>> Here's the pull request:
>>
>> https://bitbucket.org/yt_analysis/yt/pull-request/463/attempting-to-create-a-backwards
>>
>> Let me know if it works for you. The changeset hash to pull:
>>
>> hg pull -r 15d72caf0ea6 https://bitbucket.org/MatthewTurk/yt/
>>
>> -Matt
>>
>> On Fri, Mar 15, 2013 at 11:59 AM, Matthew Turk <matthewturk at gmail.com> wrote:
>>> Hi Elizabeth,
>>>
>>> Okay -- cool. I think I will work around the h5py thing for now. But
>>> it is worrisome; I think save_object is currently broken as h5py
>>> changed how it deals with VLEN strings between 2.0 and 2.1.
>>>
>>> I've reproduced it by using Pickle by itself. More as I get it.
>>>
>>> -Matt
>>>
>>> On Fri, Mar 15, 2013 at 11:25 AM, Elizabeth Tasker
>>> <tasker at astro1.sci.hokudai.ac.jp> wrote:
>>>> Hi Matt,
>>>>
>>>> According to the directory I have in yt, an older version:
>>>>
>>>> /Users/Elizabeth/yt/src/h5py-2.0.1/
>>>>
>>>> Elizabeth
>>>>
>>>> On Mar 15, 2013, at 10:04 AM, Matthew Turk <matthewturk at gmail.com> wrote:
>>>>
>>>>> Hi Elizabeth,
>>>>>
>>>>> On Wed, Mar 13, 2013 at 4:45 PM, Elizabeth Tasker
>>>>> <tasker at astro1.sci.hokudai.ac.jp> wrote:
>>>>>> Hi Matt,
>>>>>>
>>>>>> So ironically, it took me a long time to find the right combination to make this break in the test script! The fatal combination seems to be an extract region from a boolean object.
>>>>>>
>>>>>> This is the script I used:
>>>>>>
>>>>>> http://paste.yt-project.org/show/3228/
>>>>>>
>>>>>> It should work with any data set.
>>>>>>
>>>>>> It creates 6 saved objects. Objects 1-4 work fine but 5 and 6 have the attribute error.
>>>>>
>>>>> I'm getting h5py errors when I even run the script. Which version of
>>>>> h5py are you on?
>>>>>
>>>>>>>> h5py.version.version
>>>>> '2.1.0'
>>>>>
>>>>>
>>>>>>
>>>>>> Elizabeth
>>>>>>
>>>>>>
>>>>>> On Mar 13, 2013, at 3:08 PM, Matthew Turk <matthewturk at gmail.com> wrote:
>>>>>>
>>>>>>> Hi Elizabeth,
>>>>>>>
>>>>>>> Okay, this seems like it's bumped up against a problem with the way yt
>>>>>>> handles pickles. At some point in the reconstruction, data objects
>>>>>>> aren't getting separated from their parameter files.
>>>>>>>
>>>>>>> Any chance you could make something that reproduces this on one of the
>>>>>>> workshop datasets?
>>>>>>>
>>>>>>> http://yt-project.org/data/
>>>>>>>
>>>>>>> -Matt
>>>>>>>
>>>>>>> On Wed, Mar 13, 2013 at 3:03 PM, Elizabeth Tasker
>>>>>>> <tasker at astro1.sci.hokudai.ac.jp> wrote:
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> I'm struggling to unpickle / load_object a previously pickled / save_object.
>>>>>>>>
>>>>>>>> I've done:
>>>>>>>>
>>>>>>>> pf.h.save_object(allclouds, "clouds_pyfindclouds2_yt")
>>>>>>>>
>>>>>>>> but when I do:
>>>>>>>>
>>>>>>>> clouds = pf.h.load_object("clouds_pyfindclouds2_yt")
>>>>>>>>
>>>>>>>> I get:
>>>>>>>>
>>>>>>>> AttributeError Traceback (most recent call last)
>>>>>>>> /Users/Elizabeth/yt/src/yt-hg/scripts/iyt in <module>()
>>>>>>>> ----> 1 execfile("cloud_surfacedensity_b.py")
>>>>>>>>
>>>>>>>> /Users/Elizabeth/yt/src/yt-hg/scripts/iyt in <module>()
>>>>>>>> 13 #file.close()
>>>>>>>> 14
>>>>>>>> ---> 15 clouds = pf.h.load_object("clouds_pyfindclouds2_yt")
>>>>>>>>
>>>>>>>> /Users/Elizabeth/yt/src/yt-hg/yt/data_objects/hierarchy.pyc in load_object(self, name)
>>>>>>>> 328 if obj is None:
>>>>>>>> 329 return
>>>>>>>> --> 330 obj = cPickle.loads(obj.value)
>>>>>>>> 331 if iterable(obj) and len(obj) == 2:
>>>>>>>> 332 obj = obj[1] # Just the object, not the pf
>>>>>>>>
>>>>>>>> /Users/Elizabeth/yt/src/yt-hg/yt/data_objects/data_containers.pyc in _reconstruct_object(*args, **kwargs)
>>>>>>>> 4623 pf = pfs.get_pf_hash(pfid)
>>>>>>>> 4624 cls = getattr(pf.h, dtype)
>>>>>>>> -> 4625 obj = cls(*new_args)
>>>>>>>> 4626 obj.field_parameters.update(field_parameters)
>>>>>>>> 4627 return pf, obj
>>>>>>>>
>>>>>>>> /Users/Elizabeth/yt/src/yt-hg/yt/data_objects/data_containers.pyc in __init__(self, regions, fields, pf, **kwargs)
>>>>>>>> 4046 self._all_overlap = []
>>>>>>>> 4047 self._cut_masks = {}
>>>>>>>> -> 4048 self._get_all_regions()
>>>>>>>> 4049 self._make_overlaps()
>>>>>>>> 4050 self._get_list_of_grids()
>>>>>>>>
>>>>>>>> /Users/Elizabeth/yt/src/yt-hg/yt/data_objects/data_containers.pyc in _get_all_regions(self)
>>>>>>>> 4057 self._all_regions.append(item)
>>>>>>>> 4058 # So cut_masks don't get messed up.
>>>>>>>> -> 4059 item._boolean_touched = True
>>>>>>>> 4060 self._all_regions = np.unique(self._all_regions)
>>>>>>>> 4061
>>>>>>>>
>>>>>>>> AttributeError: ("'tuple' object has no attribute '_boolean_touched'", <function _reconstruct_object at 0x101dcd578>, ('73850deee907adf5217f43156c5057cd', 'boolean', [(GR_Enzo2_128amr5_zeus_turb_psupp_rr_0240, AMRCylinder (GR_Enzo2_128amr5_zeus_turb_psupp_rr_0240): center=[ 16. 16. 16.], _norm_vec=[ 0. 0. 1.], _radius=7.5, _height=0.100000208264), 'NOT', (GR_Enzo2_128amr5_zeus_turb_psupp_rr_0240, AMRCylinder (GR_Enzo2_128amr5_zeus_turb_psupp_rr_0240): center=[ 16. 16. 16.], _norm_vec=[ 0. 0. 1.], _radius=4.5, _height=0.100000208264)], {'bulk_velocity': array([ 0., 0., 0.]), 'center': array([ 0.5, 0.5, 0.5]), 'normal': array([ 0., 0., 1.])}))
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> This is all on the same computer.
>>>>>>>>
>>>>>>>> Elizabeth
>>>>>>>> _______________________________________________
>>>>>>>> yt-users mailing list
>>>>>>>> yt-users at lists.spacepope.org
>>>>>>>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>>>>>>> _______________________________________________
>>>>>>> yt-users mailing list
>>>>>>> yt-users at lists.spacepope.org
>>>>>>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>>>>>>
>>>>>> _______________________________________________
>>>>>> yt-users mailing list
>>>>>> yt-users at lists.spacepope.org
>>>>>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>>>>> _______________________________________________
>>>>> yt-users mailing list
>>>>> yt-users at lists.spacepope.org
>>>>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>>>>
>>>> _______________________________________________
>>>> yt-users mailing list
>>>> yt-users at lists.spacepope.org
>>>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>> _______________________________________________
>> yt-users mailing list
>> yt-users at lists.spacepope.org
>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>
> _______________________________________________
> yt-users mailing list
> yt-users at lists.spacepope.org
> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
More information about the yt-users
mailing list