[yt-users] unpickle / load problems

Elizabeth Tasker tasker at astro1.sci.hokudai.ac.jp
Fri Mar 15 14:37:25 PDT 2013


Hi Matt,

Great -- that now works for me and I've approved the pull (but only tested it on new pickles).

For a testing script, is there a specific format? If so (and I should know about it!) could I be pointed to the appropriate thread / example for writing one? 

Elizabeth

On Mar 15, 2013, at 1:01 PM, Matthew Turk <matthewturk at gmail.com> wrote:

> Hi Elizabeth,
> 
> Okay, I have a fix.  I think.  Unfortunately, pickle files are not
> currently tested -- they were, once upon a time, but are not now.
> Would you be willing to help out and convert this script to a testing
> script?  I think it draws out some important things.
> 
> The code I came up with is not terribly pretty.  Unfortunately,
> keeping backwards compatibility for pickle files means making a few
> compromises in the niceness of the code.  So hopefully this will work,
> and not break a bunch of pre-existing pickles.  If it does (paging
> David Collins) then it's a bug and we cannot accept the PR.
> 
> Here's the pull request:
> 
> https://bitbucket.org/yt_analysis/yt/pull-request/463/attempting-to-create-a-backwards
> 
> Let me know if it works for you.  The changeset hash to pull:
> 
> hg pull -r 15d72caf0ea6 https://bitbucket.org/MatthewTurk/yt/
> 
> -Matt
> 
> On Fri, Mar 15, 2013 at 11:59 AM, Matthew Turk <matthewturk at gmail.com> wrote:
>> Hi Elizabeth,
>> 
>> Okay -- cool.  I think I will work around the h5py thing for now.  But
>> it is worrisome; I think save_object is currently broken as h5py
>> changed how it deals with VLEN strings between 2.0 and 2.1.
>> 
>> I've reproduced it by using Pickle by itself.  More as I get it.
>> 
>> -Matt
>> 
>> On Fri, Mar 15, 2013 at 11:25 AM, Elizabeth Tasker
>> <tasker at astro1.sci.hokudai.ac.jp> wrote:
>>> Hi Matt,
>>> 
>>> According to the directory I have in yt, an older version:
>>> 
>>> /Users/Elizabeth/yt/src/h5py-2.0.1/
>>> 
>>> Elizabeth
>>> 
>>> On Mar 15, 2013, at 10:04 AM, Matthew Turk <matthewturk at gmail.com> wrote:
>>> 
>>>> Hi Elizabeth,
>>>> 
>>>> On Wed, Mar 13, 2013 at 4:45 PM, Elizabeth Tasker
>>>> <tasker at astro1.sci.hokudai.ac.jp> wrote:
>>>>> Hi Matt,
>>>>> 
>>>>> So ironically, it took me a long time to find the right combination to make this break in the test script! The fatal combination seems to be an extract region from a boolean object.
>>>>> 
>>>>> This is the script I used:
>>>>> 
>>>>> http://paste.yt-project.org/show/3228/
>>>>> 
>>>>> It should work with any data set.
>>>>> 
>>>>> It creates 6 saved objects. Objects 1-4 work fine but 5 and 6 have the attribute error.
>>>> 
>>>> I'm getting h5py errors when I even run the script.  Which version of
>>>> h5py are you on?
>>>> 
>>>>>>> h5py.version.version
>>>> '2.1.0'
>>>> 
>>>> 
>>>>> 
>>>>> Elizabeth
>>>>> 
>>>>> 
>>>>> On Mar 13, 2013, at 3:08 PM, Matthew Turk <matthewturk at gmail.com> wrote:
>>>>> 
>>>>>> Hi Elizabeth,
>>>>>> 
>>>>>> Okay, this seems like it's bumped up against a problem with the way yt
>>>>>> handles pickles.  At some point in the reconstruction, data objects
>>>>>> aren't getting separated from their parameter files.
>>>>>> 
>>>>>> Any chance you could make something that reproduces this on one of the
>>>>>> workshop datasets?
>>>>>> 
>>>>>> http://yt-project.org/data/
>>>>>> 
>>>>>> -Matt
>>>>>> 
>>>>>> On Wed, Mar 13, 2013 at 3:03 PM, Elizabeth Tasker
>>>>>> <tasker at astro1.sci.hokudai.ac.jp> wrote:
>>>>>>> Hi,
>>>>>>> 
>>>>>>> I'm struggling to unpickle / load_object a previously pickled / save_object.
>>>>>>> 
>>>>>>> I've done:
>>>>>>> 
>>>>>>> pf.h.save_object(allclouds, "clouds_pyfindclouds2_yt")
>>>>>>> 
>>>>>>> but when I do:
>>>>>>> 
>>>>>>> clouds = pf.h.load_object("clouds_pyfindclouds2_yt")
>>>>>>> 
>>>>>>> I get:
>>>>>>> 
>>>>>>> AttributeError                            Traceback (most recent call last)
>>>>>>> /Users/Elizabeth/yt/src/yt-hg/scripts/iyt in <module>()
>>>>>>> ----> 1 execfile("cloud_surfacedensity_b.py")
>>>>>>> 
>>>>>>> /Users/Elizabeth/yt/src/yt-hg/scripts/iyt in <module>()
>>>>>>>   13 #file.close()
>>>>>>>   14
>>>>>>> ---> 15 clouds = pf.h.load_object("clouds_pyfindclouds2_yt")
>>>>>>> 
>>>>>>> /Users/Elizabeth/yt/src/yt-hg/yt/data_objects/hierarchy.pyc in load_object(self, name)
>>>>>>>  328         if obj is None:
>>>>>>>  329             return
>>>>>>> --> 330         obj = cPickle.loads(obj.value)
>>>>>>>  331         if iterable(obj) and len(obj) == 2:
>>>>>>>  332             obj = obj[1] # Just the object, not the pf
>>>>>>> 
>>>>>>> /Users/Elizabeth/yt/src/yt-hg/yt/data_objects/data_containers.pyc in _reconstruct_object(*args, **kwargs)
>>>>>>> 4623     pf = pfs.get_pf_hash(pfid)
>>>>>>> 4624     cls = getattr(pf.h, dtype)
>>>>>>> -> 4625     obj = cls(*new_args)
>>>>>>> 4626     obj.field_parameters.update(field_parameters)
>>>>>>> 4627     return pf, obj
>>>>>>> 
>>>>>>> /Users/Elizabeth/yt/src/yt-hg/yt/data_objects/data_containers.pyc in __init__(self, regions, fields, pf, **kwargs)
>>>>>>> 4046         self._all_overlap = []
>>>>>>> 4047         self._cut_masks = {}
>>>>>>> -> 4048         self._get_all_regions()
>>>>>>> 4049         self._make_overlaps()
>>>>>>> 4050         self._get_list_of_grids()
>>>>>>> 
>>>>>>> /Users/Elizabeth/yt/src/yt-hg/yt/data_objects/data_containers.pyc in _get_all_regions(self)
>>>>>>> 4057             self._all_regions.append(item)
>>>>>>> 4058             # So cut_masks don't get messed up.
>>>>>>> -> 4059             item._boolean_touched = True
>>>>>>> 4060         self._all_regions = np.unique(self._all_regions)
>>>>>>> 4061
>>>>>>> 
>>>>>>> AttributeError: ("'tuple' object has no attribute '_boolean_touched'", <function _reconstruct_object at 0x101dcd578>, ('73850deee907adf5217f43156c5057cd', 'boolean', [(GR_Enzo2_128amr5_zeus_turb_psupp_rr_0240, AMRCylinder (GR_Enzo2_128amr5_zeus_turb_psupp_rr_0240): center=[ 16.  16.  16.], _norm_vec=[ 0.  0.  1.], _radius=7.5, _height=0.100000208264), 'NOT', (GR_Enzo2_128amr5_zeus_turb_psupp_rr_0240, AMRCylinder (GR_Enzo2_128amr5_zeus_turb_psupp_rr_0240): center=[ 16.  16.  16.], _norm_vec=[ 0.  0.  1.], _radius=4.5, _height=0.100000208264)], {'bulk_velocity': array([ 0.,  0.,  0.]), 'center': array([ 0.5,  0.5,  0.5]), 'normal': array([ 0.,  0.,  1.])}))
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> This is all on the same computer.
>>>>>>> 
>>>>>>> Elizabeth
>>>>>>> _______________________________________________
>>>>>>> yt-users mailing list
>>>>>>> yt-users at lists.spacepope.org
>>>>>>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>>>>>> _______________________________________________
>>>>>> yt-users mailing list
>>>>>> yt-users at lists.spacepope.org
>>>>>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>>>>> 
>>>>> _______________________________________________
>>>>> yt-users mailing list
>>>>> yt-users at lists.spacepope.org
>>>>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>>>> _______________________________________________
>>>> yt-users mailing list
>>>> yt-users at lists.spacepope.org
>>>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>>> 
>>> _______________________________________________
>>> yt-users mailing list
>>> yt-users at lists.spacepope.org
>>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
> _______________________________________________
> yt-users mailing list
> yt-users at lists.spacepope.org
> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org




More information about the yt-users mailing list