[yt-users] pickle

Matthew Turk matthewturk at gmail.com
Thu Dec 1 07:31:35 PST 2011


Hi Elizabeth,

The hashes don't match, so it thinks you're loading a pickle for a
file it doesn't have in memory.  So, two possibilities:

1) The pickles were generated from a different static output than the
one you are pf = load()ing earlier.  If you load() the one that
generated them, this will work.

2) The hash for your parameter file changed.  The hash is generated by
taking the md5sum of the following three items:

basename
current_time (for enzo this is InitialTime in the parameter file)
unique_identifier

The third one is the important one that you should check, and the one
that has the most possibility of variation.  I assume you're running
Enzo.  If your simulation has the parameter CurrentTimeIdentifier
(which was added to the LCA enzo 1.5, and has persisted since then)
this is used.  If it does not, then the modification time of the
parameter file is used.  This can be changed if you edit or copy
(without using -p) the file.

-Matt

On Thu, Dec 1, 2011 at 10:03 AM, Elizabeth Tasker
<tasker at astro1.sci.hokudai.ac.jp> wrote:
> Tis:
>
> In [1]: pf._hash()
> Out[1]: '283b7c4d88671dbff7acf083098da6ae'
>
> E.
>
>
> On 2011-12-01, at 11:57 PM, Matthew Turk wrote:
>
>> Hi Elizabeth,
>>
>> What is the output of pf._hash() ?
>>
>> -Matt
>>
>> On Thu, Dec 1, 2011 at 9:32 AM, Elizabeth Tasker
>> <tasker at astro1.sci.hokudai.ac.jp> wrote:
>>> Sorry, still no luck!
>>>
>>> The current version of the code is:
>>>
>>> ---
>>> c1a446be3a7a (yt) tip
>>> ---
>>>
>>>
>>> unpickle core data
>>> ---------------------------------------------------------------------------
>>> KeyError                                  Traceback (most recent call last)
>>>
>>> /home/taskere/yt-2/src/yt-hg/scripts/iyt in <module>()
>>>    199
>>>    200     file = open('cores.pickle','rb')
>>> --> 201     allcloudcores = cPickle.load(file)
>>>    202     file.close()
>>>    203
>>>
>>> /home/taskere/yt-2/src/yt-hg/yt/data_objects/data_containers.pyc in _reconstruct_object(*args, **kwargs)
>>>   3640         else: new_args.append(arg)
>>>   3641     pfs = ParameterFileStore()
>>> -> 3642     pf = pfs.get_pf_hash(pfid)
>>>   3643     cls = getattr(pf.h, dtype)
>>>   3644     obj = cls(*new_args)
>>>
>>> /home/taskere/yt-2/src/yt-hg/yt/utilities/parameter_file_storage.pyc in get_pf_hash(self, hash)
>>>    109     def get_pf_hash(self, hash):
>>>    110         """ This returns a parameter file based on a hash. """
>>> --> 111         return self._convert_pf(self._records[hash])
>>>    112
>>>    113     def get_pf_ctid(self, ctid):
>>>
>>> KeyError: (('283b7c4d88671dbff7acf083098da6ae',), <function _reconstruct_object at 0xa8cd848>, ('283b7c4d88671dbff7acf083098da6ae', 'region', array([ 16.,  16.,  16.]), array([ 0.,  0.,  0.]), array([ 32.,  32.,  32.]), {'disk_center': array([16, 16, 16]), 'center': array([ 16.,  16.,  16.]), 'bulk_velocity': array([ 0.,  0.,  0.]), 'disk_vector': array([0, 0, 1]), 'disk_radius': array([  0.1   ,   0.1238,   0.1476,   0.1714,   0.1952,   0.219 ,
>>>         0.2428,   0.2666,   0.2904,   0.3142,   0.33
>>>
>>>
>>>
>>> Unless I need to re-pickle first?
>>>
>>> Elizabeth
>>>
>>>
>>>
>>>
>>> On 2011-12-01, at 8:20 PM, Matthew Turk wrote:
>>>
>>>> Thanks, Elizabeth.  The move to in-memory brought up a funny bug where
>>>> the initialization process was reseting the list of known parameter
>>>> hashes.  This is fixed in c1a446be3a7a.
>>>>
>>>> Also, save_object won't work except on yt objects; the result of
>>>> extract_connected_sets is a tuple of values and python objects.
>>>> Pickle will, tho.
>>>>
>>>> On Thu, Dec 1, 2011 at 4:33 AM, Elizabeth Tasker
>>>> <tasker at astro1.sci.hokudai.ac.jp> wrote:
>>>>> I also tried switching out pickle for the yt method of storing
>>>>> objects. Unfortunately, it doesn't seem to save the object I want. I
>>>>> tried:
>>>>>
>>>>> contours = dd.extract_connected_sets("NegEscapeVelocity", 1, 30.0,
>>>>> maxv, log_space=False)
>>>>> contours.save_object("cloudcores", "storedcores.cpkl")
>>>>>
>>>>> In [1]: contours.save_object("cloudcores", "storedcores.cpkl")
>>>>> ---------------------------------------------------------------------------
>>>>> AttributeError                            Traceback (most recent call last)
>>>>>
>>>>> /home/tasker/yt/src/yt-hg/scripts/iyt in <module>()
>>>>> ----> 1
>>>>>      2
>>>>>      3
>>>>>      4
>>>>>      5
>>>>>
>>>>> AttributeError: 'tuple' object has no attribute 'save_object'
>>>>>
>>>>> The alternative method --of saving it via the hierarchy-- is slightly
>>>>> less good since I'm running multiple yt scripts on the same data set.
>>>>> (As least, when I tried to load the values back in, it produced an
>>>>> empty set which I presume was because I'd inadvertently overwritten
>>>>> the .yt file?).
>>>>>
>>>>> If I could get one of these working again, then I'd be super happy. I
>>>>> don't need the all :)
>>>>>
>>>>> Elizabeth
>>>>>
>>>>>
>>>>> On 1 December 2011 12:35, Elizabeth Tasker
>>>>> <tasker at astro1.sci.hokudai.ac.jp> wrote:
>>>>>> Hi Matt,
>>>>>>
>>>>>> Sorry, that still doesn't work.... unless I've misunderstood your
>>>>>> suggestion? The code is here:
>>>>>>
>>>>>> http://paste.yt-project.org/show/1974/
>>>>>>
>>>>>> Pickle + unpickle on a pile of properties I calculate from the
>>>>>> connected sets is fine, but pickling the connected sets themselves is
>>>>>> a problem. Could it be the later pickle doesn't require the parameter
>>>>>> file? Or is it possible there is an unrelated write error happening?
>>>>>>
>>>>>> Elizabeth
>>>>>>
>>>>>> On 30 November 2011 23:43, Matthew Turk <matthewturk at gmail.com> wrote:
>>>>>>> Hi Elizabeth,
>>>>>>>
>>>>>>> We recently changed the mechanism of storing parameter files to be
>>>>>>> off-by-default, which means that rather than dumping them to a .csv
>>>>>>> file, they only stay in memory.  This functionality can be turned back
>>>>>>> on (this change is documented in the development documentation.)  Can
>>>>>>> you try:
>>>>>>>
>>>>>>> 1) Loading your parameter file, *then* loading your pickle.  I am
>>>>>>> optimistic this will fix it.
>>>>>>> 2) If it doesn't, at the very top of your python script, put this:
>>>>>>>
>>>>>>> from yt.config import ytcfg; ytcfg["yt","storeparameterfiles"] = "True"
>>>>>>>
>>>>>>> -Matt
>>>>>>>
>>>>>>> On Wed, Nov 30, 2011 at 4:46 AM, Elizabeth Tasker
>>>>>>> <tasker at astro1.sci.hokudai.ac.jp> wrote:
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> A bigger problem....
>>>>>>>>
>>>>>>>> The lastest version of yt
>>>>>>>>
>>>>>>>> The current version of the code is:
>>>>>>>>
>>>>>>>> ---
>>>>>>>> a519b8754ba8 (yt) tip
>>>>>>>> ---
>>>>>>>>
>>>>>>>>
>>>>>>>> has problems with pickle. If I pickle a data set and then try and
>>>>>>>> unpickle it, I see:
>>>>>>>>
>>>>>>>> unpickle core data
>>>>>>>> ---------------------------------------------------------------------------
>>>>>>>> KeyError                                  Traceback (most recent call last)
>>>>>>>>
>>>>>>>> /home/tasker/yt/src/yt-hg/scripts/iyt in <module>()
>>>>>>>>    338
>>>>>>>>    339     #file = open('cores.pickle','rb')
>>>>>>>> --> 340     allcloudcores = cPickle.load(file('cores.pickle', 'rb'))
>>>>>>>>    341     file.close()
>>>>>>>>    342
>>>>>>>>
>>>>>>>> /home/tasker/yt/src/yt-hg/yt/data_objects/data_containers.pyc in
>>>>>>>> _reconstruct_object(*args, **kwargs)
>>>>>>>>   3676         else: new_args.append(arg)
>>>>>>>>   3677     pfs = ParameterFileStore()
>>>>>>>> -> 3678     pf = pfs.get_pf_hash(pfid)
>>>>>>>>   3679     cls = getattr(pf.h, dtype)
>>>>>>>>   3680     obj = cls(*new_args)
>>>>>>>>
>>>>>>>> /home/tasker/yt/src/yt-hg/yt/utilities/parameter_file_storage.pyc in
>>>>>>>> get_pf_hash(self, hash)
>>>>>>>>    106     def get_pf_hash(self, hash):
>>>>>>>>    107         """ This returns a parameter file based on a hash. """
>>>>>>>> --> 108         return self._convert_pf(self._records[hash])
>>>>>>>>    109
>>>>>>>>    110     def get_pf_ctid(self, ctid):
>>>>>>>>
>>>>>>>> KeyError: (('283b7c4d88671dbff7acf083098da6ae',), <function
>>>>>>>> _reconstruct_object at 0x272c938>,
>>>>>>>> ('283b7c4d88671dbff7acf083098da6ae', 'region', array([ 16.,  16.,
>>>>>>>> 16.]), array([ 0.,  0.,  0.]), array([ 32.,  32.,  32.]),
>>>>>>>> {'disk_center': array([16, 16, 16]), 'center': array([ 16.,  16.,
>>>>>>>> 16.]), 'bulk_velocity': array([ 0.,  0.,  0.]), 'disk_vector':
>>>>>>>> array([0, 0, 1]), 'disk_radius': array([  0.1   ,   0.1358,   0.1716,
>>>>>>>>  0.2074,   0.2432,   0.279 ,
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> The fact it mentions the field parameters in the last line, might mean
>>>>>>>> it's an error introduced because of the corrections to them (that I
>>>>>>>> have been demanding!)?
>>>>>>>>
>>>>>>>> The script I have works fine (pickles and all) on yt version:
>>>>>>>>
>>>>>>>> ---
>>>>>>>> 16e8d749a806 (yt) tip
>>>>>>>> ---
>>>>>>>>
>>>>>>>>
>>>>>>>> Elizabeth
>>>>>>>> _______________________________________________
>>>>>>>> yt-users mailing list
>>>>>>>> yt-users at lists.spacepope.org
>>>>>>>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>>>>>>> _______________________________________________
>>>>>>> yt-users mailing list
>>>>>>> yt-users at lists.spacepope.org
>>>>>>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>>>>> _______________________________________________
>>>>> yt-users mailing list
>>>>> yt-users at lists.spacepope.org
>>>>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>>>> _______________________________________________
>>>> yt-users mailing list
>>>> yt-users at lists.spacepope.org
>>>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>>>
>>> _______________________________________________
>>> yt-users mailing list
>>> yt-users at lists.spacepope.org
>>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>> _______________________________________________
>> yt-users mailing list
>> yt-users at lists.spacepope.org
>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>
> _______________________________________________
> yt-users mailing list
> yt-users at lists.spacepope.org
> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org



More information about the yt-users mailing list