[yt-users] pickle

Matthew Turk matthewturk at gmail.com
Thu Dec 1 06:57:58 PST 2011


Hi Elizabeth,

What is the output of pf._hash() ?

-Matt

On Thu, Dec 1, 2011 at 9:32 AM, Elizabeth Tasker
<tasker at astro1.sci.hokudai.ac.jp> wrote:
> Sorry, still no luck!
>
> The current version of the code is:
>
> ---
> c1a446be3a7a (yt) tip
> ---
>
>
> unpickle core data
> ---------------------------------------------------------------------------
> KeyError                                  Traceback (most recent call last)
>
> /home/taskere/yt-2/src/yt-hg/scripts/iyt in <module>()
>    199
>    200     file = open('cores.pickle','rb')
> --> 201     allcloudcores = cPickle.load(file)
>    202     file.close()
>    203
>
> /home/taskere/yt-2/src/yt-hg/yt/data_objects/data_containers.pyc in _reconstruct_object(*args, **kwargs)
>   3640         else: new_args.append(arg)
>   3641     pfs = ParameterFileStore()
> -> 3642     pf = pfs.get_pf_hash(pfid)
>   3643     cls = getattr(pf.h, dtype)
>   3644     obj = cls(*new_args)
>
> /home/taskere/yt-2/src/yt-hg/yt/utilities/parameter_file_storage.pyc in get_pf_hash(self, hash)
>    109     def get_pf_hash(self, hash):
>    110         """ This returns a parameter file based on a hash. """
> --> 111         return self._convert_pf(self._records[hash])
>    112
>    113     def get_pf_ctid(self, ctid):
>
> KeyError: (('283b7c4d88671dbff7acf083098da6ae',), <function _reconstruct_object at 0xa8cd848>, ('283b7c4d88671dbff7acf083098da6ae', 'region', array([ 16.,  16.,  16.]), array([ 0.,  0.,  0.]), array([ 32.,  32.,  32.]), {'disk_center': array([16, 16, 16]), 'center': array([ 16.,  16.,  16.]), 'bulk_velocity': array([ 0.,  0.,  0.]), 'disk_vector': array([0, 0, 1]), 'disk_radius': array([  0.1   ,   0.1238,   0.1476,   0.1714,   0.1952,   0.219 ,
>         0.2428,   0.2666,   0.2904,   0.3142,   0.33
>
>
>
> Unless I need to re-pickle first?
>
> Elizabeth
>
>
>
>
> On 2011-12-01, at 8:20 PM, Matthew Turk wrote:
>
>> Thanks, Elizabeth.  The move to in-memory brought up a funny bug where
>> the initialization process was reseting the list of known parameter
>> hashes.  This is fixed in c1a446be3a7a.
>>
>> Also, save_object won't work except on yt objects; the result of
>> extract_connected_sets is a tuple of values and python objects.
>> Pickle will, tho.
>>
>> On Thu, Dec 1, 2011 at 4:33 AM, Elizabeth Tasker
>> <tasker at astro1.sci.hokudai.ac.jp> wrote:
>>> I also tried switching out pickle for the yt method of storing
>>> objects. Unfortunately, it doesn't seem to save the object I want. I
>>> tried:
>>>
>>> contours = dd.extract_connected_sets("NegEscapeVelocity", 1, 30.0,
>>> maxv, log_space=False)
>>> contours.save_object("cloudcores", "storedcores.cpkl")
>>>
>>> In [1]: contours.save_object("cloudcores", "storedcores.cpkl")
>>> ---------------------------------------------------------------------------
>>> AttributeError                            Traceback (most recent call last)
>>>
>>> /home/tasker/yt/src/yt-hg/scripts/iyt in <module>()
>>> ----> 1
>>>      2
>>>      3
>>>      4
>>>      5
>>>
>>> AttributeError: 'tuple' object has no attribute 'save_object'
>>>
>>> The alternative method --of saving it via the hierarchy-- is slightly
>>> less good since I'm running multiple yt scripts on the same data set.
>>> (As least, when I tried to load the values back in, it produced an
>>> empty set which I presume was because I'd inadvertently overwritten
>>> the .yt file?).
>>>
>>> If I could get one of these working again, then I'd be super happy. I
>>> don't need the all :)
>>>
>>> Elizabeth
>>>
>>>
>>> On 1 December 2011 12:35, Elizabeth Tasker
>>> <tasker at astro1.sci.hokudai.ac.jp> wrote:
>>>> Hi Matt,
>>>>
>>>> Sorry, that still doesn't work.... unless I've misunderstood your
>>>> suggestion? The code is here:
>>>>
>>>> http://paste.yt-project.org/show/1974/
>>>>
>>>> Pickle + unpickle on a pile of properties I calculate from the
>>>> connected sets is fine, but pickling the connected sets themselves is
>>>> a problem. Could it be the later pickle doesn't require the parameter
>>>> file? Or is it possible there is an unrelated write error happening?
>>>>
>>>> Elizabeth
>>>>
>>>> On 30 November 2011 23:43, Matthew Turk <matthewturk at gmail.com> wrote:
>>>>> Hi Elizabeth,
>>>>>
>>>>> We recently changed the mechanism of storing parameter files to be
>>>>> off-by-default, which means that rather than dumping them to a .csv
>>>>> file, they only stay in memory.  This functionality can be turned back
>>>>> on (this change is documented in the development documentation.)  Can
>>>>> you try:
>>>>>
>>>>> 1) Loading your parameter file, *then* loading your pickle.  I am
>>>>> optimistic this will fix it.
>>>>> 2) If it doesn't, at the very top of your python script, put this:
>>>>>
>>>>> from yt.config import ytcfg; ytcfg["yt","storeparameterfiles"] = "True"
>>>>>
>>>>> -Matt
>>>>>
>>>>> On Wed, Nov 30, 2011 at 4:46 AM, Elizabeth Tasker
>>>>> <tasker at astro1.sci.hokudai.ac.jp> wrote:
>>>>>> Hi,
>>>>>>
>>>>>> A bigger problem....
>>>>>>
>>>>>> The lastest version of yt
>>>>>>
>>>>>> The current version of the code is:
>>>>>>
>>>>>> ---
>>>>>> a519b8754ba8 (yt) tip
>>>>>> ---
>>>>>>
>>>>>>
>>>>>> has problems with pickle. If I pickle a data set and then try and
>>>>>> unpickle it, I see:
>>>>>>
>>>>>> unpickle core data
>>>>>> ---------------------------------------------------------------------------
>>>>>> KeyError                                  Traceback (most recent call last)
>>>>>>
>>>>>> /home/tasker/yt/src/yt-hg/scripts/iyt in <module>()
>>>>>>    338
>>>>>>    339     #file = open('cores.pickle','rb')
>>>>>> --> 340     allcloudcores = cPickle.load(file('cores.pickle', 'rb'))
>>>>>>    341     file.close()
>>>>>>    342
>>>>>>
>>>>>> /home/tasker/yt/src/yt-hg/yt/data_objects/data_containers.pyc in
>>>>>> _reconstruct_object(*args, **kwargs)
>>>>>>   3676         else: new_args.append(arg)
>>>>>>   3677     pfs = ParameterFileStore()
>>>>>> -> 3678     pf = pfs.get_pf_hash(pfid)
>>>>>>   3679     cls = getattr(pf.h, dtype)
>>>>>>   3680     obj = cls(*new_args)
>>>>>>
>>>>>> /home/tasker/yt/src/yt-hg/yt/utilities/parameter_file_storage.pyc in
>>>>>> get_pf_hash(self, hash)
>>>>>>    106     def get_pf_hash(self, hash):
>>>>>>    107         """ This returns a parameter file based on a hash. """
>>>>>> --> 108         return self._convert_pf(self._records[hash])
>>>>>>    109
>>>>>>    110     def get_pf_ctid(self, ctid):
>>>>>>
>>>>>> KeyError: (('283b7c4d88671dbff7acf083098da6ae',), <function
>>>>>> _reconstruct_object at 0x272c938>,
>>>>>> ('283b7c4d88671dbff7acf083098da6ae', 'region', array([ 16.,  16.,
>>>>>> 16.]), array([ 0.,  0.,  0.]), array([ 32.,  32.,  32.]),
>>>>>> {'disk_center': array([16, 16, 16]), 'center': array([ 16.,  16.,
>>>>>> 16.]), 'bulk_velocity': array([ 0.,  0.,  0.]), 'disk_vector':
>>>>>> array([0, 0, 1]), 'disk_radius': array([  0.1   ,   0.1358,   0.1716,
>>>>>>  0.2074,   0.2432,   0.279 ,
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> The fact it mentions the field parameters in the last line, might mean
>>>>>> it's an error introduced because of the corrections to them (that I
>>>>>> have been demanding!)?
>>>>>>
>>>>>> The script I have works fine (pickles and all) on yt version:
>>>>>>
>>>>>> ---
>>>>>> 16e8d749a806 (yt) tip
>>>>>> ---
>>>>>>
>>>>>>
>>>>>> Elizabeth
>>>>>> _______________________________________________
>>>>>> yt-users mailing list
>>>>>> yt-users at lists.spacepope.org
>>>>>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>>>>> _______________________________________________
>>>>> yt-users mailing list
>>>>> yt-users at lists.spacepope.org
>>>>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>>> _______________________________________________
>>> yt-users mailing list
>>> yt-users at lists.spacepope.org
>>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>> _______________________________________________
>> yt-users mailing list
>> yt-users at lists.spacepope.org
>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>
> _______________________________________________
> yt-users mailing list
> yt-users at lists.spacepope.org
> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org



More information about the yt-users mailing list