[yt-users] pickle
Elizabeth Tasker
tasker at astro1.sci.hokudai.ac.jp
Thu Dec 1 07:03:55 PST 2011
Tis:
In [1]: pf._hash()
Out[1]: '283b7c4d88671dbff7acf083098da6ae'
E.
On 2011-12-01, at 11:57 PM, Matthew Turk wrote:
> Hi Elizabeth,
>
> What is the output of pf._hash() ?
>
> -Matt
>
> On Thu, Dec 1, 2011 at 9:32 AM, Elizabeth Tasker
> <tasker at astro1.sci.hokudai.ac.jp> wrote:
>> Sorry, still no luck!
>>
>> The current version of the code is:
>>
>> ---
>> c1a446be3a7a (yt) tip
>> ---
>>
>>
>> unpickle core data
>> ---------------------------------------------------------------------------
>> KeyError Traceback (most recent call last)
>>
>> /home/taskere/yt-2/src/yt-hg/scripts/iyt in <module>()
>> 199
>> 200 file = open('cores.pickle','rb')
>> --> 201 allcloudcores = cPickle.load(file)
>> 202 file.close()
>> 203
>>
>> /home/taskere/yt-2/src/yt-hg/yt/data_objects/data_containers.pyc in _reconstruct_object(*args, **kwargs)
>> 3640 else: new_args.append(arg)
>> 3641 pfs = ParameterFileStore()
>> -> 3642 pf = pfs.get_pf_hash(pfid)
>> 3643 cls = getattr(pf.h, dtype)
>> 3644 obj = cls(*new_args)
>>
>> /home/taskere/yt-2/src/yt-hg/yt/utilities/parameter_file_storage.pyc in get_pf_hash(self, hash)
>> 109 def get_pf_hash(self, hash):
>> 110 """ This returns a parameter file based on a hash. """
>> --> 111 return self._convert_pf(self._records[hash])
>> 112
>> 113 def get_pf_ctid(self, ctid):
>>
>> KeyError: (('283b7c4d88671dbff7acf083098da6ae',), <function _reconstruct_object at 0xa8cd848>, ('283b7c4d88671dbff7acf083098da6ae', 'region', array([ 16., 16., 16.]), array([ 0., 0., 0.]), array([ 32., 32., 32.]), {'disk_center': array([16, 16, 16]), 'center': array([ 16., 16., 16.]), 'bulk_velocity': array([ 0., 0., 0.]), 'disk_vector': array([0, 0, 1]), 'disk_radius': array([ 0.1 , 0.1238, 0.1476, 0.1714, 0.1952, 0.219 ,
>> 0.2428, 0.2666, 0.2904, 0.3142, 0.33
>>
>>
>>
>> Unless I need to re-pickle first?
>>
>> Elizabeth
>>
>>
>>
>>
>> On 2011-12-01, at 8:20 PM, Matthew Turk wrote:
>>
>>> Thanks, Elizabeth. The move to in-memory brought up a funny bug where
>>> the initialization process was reseting the list of known parameter
>>> hashes. This is fixed in c1a446be3a7a.
>>>
>>> Also, save_object won't work except on yt objects; the result of
>>> extract_connected_sets is a tuple of values and python objects.
>>> Pickle will, tho.
>>>
>>> On Thu, Dec 1, 2011 at 4:33 AM, Elizabeth Tasker
>>> <tasker at astro1.sci.hokudai.ac.jp> wrote:
>>>> I also tried switching out pickle for the yt method of storing
>>>> objects. Unfortunately, it doesn't seem to save the object I want. I
>>>> tried:
>>>>
>>>> contours = dd.extract_connected_sets("NegEscapeVelocity", 1, 30.0,
>>>> maxv, log_space=False)
>>>> contours.save_object("cloudcores", "storedcores.cpkl")
>>>>
>>>> In [1]: contours.save_object("cloudcores", "storedcores.cpkl")
>>>> ---------------------------------------------------------------------------
>>>> AttributeError Traceback (most recent call last)
>>>>
>>>> /home/tasker/yt/src/yt-hg/scripts/iyt in <module>()
>>>> ----> 1
>>>> 2
>>>> 3
>>>> 4
>>>> 5
>>>>
>>>> AttributeError: 'tuple' object has no attribute 'save_object'
>>>>
>>>> The alternative method --of saving it via the hierarchy-- is slightly
>>>> less good since I'm running multiple yt scripts on the same data set.
>>>> (As least, when I tried to load the values back in, it produced an
>>>> empty set which I presume was because I'd inadvertently overwritten
>>>> the .yt file?).
>>>>
>>>> If I could get one of these working again, then I'd be super happy. I
>>>> don't need the all :)
>>>>
>>>> Elizabeth
>>>>
>>>>
>>>> On 1 December 2011 12:35, Elizabeth Tasker
>>>> <tasker at astro1.sci.hokudai.ac.jp> wrote:
>>>>> Hi Matt,
>>>>>
>>>>> Sorry, that still doesn't work.... unless I've misunderstood your
>>>>> suggestion? The code is here:
>>>>>
>>>>> http://paste.yt-project.org/show/1974/
>>>>>
>>>>> Pickle + unpickle on a pile of properties I calculate from the
>>>>> connected sets is fine, but pickling the connected sets themselves is
>>>>> a problem. Could it be the later pickle doesn't require the parameter
>>>>> file? Or is it possible there is an unrelated write error happening?
>>>>>
>>>>> Elizabeth
>>>>>
>>>>> On 30 November 2011 23:43, Matthew Turk <matthewturk at gmail.com> wrote:
>>>>>> Hi Elizabeth,
>>>>>>
>>>>>> We recently changed the mechanism of storing parameter files to be
>>>>>> off-by-default, which means that rather than dumping them to a .csv
>>>>>> file, they only stay in memory. This functionality can be turned back
>>>>>> on (this change is documented in the development documentation.) Can
>>>>>> you try:
>>>>>>
>>>>>> 1) Loading your parameter file, *then* loading your pickle. I am
>>>>>> optimistic this will fix it.
>>>>>> 2) If it doesn't, at the very top of your python script, put this:
>>>>>>
>>>>>> from yt.config import ytcfg; ytcfg["yt","storeparameterfiles"] = "True"
>>>>>>
>>>>>> -Matt
>>>>>>
>>>>>> On Wed, Nov 30, 2011 at 4:46 AM, Elizabeth Tasker
>>>>>> <tasker at astro1.sci.hokudai.ac.jp> wrote:
>>>>>>> Hi,
>>>>>>>
>>>>>>> A bigger problem....
>>>>>>>
>>>>>>> The lastest version of yt
>>>>>>>
>>>>>>> The current version of the code is:
>>>>>>>
>>>>>>> ---
>>>>>>> a519b8754ba8 (yt) tip
>>>>>>> ---
>>>>>>>
>>>>>>>
>>>>>>> has problems with pickle. If I pickle a data set and then try and
>>>>>>> unpickle it, I see:
>>>>>>>
>>>>>>> unpickle core data
>>>>>>> ---------------------------------------------------------------------------
>>>>>>> KeyError Traceback (most recent call last)
>>>>>>>
>>>>>>> /home/tasker/yt/src/yt-hg/scripts/iyt in <module>()
>>>>>>> 338
>>>>>>> 339 #file = open('cores.pickle','rb')
>>>>>>> --> 340 allcloudcores = cPickle.load(file('cores.pickle', 'rb'))
>>>>>>> 341 file.close()
>>>>>>> 342
>>>>>>>
>>>>>>> /home/tasker/yt/src/yt-hg/yt/data_objects/data_containers.pyc in
>>>>>>> _reconstruct_object(*args, **kwargs)
>>>>>>> 3676 else: new_args.append(arg)
>>>>>>> 3677 pfs = ParameterFileStore()
>>>>>>> -> 3678 pf = pfs.get_pf_hash(pfid)
>>>>>>> 3679 cls = getattr(pf.h, dtype)
>>>>>>> 3680 obj = cls(*new_args)
>>>>>>>
>>>>>>> /home/tasker/yt/src/yt-hg/yt/utilities/parameter_file_storage.pyc in
>>>>>>> get_pf_hash(self, hash)
>>>>>>> 106 def get_pf_hash(self, hash):
>>>>>>> 107 """ This returns a parameter file based on a hash. """
>>>>>>> --> 108 return self._convert_pf(self._records[hash])
>>>>>>> 109
>>>>>>> 110 def get_pf_ctid(self, ctid):
>>>>>>>
>>>>>>> KeyError: (('283b7c4d88671dbff7acf083098da6ae',), <function
>>>>>>> _reconstruct_object at 0x272c938>,
>>>>>>> ('283b7c4d88671dbff7acf083098da6ae', 'region', array([ 16., 16.,
>>>>>>> 16.]), array([ 0., 0., 0.]), array([ 32., 32., 32.]),
>>>>>>> {'disk_center': array([16, 16, 16]), 'center': array([ 16., 16.,
>>>>>>> 16.]), 'bulk_velocity': array([ 0., 0., 0.]), 'disk_vector':
>>>>>>> array([0, 0, 1]), 'disk_radius': array([ 0.1 , 0.1358, 0.1716,
>>>>>>> 0.2074, 0.2432, 0.279 ,
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> The fact it mentions the field parameters in the last line, might mean
>>>>>>> it's an error introduced because of the corrections to them (that I
>>>>>>> have been demanding!)?
>>>>>>>
>>>>>>> The script I have works fine (pickles and all) on yt version:
>>>>>>>
>>>>>>> ---
>>>>>>> 16e8d749a806 (yt) tip
>>>>>>> ---
>>>>>>>
>>>>>>>
>>>>>>> Elizabeth
>>>>>>> _______________________________________________
>>>>>>> yt-users mailing list
>>>>>>> yt-users at lists.spacepope.org
>>>>>>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>>>>>> _______________________________________________
>>>>>> yt-users mailing list
>>>>>> yt-users at lists.spacepope.org
>>>>>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>>>> _______________________________________________
>>>> yt-users mailing list
>>>> yt-users at lists.spacepope.org
>>>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>>> _______________________________________________
>>> yt-users mailing list
>>> yt-users at lists.spacepope.org
>>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>>
>> _______________________________________________
>> yt-users mailing list
>> yt-users at lists.spacepope.org
>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
> _______________________________________________
> yt-users mailing list
> yt-users at lists.spacepope.org
> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
More information about the yt-users
mailing list