[yt-users] pickle

[Matthew Turk]

Thanks, Elizabeth.  The move to in-memory brought up a funny bug where
the initialization process was reseting the list of known parameter
hashes.  This is fixed in c1a446be3a7a.

Also, save_object won't work except on yt objects; the result of
extract_connected_sets is a tuple of values and python objects.
Pickle will, tho.

On Thu, Dec 1, 2011 at 4:33 AM, Elizabeth Tasker
<tasker at astro1.sci.hokudai.ac.jp> wrote:
> I also tried switching out pickle for the yt method of storing
> objects. Unfortunately, it doesn't seem to save the object I want. I
> tried:
>
> contours = dd.extract_connected_sets("NegEscapeVelocity", 1, 30.0,
> maxv, log_space=False)
> contours.save_object("cloudcores", "storedcores.cpkl")
>
> In [1]: contours.save_object("cloudcores", "storedcores.cpkl")
> ---------------------------------------------------------------------------
> AttributeError                            Traceback (most recent call last)
>
> /home/tasker/yt/src/yt-hg/scripts/iyt in <module>()
> ----> 1
>      2
>      3
>      4
>      5
>
> AttributeError: 'tuple' object has no attribute 'save_object'
>
> The alternative method --of saving it via the hierarchy-- is slightly
> less good since I'm running multiple yt scripts on the same data set.
> (As least, when I tried to load the values back in, it produced an
> empty set which I presume was because I'd inadvertently overwritten
> the .yt file?).
>
> If I could get one of these working again, then I'd be super happy. I
> don't need the all :)
>
> Elizabeth
>
>
> On 1 December 2011 12:35, Elizabeth Tasker
> <tasker at astro1.sci.hokudai.ac.jp> wrote:
>> Hi Matt,
>>
>> Sorry, that still doesn't work.... unless I've misunderstood your
>> suggestion? The code is here:
>>
>> http://paste.yt-project.org/show/1974/
>>
>> Pickle + unpickle on a pile of properties I calculate from the
>> connected sets is fine, but pickling the connected sets themselves is
>> a problem. Could it be the later pickle doesn't require the parameter
>> file? Or is it possible there is an unrelated write error happening?
>>
>> Elizabeth
>>
>> On 30 November 2011 23:43, Matthew Turk <matthewturk at gmail.com> wrote:
>>> Hi Elizabeth,
>>>
>>> We recently changed the mechanism of storing parameter files to be
>>> off-by-default, which means that rather than dumping them to a .csv
>>> file, they only stay in memory.  This functionality can be turned back
>>> on (this change is documented in the development documentation.)  Can
>>> you try:
>>>
>>> 1) Loading your parameter file, *then* loading your pickle.  I am
>>> optimistic this will fix it.
>>> 2) If it doesn't, at the very top of your python script, put this:
>>>
>>> from yt.config import ytcfg; ytcfg["yt","storeparameterfiles"] = "True"
>>>
>>> -Matt
>>>
>>> On Wed, Nov 30, 2011 at 4:46 AM, Elizabeth Tasker
>>> <tasker at astro1.sci.hokudai.ac.jp> wrote:
>>>> Hi,
>>>>
>>>> A bigger problem....
>>>>
>>>> The lastest version of yt
>>>>
>>>> The current version of the code is:
>>>>
>>>> ---
>>>> a519b8754ba8 (yt) tip
>>>> ---
>>>>
>>>>
>>>> has problems with pickle. If I pickle a data set and then try and
>>>> unpickle it, I see:
>>>>
>>>> unpickle core data
>>>> ---------------------------------------------------------------------------
>>>> KeyError                                  Traceback (most recent call last)
>>>>
>>>> /home/tasker/yt/src/yt-hg/scripts/iyt in <module>()
>>>>    338
>>>>    339     #file = open('cores.pickle','rb')
>>>> --> 340     allcloudcores = cPickle.load(file('cores.pickle', 'rb'))
>>>>    341     file.close()
>>>>    342
>>>>
>>>> /home/tasker/yt/src/yt-hg/yt/data_objects/data_containers.pyc in
>>>> _reconstruct_object(*args, **kwargs)
>>>>   3676         else: new_args.append(arg)
>>>>   3677     pfs = ParameterFileStore()
>>>> -> 3678     pf = pfs.get_pf_hash(pfid)
>>>>   3679     cls = getattr(pf.h, dtype)
>>>>   3680     obj = cls(*new_args)
>>>>
>>>> /home/tasker/yt/src/yt-hg/yt/utilities/parameter_file_storage.pyc in
>>>> get_pf_hash(self, hash)
>>>>    106     def get_pf_hash(self, hash):
>>>>    107         """ This returns a parameter file based on a hash. """
>>>> --> 108         return self._convert_pf(self._records[hash])
>>>>    109
>>>>    110     def get_pf_ctid(self, ctid):
>>>>
>>>> KeyError: (('283b7c4d88671dbff7acf083098da6ae',), <function
>>>> _reconstruct_object at 0x272c938>,
>>>> ('283b7c4d88671dbff7acf083098da6ae', 'region', array([ 16.,  16.,
>>>> 16.]), array([ 0.,  0.,  0.]), array([ 32.,  32.,  32.]),
>>>> {'disk_center': array([16, 16, 16]), 'center': array([ 16.,  16.,
>>>> 16.]), 'bulk_velocity': array([ 0.,  0.,  0.]), 'disk_vector':
>>>> array([0, 0, 1]), 'disk_radius': array([  0.1   ,   0.1358,   0.1716,
>>>>  0.2074,   0.2432,   0.279 ,
>>>>
>>>>
>>>>
>>>>
>>>> The fact it mentions the field parameters in the last line, might mean
>>>> it's an error introduced because of the corrections to them (that I
>>>> have been demanding!)?
>>>>
>>>> The script I have works fine (pickles and all) on yt version:
>>>>
>>>> ---
>>>> 16e8d749a806 (yt) tip
>>>> ---
>>>>
>>>>
>>>> Elizabeth
>>>> _______________________________________________
>>>> yt-users mailing list
>>>> yt-users at lists.spacepope.org
>>>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>>> _______________________________________________
>>> yt-users mailing list
>>> yt-users at lists.spacepope.org
>>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
> _______________________________________________
> yt-users mailing list
> yt-users at lists.spacepope.org
> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org