[yt-users] pickling clumps

Matthew Turk matthewturk at gmail.com
Fri May 28 10:14:14 PDT 2010


Hi Britton, Charles,

Britton is correct, YT has its own routine for pickling.  This is a
couple step process, because specifically the pickling of parameter
files is somewhat ill-defined as a problem.  When you initialize a
parameter file, it's given a (likely) unique hash that corresponds to
its location, some of its contents, last modified time (or in Enzo's
case, the CurrentTimeIdentifier in the parameter file) and this is
stored in ~/.yt/parameter_files.csv.  By this mechanism, one can store
the pointer to a parameter file that exists on disk with a
bidirectional hash.  When a new parameter file is opened, it checks to
see if the path location in the .csv file needs to be updated, and if
so, it does.

So when pickling a YT object, it stores the minimum set of information
necessary to recreate that object.  For a sphere, region, etc etc,
this is really just the necessary arguments to recreate the empty
object, along with the hash of the parameter file that it belongs to.
So the actual information necessary to pickle simple objects is very
small: hash, arguments.  (Called "_con_args" in the source, for
Constructor Arguments.)  When this is unpickled, the parameter file is
recreated (or grabbed from in memory, if it has already been
instantiated) and then the object is recreated.

For more complicated objects, like Clumps and ExtractedSets, we have
to store more information -- because the constructor arguments are
much larger and more complicated.  But the basic idea is the same.

When using save_object, the same pickling routine gets called, but
instead of being loosely affiliated and put into a separate file, it
gets stored as a dataset in a the corresponding .yt file, which is
itself an HDF5 file.  So the results should be the same, except
save_object and load_object remove the extra file from the equation.

The error you're seeing above seems to be related to an incompletely
saved or corrupted file; can you replicate this in a very simple,
quick fashion?  I would suggest getting a clump that is very, very
small, taken from a small data region, and seeing if that will pickle
and unpickle correctly.  If it does, then we know that perhaps the
previous pickle file was corrupted, or that something is wrong with
the yt pickling protocol.  If it doesn't work, then we *know*
something is wrong with the yt pickling protocol.

Best,

Matt

On Thu, May 27, 2010 at 6:43 PM, Britton Smith <brittonsmith at gmail.com> wrote:
> Hi Chris,
>
> YT has its own routine designed to pickle data objects for saving and
> reloading.  For a given pf, you can do pf.h.save_object(object, name) and
> the object will be saved to the .yt file associated with the dataset.  You
> can then get it back with object = pf.h.load_object(name).  I have used this
> successfully to save the mast_clump structure from clump finding.
>
> Britton
>
> On Thu, May 27, 2010 at 9:26 PM, Charles Hansen <chansen at astro.berkeley.edu>
> wrote:
>>
>> The find_clumps function is working correctly for me I believe, but the
>> analysis takes a very long time and I'd like to save the master_clump object
>> (from
>> http://yt.enzotools.org/doc/cookbook/recipes.html#cookbook-find-clumps) for
>> further analysis.  I've tried pickling master_clump.  It dumps correctly,
>> but gives an EOF error (below) on loading.  Is there another way to save
>> master_clump?
>>
>> Thanks,
>> Charles
>>
>> /nics/c/home/chansen/lib/yt/lib/python2.6/pickle.pyc in load(file)
>>   1368
>>   1369 def load(file):
>> -> 1370     return Unpickler(file).load()
>>   1371
>>   1372 def loads(str):
>>
>> /nics/c/home/chansen/lib/yt/lib/python2.6/pickle.pyc in load(self)
>>    856             while 1:
>>    857                 key = read(1)
>> --> 858                 dispatch[key](self)
>>    859         except _Stop, stopinst:
>>    860             return stopinst.value
>>
>> /nics/c/home/chansen/lib/yt/lib/python2.6/pickle.pyc in load_eof(self)
>>    878
>>    879     def load_eof(self):
>> --> 880         raise EOFError
>>    881     dispatch[''] = load_eof
>>    882
>>
>> EOFError:
>>
>> _______________________________________________
>> yt-users mailing list
>> yt-users at lists.spacepope.org
>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>
>
> _______________________________________________
> yt-users mailing list
> yt-users at lists.spacepope.org
> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>
>



More information about the yt-users mailing list