[yt-dev] Turn data serialization off by default

[David Collins]

I'm +1 on changing the default.  Thanks for making a announcement about the
change.

How hard would it be to make an individual routine get serialized on
demand?  For instance,
proj = pf.h.proj( ... serizlize = True)

Or, would it work to do
ytcfg['yt', 'serialize'] = 'True'
do stuff
ytcfg['yt', 'serialize'] = 'False'
?

d.



On Tue, Jul 23, 2013 at 6:20 PM, j s oishi <jsoishi at gmail.com> wrote:

> Oh god...+100000000000 <sound of coins dinging in 8 bit glory>
> On Jul 23, 2013 7:08 PM, "Matthew Turk" <matthewturk at gmail.com> wrote:
>
>> On Tue, Jul 23, 2013 at 3:27 PM, Nathan Goldbaum <nathan12343 at gmail.com>
>> wrote:
>> > Hi all,
>> >
>> > I've just issued a PR that will hopefully fix a whole class of buggy
>> > behavior that both new and experienced yt users commonly run into.
>> > Specifically, I'd like it if we could turn off data serialization by
>> > default.  This changes a long-lived default value in yt's
>> configuration, so
>> > I wanted to bring this change to the attention of both the yt user and
>> > developer community.
>> >
>> > What is data serialization?  Currently, yt will save the result of
>> certain
>> > expensive calculations, including projections, the structure of the grid
>> > hierarchy, and the list of fields present in the data.  While this does
>> have
>> > the beneficial effect of saving time when a user needs to repetitively
>> > calculate these quantities on the same dataset, it has a number of
>> features
>> > which lead to buggy, annoying behavior.
>> >
>> > Specifically, If I am developing my simulation code or repeatedly
>> restarting
>> > my code, searching for a way to grind past a code crash, I will quite
>> often
>> > regenerate the same simulation output file over and over, changing a
>> line of
>> > code or switching out the value of a parameter each time.
>> >
>> > If yt's data serialization is turned on, it's likely that yt's
>> > visualizations will correspond to old versions of the data file.  Since
>> only
>> > certain operations are serialized, it's also possible for yt to get
>> into an
>> > inconsistent state - one operation will show the current data file,
>> while
>> > another operation will show an old version.
>> >
>> > It's possible to fix a bug in your code, but because yt is still
>> loading the
>> > old data, you won't be able to tell that your bug is fixed until you
>> realize
>> > that you have .yt and .harrays files littering your filesystem.
>> >
>> > I've personally wasted a lot of time due to yt's serialization
>> 'feature' and
>> > denizens of our IRC channel and mailing list can attest to how often new
>> > users run into this behavior as well.
>> >
>> > My pull request only turns off serlialization by default, it doesn't
>> disable
>> > the capability completely.  Once the pull request is merged in, you can
>> turn
>> > on serialization either by adding an entry to your config file:
>> >
>> > $ cat ~/.yt/config
>> >
>> > [yt]
>> > serialize = True
>> >
>> > Or on a per-script basis:
>> >
>> > from yt.config import ytcfg
>> > ytcfg['yt', 'serialize'] = 'True'
>> > from yt.mods import *
>> >
>> > The pull request is here:
>> > https://bitbucket.org/yt_analysis/yt/pull-request/558
>> >
>> > I know several of you are big fans of this feature, so if you object to
>> this
>> > change please leave a comment on the pull request so we can figure out
>> a way
>> > forward.
>>
>> I think this is long overdue, for all the reasons you list.
>> Auto-serialization treated a lot of symptoms that we have since
>> improved, or that we should address more directly -- speed of
>> hierarchy construction, saving data that we want to retain, and
>> detecting fields.
>>
>> +1!
>>
>> -Matt
>>
>> >
>> > -Nathan
>> >
>> > _______________________________________________
>> > yt-dev mailing list
>> > yt-dev at lists.spacepope.org
>> > http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>> >
>> _______________________________________________
>> yt-dev mailing list
>> yt-dev at lists.spacepope.org
>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>>
>
> _______________________________________________
> yt-dev mailing list
> yt-dev at lists.spacepope.org
> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>
>


-- 
-- Sent from a computer.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.spacepope.org/pipermail/yt-dev-spacepope.org/attachments/20130723/fa85c932/attachment.html>