<div dir="ltr"><div>Thanks for the examples. <br><br></div><div>I'm a little unclear about your last statement-- will pickling the objects directly work with serialization off? <br><br></div><div>Thanks!<br>d.<br></div>
</div><div class="gmail_extra"><br><br><div class="gmail_quote">On Thu, Jul 25, 2013 at 12:55 AM, Nathan Goldbaum <span dir="ltr"><<a href="mailto:goldbaum@ucolick.org" target="_blank">goldbaum@ucolick.org</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Hey David,<div><br></div><div>I don't think you can modify the ytcfg object after loading up yt, so your second example won't work.</div>
<div><br></div><div>As for your first example, I think that's possible via pickling:</div>
<div><br></div><div>with open('data.pickle', 'wb') as pkl_file:</div><div> s = cPickle.dumps(proj, pkl_file, protocol=-1)</div><div><br></div><div>You can then load it later like so:</div><div><br></div>
<div>with open('data.pickle', 'rb') as pkl_file:</div><div> proj = cPickle.load(pkl_file)</div><div><br></div><div>You can do similar things using pf.h.save_object() and load_object(), but in a bit of a chicken and egg situation, you'll need serialization turned on in your config parameters for that to work.<span class="HOEnZb"><font color="#888888"><br>
</font></span></div><span class="HOEnZb"><font color="#888888"><div><br></div><div>-Nathan</div></font></span></div><div class="HOEnZb"><div class="h5"><div class="gmail_extra"><br><br><div class="gmail_quote">On Tue, Jul 23, 2013 at 8:03 PM, David Collins <span dir="ltr"><<a href="mailto:dcollins4096@gmail.com" target="_blank">dcollins4096@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><br><div>I'm +1 on changing the default. Thanks for making a announcement about the change.<br><br>
<div>How hard would it be to make an individual routine get serialized on demand? For instance,<br></div>
<div>proj = pf.h.proj( ... serizlize = True)<br><br></div><div>Or, would it work to do<div><br>ytcfg['yt', 'serialize'] = 'True'<br></div></div>do stuff<br>ytcfg['yt', 'serialize'] = 'False'<br>
?<br><br></div><div>d.<br></div><div><br></div></div><div class="gmail_extra"><div><div><br><br><div class="gmail_quote">On Tue, Jul 23, 2013 at 6:20 PM, j s oishi <span dir="ltr"><<a href="mailto:jsoishi@gmail.com" target="_blank">jsoishi@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><p dir="ltr">Oh god...+100000000000 <sound of coins dinging in 8 bit glory></p><div><div>
<div class="gmail_quote">On Jul 23, 2013 7:08 PM, "Matthew Turk" <<a href="mailto:matthewturk@gmail.com" target="_blank">matthewturk@gmail.com</a>> wrote:<br type="attribution"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
On Tue, Jul 23, 2013 at 3:27 PM, Nathan Goldbaum <<a href="mailto:nathan12343@gmail.com" target="_blank">nathan12343@gmail.com</a>> wrote:<br>
> Hi all,<br>
><br>
> I've just issued a PR that will hopefully fix a whole class of buggy<br>
> behavior that both new and experienced yt users commonly run into.<br>
> Specifically, I'd like it if we could turn off data serialization by<br>
> default. This changes a long-lived default value in yt's configuration, so<br>
> I wanted to bring this change to the attention of both the yt user and<br>
> developer community.<br>
><br>
> What is data serialization? Currently, yt will save the result of certain<br>
> expensive calculations, including projections, the structure of the grid<br>
> hierarchy, and the list of fields present in the data. While this does have<br>
> the beneficial effect of saving time when a user needs to repetitively<br>
> calculate these quantities on the same dataset, it has a number of features<br>
> which lead to buggy, annoying behavior.<br>
><br>
> Specifically, If I am developing my simulation code or repeatedly restarting<br>
> my code, searching for a way to grind past a code crash, I will quite often<br>
> regenerate the same simulation output file over and over, changing a line of<br>
> code or switching out the value of a parameter each time.<br>
><br>
> If yt's data serialization is turned on, it's likely that yt's<br>
> visualizations will correspond to old versions of the data file. Since only<br>
> certain operations are serialized, it's also possible for yt to get into an<br>
> inconsistent state - one operation will show the current data file, while<br>
> another operation will show an old version.<br>
><br>
> It's possible to fix a bug in your code, but because yt is still loading the<br>
> old data, you won't be able to tell that your bug is fixed until you realize<br>
> that you have .yt and .harrays files littering your filesystem.<br>
><br>
> I've personally wasted a lot of time due to yt's serialization 'feature' and<br>
> denizens of our IRC channel and mailing list can attest to how often new<br>
> users run into this behavior as well.<br>
><br>
> My pull request only turns off serlialization by default, it doesn't disable<br>
> the capability completely. Once the pull request is merged in, you can turn<br>
> on serialization either by adding an entry to your config file:<br>
><br>
> $ cat ~/.yt/config<br>
><br>
> [yt]<br>
> serialize = True<br>
><br>
> Or on a per-script basis:<br>
><br>
> from yt.config import ytcfg<br>
> ytcfg['yt', 'serialize'] = 'True'<br>
> from yt.mods import *<br>
><br>
> The pull request is here:<br>
> <a href="https://bitbucket.org/yt_analysis/yt/pull-request/558" target="_blank">https://bitbucket.org/yt_analysis/yt/pull-request/558</a><br>
><br>
> I know several of you are big fans of this feature, so if you object to this<br>
> change please leave a comment on the pull request so we can figure out a way<br>
> forward.<br>
<br>
I think this is long overdue, for all the reasons you list.<br>
Auto-serialization treated a lot of symptoms that we have since<br>
improved, or that we should address more directly -- speed of<br>
hierarchy construction, saving data that we want to retain, and<br>
detecting fields.<br>
<br>
+1!<br>
<br>
-Matt<br>
<br>
><br>
> -Nathan<br>
><br>
> _______________________________________________<br>
> yt-dev mailing list<br>
> <a href="mailto:yt-dev@lists.spacepope.org" target="_blank">yt-dev@lists.spacepope.org</a><br>
> <a href="http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org" target="_blank">http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org</a><br>
><br>
_______________________________________________<br>
yt-dev mailing list<br>
<a href="mailto:yt-dev@lists.spacepope.org" target="_blank">yt-dev@lists.spacepope.org</a><br>
<a href="http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org" target="_blank">http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org</a><br>
</blockquote></div>
</div></div><br>_______________________________________________<br>
yt-dev mailing list<br>
<a href="mailto:yt-dev@lists.spacepope.org" target="_blank">yt-dev@lists.spacepope.org</a><br>
<a href="http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org" target="_blank">http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org</a><br>
<br></blockquote></div><br><br clear="all"><br></div></div><span><font color="#888888">-- <br><div dir="ltr">-- Sent from a computer.<br></div>
</font></span></div>
<br>_______________________________________________<br>
yt-dev mailing list<br>
<a href="mailto:yt-dev@lists.spacepope.org" target="_blank">yt-dev@lists.spacepope.org</a><br>
<a href="http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org" target="_blank">http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org</a><br>
<br></blockquote></div><br></div>
</div></div><br>_______________________________________________<br>
yt-dev mailing list<br>
<a href="mailto:yt-dev@lists.spacepope.org">yt-dev@lists.spacepope.org</a><br>
<a href="http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org" target="_blank">http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org</a><br>
<br></blockquote></div><br><br clear="all"><br>-- <br><div dir="ltr">-- Sent from a computer.<br></div>
</div>