[yt-dev] When can we turn off serialization by default?

[Matthew Turk]

On Fri, Apr 19, 2013 at 12:01 PM, david collins <antpuncher at gmail.com> wrote:
> I just did a test.  512^3 with 4 levels everywhere.  On my desktop, it took
> 78 seconds.  On Kraken it took 19 minutes. Unsurprisingly, Kraken's disk is
> not that fast today.  This is, of course, and extreme example, but it isn't
> by any means rare for that system.  78 seconds isn't too bad, but today I'll
> be looking at >20 datasets, so it adds up.

I understand, 78 seconds will definitely add up.  I'm impressed by the
difference between Kraken and local.  Or maybe flabberghasted?  Either
way, it's definitely something.

>
> I definitely am in favor of an opt-in system, though, as many applications
> won't be this shape or on this disk.  Or even an explicit call, such as
> ProjectinPlot.serialize()

Okay, cool.  I think that'd be a good one, like Britton mentioned too.

>
>
> pf = load(fname)
> t0=time.time()
> proj=ProjectionPlot(pf,2,'Density')
> t1=time.time()
> print t1-t0
> Parsing Hierarchy100%
> ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
> Time: 00:00:00
> Initializing tree  0 /  4100%
> ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
> Time: 00:00:00
> Projecting  level  0 /  4 100%
> |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
> Time: 00:03:39
> Initializing tree  1 /  4100%
> ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
> Time: 00:00:00
> Projecting  level  1 /  4 100%
> |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
> Time: 00:04:12
> Initializing tree  2 /  4100%
> ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
> Time: 00:00:00
> Projecting  level  2 /  4  80%
> |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
> | ETA:  00:00:52
> Projecting  level  2 /  4 100%
> |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
> Time: 00:04:17
> Initializing tree  3 /  4100%
> ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
> Time: 00:00:00
> Projecting  level  3 /  4 100%
> |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
> Time: 00:02:58
> Initializing tree  4 /  4100%
> ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
> Time: 00:00:00
> Projecting  level  4 /  4 100%
> |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
> Time: 00:04:06
> 1167.0001719
>
>
>
>
> On Fri, Apr 19, 2013 at 9:18 AM, Matthew Turk <matthewturk at gmail.com> wrote:
>>
>> Dave, out of curiosity, how long does a projection of one of your datasets
>> take?
>>
>> On Fri, Apr 19, 2013 at 11:17 AM, david collins <antpuncher at gmail.com>
>> wrote:
>> >
>> > I use serialized projections quite frequently.  As long as the current
>> > behavior, or some reasonable facsimile, is still available I'm for doing
>> > it
>> > now.
>> >
>> >
>> > On Fri, Apr 19, 2013 at 9:05 AM, Matthew Turk <matthewturk at gmail.com>
>> > wrote:
>> >>
>> >> On Fri, Apr 19, 2013 at 11:02 AM, Stephen Skory <s at skory.us> wrote:
>> >> > Just to play devil's advocate here, what would it take to "fix"
>> >> > serialization? Adding a checksum so that data changes can be
>> >> > detected?
>> >> > More than that?
>> >>
>> >> That would partly fix the issue of fields/hierarchy changes not being
>> >> detected (unless you manually mess with the data).  But it wouldn't
>> >> fix the deeper problem, which is that a) we scatter files willy nilly
>> >> about the directory, which I am coming to feel is a really gross
>> >> violation of expectations, and b) the process of auto-serialization
>> >> doesn't save a huge amount of time in most cases.  JohnW and I
>> >> spitballed last fall about some of the biggest hierarchies he's dealt
>> >> with in Enzo and I promised to write a Cython parser, which I never
>> >> succeeded at.
>> >>
>> >> -Matt
>> >>
>> >> >
>> >> >
>> >> >
>> >> > --
>> >> > Stephen Skory
>> >> > s at skory.us
>> >> > http://stephenskory.com/
>> >> > 510.621.3687 (google voice)
>> >> > _______________________________________________
>> >> > yt-dev mailing list
>> >> > yt-dev at lists.spacepope.org
>> >> > http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>> >> _______________________________________________
>> >> yt-dev mailing list
>> >> yt-dev at lists.spacepope.org
>> >> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>> >
>> >
>> >
>> >
>> > --
>> > Sent from my computer.
>> >
>> > _______________________________________________
>> > yt-dev mailing list
>> > yt-dev at lists.spacepope.org
>> > http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>> >
>> _______________________________________________
>> yt-dev mailing list
>> yt-dev at lists.spacepope.org
>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>
>
>
>
> --
> Sent from my computer.
>
> _______________________________________________
> yt-dev mailing list
> yt-dev at lists.spacepope.org
> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>