[yt-dev] When can we turn off serialization by default?

[david collins]

I just did a test.  512^3 with 4 levels everywhere.  On my desktop, it took
78 seconds.  On Kraken it took 19 minutes. Unsurprisingly, Kraken's disk is
not that fast today.  This is, of course, and extreme example, but it isn't
by any means rare for that system.  78 seconds isn't too bad, but today
I'll be looking at >20 datasets, so it adds up.

I definitely am in favor of an opt-in system, though, as many applications
won't be this shape or on this disk.  Or even an explicit call, such as
ProjectinPlot.serialize()


pf = load(fname)
t0=time.time()
proj=ProjectionPlot(pf,2,'Density')
t1=time.time()
print t1-t0
Parsing Hierarchy100%
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Time: 00:00:00
Initializing tree  0 /  4100%
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Time: 00:00:00
Projecting  level  0 /  4 100%
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Time: 00:03:39
Initializing tree  1 /  4100%
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Time: 00:00:00
Projecting  level  1 /  4 100%
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Time: 00:04:12
Initializing tree  2 /  4100%
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Time: 00:00:00
Projecting  level  2 /  4  80%
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
                    | ETA:  00:00:52
Projecting  level  2 /  4 100%
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Time: 00:04:17
Initializing tree  3 /  4100%
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Time: 00:00:00
Projecting  level  3 /  4 100%
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Time: 00:02:58
Initializing tree  4 /  4100%
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Time: 00:00:00
Projecting  level  4 /  4 100%
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Time: 00:04:06
1167.0001719




On Fri, Apr 19, 2013 at 9:18 AM, Matthew Turk <matthewturk at gmail.com> wrote:

> Dave, out of curiosity, how long does a projection of one of your datasets
> take?
>
> On Fri, Apr 19, 2013 at 11:17 AM, david collins <antpuncher at gmail.com>
> wrote:
> >
> > I use serialized projections quite frequently.  As long as the current
> > behavior, or some reasonable facsimile, is still available I'm for doing
> it
> > now.
> >
> >
> > On Fri, Apr 19, 2013 at 9:05 AM, Matthew Turk <matthewturk at gmail.com>
> wrote:
> >>
> >> On Fri, Apr 19, 2013 at 11:02 AM, Stephen Skory <s at skory.us> wrote:
> >> > Just to play devil's advocate here, what would it take to "fix"
> >> > serialization? Adding a checksum so that data changes can be detected?
> >> > More than that?
> >>
> >> That would partly fix the issue of fields/hierarchy changes not being
> >> detected (unless you manually mess with the data).  But it wouldn't
> >> fix the deeper problem, which is that a) we scatter files willy nilly
> >> about the directory, which I am coming to feel is a really gross
> >> violation of expectations, and b) the process of auto-serialization
> >> doesn't save a huge amount of time in most cases.  JohnW and I
> >> spitballed last fall about some of the biggest hierarchies he's dealt
> >> with in Enzo and I promised to write a Cython parser, which I never
> >> succeeded at.
> >>
> >> -Matt
> >>
> >> >
> >> >
> >> >
> >> > --
> >> > Stephen Skory
> >> > s at skory.us
> >> > http://stephenskory.com/
> >> > 510.621.3687 (google voice)
> >> > _______________________________________________
> >> > yt-dev mailing list
> >> > yt-dev at lists.spacepope.org
> >> > http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
> >> _______________________________________________
> >> yt-dev mailing list
> >> yt-dev at lists.spacepope.org
> >> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
> >
> >
> >
> >
> > --
> > Sent from my computer.
> >
> > _______________________________________________
> > yt-dev mailing list
> > yt-dev at lists.spacepope.org
> > http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
> >
> _______________________________________________
> yt-dev mailing list
> yt-dev at lists.spacepope.org
> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>



-- 
Sent from my computer.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.spacepope.org/pipermail/yt-dev-spacepope.org/attachments/20130419/6b05c6dc/attachment.html>