[yt-dev] When can we turn off serialization by default?
Cameron Hummels
chummels at gmail.com
Fri Apr 19 11:02:17 PDT 2013
I am not surprised--Kraken is the worst cluster on the planet.
On Fri, Apr 19, 2013 at 10:26 AM, Matthew Turk <matthewturk at gmail.com>wrote:
> On Fri, Apr 19, 2013 at 12:01 PM, david collins <antpuncher at gmail.com>
> wrote:
> > I just did a test. 512^3 with 4 levels everywhere. On my desktop, it
> took
> > 78 seconds. On Kraken it took 19 minutes. Unsurprisingly, Kraken's disk
> is
> > not that fast today. This is, of course, and extreme example, but it
> isn't
> > by any means rare for that system. 78 seconds isn't too bad, but today
> I'll
> > be looking at >20 datasets, so it adds up.
>
> I understand, 78 seconds will definitely add up. I'm impressed by the
> difference between Kraken and local. Or maybe flabberghasted? Either
> way, it's definitely something.
>
> >
> > I definitely am in favor of an opt-in system, though, as many
> applications
> > won't be this shape or on this disk. Or even an explicit call, such as
> > ProjectinPlot.serialize()
>
> Okay, cool. I think that'd be a good one, like Britton mentioned too.
>
> >
> >
> > pf = load(fname)
> > t0=time.time()
> > proj=ProjectionPlot(pf,2,'Density')
> > t1=time.time()
> > print t1-t0
> > Parsing Hierarchy100%
> >
> ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
> > Time: 00:00:00
> > Initializing tree 0 / 4100%
> >
> ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
> > Time: 00:00:00
> > Projecting level 0 / 4 100%
> >
> |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
> > Time: 00:03:39
> > Initializing tree 1 / 4100%
> >
> ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
> > Time: 00:00:00
> > Projecting level 1 / 4 100%
> >
> |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
> > Time: 00:04:12
> > Initializing tree 2 / 4100%
> >
> ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
> > Time: 00:00:00
> > Projecting level 2 / 4 80%
> >
> |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
> > | ETA: 00:00:52
> > Projecting level 2 / 4 100%
> >
> |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
> > Time: 00:04:17
> > Initializing tree 3 / 4100%
> >
> ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
> > Time: 00:00:00
> > Projecting level 3 / 4 100%
> >
> |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
> > Time: 00:02:58
> > Initializing tree 4 / 4100%
> >
> ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
> > Time: 00:00:00
> > Projecting level 4 / 4 100%
> >
> |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
> > Time: 00:04:06
> > 1167.0001719
> >
> >
> >
> >
> > On Fri, Apr 19, 2013 at 9:18 AM, Matthew Turk <matthewturk at gmail.com>
> wrote:
> >>
> >> Dave, out of curiosity, how long does a projection of one of your
> datasets
> >> take?
> >>
> >> On Fri, Apr 19, 2013 at 11:17 AM, david collins <antpuncher at gmail.com>
> >> wrote:
> >> >
> >> > I use serialized projections quite frequently. As long as the current
> >> > behavior, or some reasonable facsimile, is still available I'm for
> doing
> >> > it
> >> > now.
> >> >
> >> >
> >> > On Fri, Apr 19, 2013 at 9:05 AM, Matthew Turk <matthewturk at gmail.com>
> >> > wrote:
> >> >>
> >> >> On Fri, Apr 19, 2013 at 11:02 AM, Stephen Skory <s at skory.us> wrote:
> >> >> > Just to play devil's advocate here, what would it take to "fix"
> >> >> > serialization? Adding a checksum so that data changes can be
> >> >> > detected?
> >> >> > More than that?
> >> >>
> >> >> That would partly fix the issue of fields/hierarchy changes not being
> >> >> detected (unless you manually mess with the data). But it wouldn't
> >> >> fix the deeper problem, which is that a) we scatter files willy nilly
> >> >> about the directory, which I am coming to feel is a really gross
> >> >> violation of expectations, and b) the process of auto-serialization
> >> >> doesn't save a huge amount of time in most cases. JohnW and I
> >> >> spitballed last fall about some of the biggest hierarchies he's dealt
> >> >> with in Enzo and I promised to write a Cython parser, which I never
> >> >> succeeded at.
> >> >>
> >> >> -Matt
> >> >>
> >> >> >
> >> >> >
> >> >> >
> >> >> > --
> >> >> > Stephen Skory
> >> >> > s at skory.us
> >> >> > http://stephenskory.com/
> >> >> > 510.621.3687 (google voice)
> >> >> > _______________________________________________
> >> >> > yt-dev mailing list
> >> >> > yt-dev at lists.spacepope.org
> >> >> > http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
> >> >> _______________________________________________
> >> >> yt-dev mailing list
> >> >> yt-dev at lists.spacepope.org
> >> >> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
> >> >
> >> >
> >> >
> >> >
> >> > --
> >> > Sent from my computer.
> >> >
> >> > _______________________________________________
> >> > yt-dev mailing list
> >> > yt-dev at lists.spacepope.org
> >> > http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
> >> >
> >> _______________________________________________
> >> yt-dev mailing list
> >> yt-dev at lists.spacepope.org
> >> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
> >
> >
> >
> >
> > --
> > Sent from my computer.
> >
> > _______________________________________________
> > yt-dev mailing list
> > yt-dev at lists.spacepope.org
> > http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
> >
> _______________________________________________
> yt-dev mailing list
> yt-dev at lists.spacepope.org
> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>
--
Cameron Hummels
Postdoctoral Researcher
Steward Observatory
University of Arizona
http://chummels.org
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.spacepope.org/pipermail/yt-dev-spacepope.org/attachments/20130419/21d5850b/attachment.htm>
More information about the yt-dev
mailing list