[yt-dev] When can we turn off serialization by default?

j s oishi jsoishi at gmail.com
Tue Jun 18 11:51:12 PDT 2013 

Can we do it yesterday???!?


On Fri, Apr 19, 2013 at 2:02 PM, Cameron Hummels <chummels at gmail.com> wrote:

> I am not surprised--Kraken is the worst cluster on the planet.
>
>
> On Fri, Apr 19, 2013 at 10:26 AM, Matthew Turk <matthewturk at gmail.com>wrote:
>
>> On Fri, Apr 19, 2013 at 12:01 PM, david collins <antpuncher at gmail.com>
>> wrote:
>> > I just did a test.  512^3 with 4 levels everywhere.  On my desktop, it
>> took
>> > 78 seconds.  On Kraken it took 19 minutes. Unsurprisingly, Kraken's
>> disk is
>> > not that fast today.  This is, of course, and extreme example, but it
>> isn't
>> > by any means rare for that system.  78 seconds isn't too bad, but today
>> I'll
>> > be looking at >20 datasets, so it adds up.
>>
>> I understand, 78 seconds will definitely add up.  I'm impressed by the
>> difference between Kraken and local.  Or maybe flabberghasted?  Either
>> way, it's definitely something.
>>
>> >
>> > I definitely am in favor of an opt-in system, though, as many
>> applications
>> > won't be this shape or on this disk.  Or even an explicit call, such as
>> > ProjectinPlot.serialize()
>>
>> Okay, cool.  I think that'd be a good one, like Britton mentioned too.
>>
>> >
>> >
>> > pf = load(fname)
>> > t0=time.time()
>> > proj=ProjectionPlot(pf,2,'Density')
>> > t1=time.time()
>> > print t1-t0
>> > Parsing Hierarchy100%
>> >
>> ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
>> > Time: 00:00:00
>> > Initializing tree  0 /  4100%
>> >
>> ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
>> > Time: 00:00:00
>> > Projecting  level  0 /  4 100%
>> >
>> |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
>> > Time: 00:03:39
>> > Initializing tree  1 /  4100%
>> >
>> ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
>> > Time: 00:00:00
>> > Projecting  level  1 /  4 100%
>> >
>> |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
>> > Time: 00:04:12
>> > Initializing tree  2 /  4100%
>> >
>> ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
>> > Time: 00:00:00
>> > Projecting  level  2 /  4  80%
>> >
>> |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
>> > | ETA:  00:00:52
>> > Projecting  level  2 /  4 100%
>> >
>> |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
>> > Time: 00:04:17
>> > Initializing tree  3 /  4100%
>> >
>> ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
>> > Time: 00:00:00
>> > Projecting  level  3 /  4 100%
>> >
>> |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
>> > Time: 00:02:58
>> > Initializing tree  4 /  4100%
>> >
>> ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
>> > Time: 00:00:00
>> > Projecting  level  4 /  4 100%
>> >
>> |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
>> > Time: 00:04:06
>> > 1167.0001719
>> >
>> >
>> >
>> >
>> > On Fri, Apr 19, 2013 at 9:18 AM, Matthew Turk <matthewturk at gmail.com>
>> wrote:
>> >>
>> >> Dave, out of curiosity, how long does a projection of one of your
>> datasets
>> >> take?
>> >>
>> >> On Fri, Apr 19, 2013 at 11:17 AM, david collins <antpuncher at gmail.com>
>> >> wrote:
>> >> >
>> >> > I use serialized projections quite frequently.  As long as the
>> current
>> >> > behavior, or some reasonable facsimile, is still available I'm for
>> doing
>> >> > it
>> >> > now.
>> >> >
>> >> >
>> >> > On Fri, Apr 19, 2013 at 9:05 AM, Matthew Turk <matthewturk at gmail.com
>> >
>> >> > wrote:
>> >> >>
>> >> >> On Fri, Apr 19, 2013 at 11:02 AM, Stephen Skory <s at skory.us> wrote:
>> >> >> > Just to play devil's advocate here, what would it take to "fix"
>> >> >> > serialization? Adding a checksum so that data changes can be
>> >> >> > detected?
>> >> >> > More than that?
>> >> >>
>> >> >> That would partly fix the issue of fields/hierarchy changes not
>> being
>> >> >> detected (unless you manually mess with the data).  But it wouldn't
>> >> >> fix the deeper problem, which is that a) we scatter files willy
>> nilly
>> >> >> about the directory, which I am coming to feel is a really gross
>> >> >> violation of expectations, and b) the process of auto-serialization
>> >> >> doesn't save a huge amount of time in most cases.  JohnW and I
>> >> >> spitballed last fall about some of the biggest hierarchies he's
>> dealt
>> >> >> with in Enzo and I promised to write a Cython parser, which I never
>> >> >> succeeded at.
>> >> >>
>> >> >> -Matt
>> >> >>
>> >> >> >
>> >> >> >
>> >> >> >
>> >> >> > --
>> >> >> > Stephen Skory
>> >> >> > s at skory.us
>> >> >> > http://stephenskory.com/
>> >> >> > 510.621.3687 (google voice)
>> >> >> > _______________________________________________
>> >> >> > yt-dev mailing list
>> >> >> > yt-dev at lists.spacepope.org
>> >> >> > http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>> >> >> _______________________________________________
>> >> >> yt-dev mailing list
>> >> >> yt-dev at lists.spacepope.org
>> >> >> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>> >> >
>> >> >
>> >> >
>> >> >
>> >> > --
>> >> > Sent from my computer.
>> >> >
>> >> > _______________________________________________
>> >> > yt-dev mailing list
>> >> > yt-dev at lists.spacepope.org
>> >> > http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>> >> >
>> >> _______________________________________________
>> >> yt-dev mailing list
>> >> yt-dev at lists.spacepope.org
>> >> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>> >
>> >
>> >
>> >
>> > --
>> > Sent from my computer.
>> >
>> > _______________________________________________
>> > yt-dev mailing list
>> > yt-dev at lists.spacepope.org
>> > http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>> >
>> _______________________________________________
>> yt-dev mailing list
>> yt-dev at lists.spacepope.org
>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>>
>
>
>
> --
> Cameron Hummels
> Postdoctoral Researcher
> Steward Observatory
> University of Arizona
> http://chummels.org
>
> _______________________________________________
> yt-dev mailing list
> yt-dev at lists.spacepope.org
> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.spacepope.org/pipermail/yt-dev-spacepope.org/attachments/20130618/24c51cb5/attachment.htm>

More information about the yt-dev mailing list