<div dir="ltr">Can we do it yesterday???!?</div><div class="gmail_extra"><br><br><div class="gmail_quote">On Fri, Apr 19, 2013 at 2:02 PM, Cameron Hummels <span dir="ltr"><<a href="mailto:chummels@gmail.com" target="_blank">chummels@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">I am not surprised--Kraken is the worst cluster on the planet.</div><div class="gmail_extra"><div><div class="h5">
<br><br><div class="gmail_quote">On Fri, Apr 19, 2013 at 10:26 AM, Matthew Turk <span dir="ltr"><<a href="mailto:matthewturk@gmail.com" target="_blank">matthewturk@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div>On Fri, Apr 19, 2013 at 12:01 PM, david collins <<a href="mailto:antpuncher@gmail.com" target="_blank">antpuncher@gmail.com</a>> wrote:<br>
> I just did a test. 512^3 with 4 levels everywhere. On my desktop, it took<br>
> 78 seconds. On Kraken it took 19 minutes. Unsurprisingly, Kraken's disk is<br>
> not that fast today. This is, of course, and extreme example, but it isn't<br>
> by any means rare for that system. 78 seconds isn't too bad, but today I'll<br>
> be looking at >20 datasets, so it adds up.<br>
<br>
</div>I understand, 78 seconds will definitely add up. I'm impressed by the<br>
difference between Kraken and local. Or maybe flabberghasted? Either<br>
way, it's definitely something.<br>
<div><br>
><br>
> I definitely am in favor of an opt-in system, though, as many applications<br>
> won't be this shape or on this disk. Or even an explicit call, such as<br>
> ProjectinPlot.serialize()<br>
<br>
</div>Okay, cool. I think that'd be a good one, like Britton mentioned too.<br>
<div><div><br>
><br>
><br>
> pf = load(fname)<br>
> t0=time.time()<br>
> proj=ProjectionPlot(pf,2,'Density')<br>
> t1=time.time()<br>
> print t1-t0<br>
> Parsing Hierarchy100%<br>
> ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||<br>
> Time: 00:00:00<br>
> Initializing tree 0 / 4100%<br>
> ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||<br>
> Time: 00:00:00<br>
> Projecting level 0 / 4 100%<br>
> |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||<br>
> Time: 00:03:39<br>
> Initializing tree 1 / 4100%<br>
> ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||<br>
> Time: 00:00:00<br>
> Projecting level 1 / 4 100%<br>
> |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||<br>
> Time: 00:04:12<br>
> Initializing tree 2 / 4100%<br>
> ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||<br>
> Time: 00:00:00<br>
> Projecting level 2 / 4 80%<br>
> |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||<br>
> | ETA: 00:00:52<br>
> Projecting level 2 / 4 100%<br>
> |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||<br>
> Time: 00:04:17<br>
> Initializing tree 3 / 4100%<br>
> ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||<br>
> Time: 00:00:00<br>
> Projecting level 3 / 4 100%<br>
> |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||<br>
> Time: 00:02:58<br>
> Initializing tree 4 / 4100%<br>
> ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||<br>
> Time: 00:00:00<br>
> Projecting level 4 / 4 100%<br>
> |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||<br>
> Time: 00:04:06<br>
> 1167.0001719<br>
><br>
><br>
><br>
><br>
> On Fri, Apr 19, 2013 at 9:18 AM, Matthew Turk <<a href="mailto:matthewturk@gmail.com" target="_blank">matthewturk@gmail.com</a>> wrote:<br>
>><br>
>> Dave, out of curiosity, how long does a projection of one of your datasets<br>
>> take?<br>
>><br>
>> On Fri, Apr 19, 2013 at 11:17 AM, david collins <<a href="mailto:antpuncher@gmail.com" target="_blank">antpuncher@gmail.com</a>><br>
>> wrote:<br>
>> ><br>
>> > I use serialized projections quite frequently. As long as the current<br>
>> > behavior, or some reasonable facsimile, is still available I'm for doing<br>
>> > it<br>
>> > now.<br>
>> ><br>
>> ><br>
>> > On Fri, Apr 19, 2013 at 9:05 AM, Matthew Turk <<a href="mailto:matthewturk@gmail.com" target="_blank">matthewturk@gmail.com</a>><br>
>> > wrote:<br>
>> >><br>
>> >> On Fri, Apr 19, 2013 at 11:02 AM, Stephen Skory <<a href="mailto:s@skory.us" target="_blank">s@skory.us</a>> wrote:<br>
>> >> > Just to play devil's advocate here, what would it take to "fix"<br>
>> >> > serialization? Adding a checksum so that data changes can be<br>
>> >> > detected?<br>
>> >> > More than that?<br>
>> >><br>
>> >> That would partly fix the issue of fields/hierarchy changes not being<br>
>> >> detected (unless you manually mess with the data). But it wouldn't<br>
>> >> fix the deeper problem, which is that a) we scatter files willy nilly<br>
>> >> about the directory, which I am coming to feel is a really gross<br>
>> >> violation of expectations, and b) the process of auto-serialization<br>
>> >> doesn't save a huge amount of time in most cases. JohnW and I<br>
>> >> spitballed last fall about some of the biggest hierarchies he's dealt<br>
>> >> with in Enzo and I promised to write a Cython parser, which I never<br>
>> >> succeeded at.<br>
>> >><br>
>> >> -Matt<br>
>> >><br>
>> >> ><br>
>> >> ><br>
>> >> ><br>
>> >> > --<br>
>> >> > Stephen Skory<br>
>> >> > <a href="mailto:s@skory.us" target="_blank">s@skory.us</a><br>
>> >> > <a href="http://stephenskory.com/" target="_blank">http://stephenskory.com/</a><br>
>> >> > <a href="tel:510.621.3687" value="+15106213687" target="_blank">510.621.3687</a> (google voice)<br>
>> >> > _______________________________________________<br>
>> >> > yt-dev mailing list<br>
>> >> > <a href="mailto:yt-dev@lists.spacepope.org" target="_blank">yt-dev@lists.spacepope.org</a><br>
>> >> > <a href="http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org" target="_blank">http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org</a><br>
>> >> _______________________________________________<br>
>> >> yt-dev mailing list<br>
>> >> <a href="mailto:yt-dev@lists.spacepope.org" target="_blank">yt-dev@lists.spacepope.org</a><br>
>> >> <a href="http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org" target="_blank">http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org</a><br>
>> ><br>
>> ><br>
>> ><br>
>> ><br>
>> > --<br>
>> > Sent from my computer.<br>
>> ><br>
>> > _______________________________________________<br>
>> > yt-dev mailing list<br>
>> > <a href="mailto:yt-dev@lists.spacepope.org" target="_blank">yt-dev@lists.spacepope.org</a><br>
>> > <a href="http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org" target="_blank">http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org</a><br>
>> ><br>
>> _______________________________________________<br>
>> yt-dev mailing list<br>
>> <a href="mailto:yt-dev@lists.spacepope.org" target="_blank">yt-dev@lists.spacepope.org</a><br>
>> <a href="http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org" target="_blank">http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org</a><br>
><br>
><br>
><br>
><br>
> --<br>
> Sent from my computer.<br>
><br>
> _______________________________________________<br>
> yt-dev mailing list<br>
> <a href="mailto:yt-dev@lists.spacepope.org" target="_blank">yt-dev@lists.spacepope.org</a><br>
> <a href="http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org" target="_blank">http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org</a><br>
><br>
_______________________________________________<br>
yt-dev mailing list<br>
<a href="mailto:yt-dev@lists.spacepope.org" target="_blank">yt-dev@lists.spacepope.org</a><br>
<a href="http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org" target="_blank">http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org</a><br>
</div></div></blockquote></div><br><br clear="all"><div><br></div></div></div><span class="HOEnZb"><font color="#888888">-- <br>Cameron Hummels<div>Postdoctoral Researcher</div><div>Steward Observatory</div><div>University of Arizona</div>
<div><a href="http://chummels.org" target="_blank">http://chummels.org</a></div>
</font></span></div>
<br>_______________________________________________<br>
yt-dev mailing list<br>
<a href="mailto:yt-dev@lists.spacepope.org">yt-dev@lists.spacepope.org</a><br>
<a href="http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org" target="_blank">http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org</a><br>
<br></blockquote></div><br></div>