Hi James,<div><br></div><div>Sorry for the delay in replying. From what I've checked in yt-2.x, this:</div><div><br></div><div>g.get_data(field_name)</div><div><br></div><div>is the same as:</div><div><br></div><div>g[field_name]</div>
<div><br></div><div>In yt-3.0, we've changed it such that .get_data() no longer returns an array but instead operates as a mechanism for filling in the arrays in a grid; this is because it can accept multiple fields (and does a much more consistent batching of IO) and so the return value is somewhat undefined.</div>
<div><br></div><div>You can and should be able to use this interchangeably with what you've done before, but I've spent a bit of time just now looking at how the FLASH frontend works in 3.0 and I think we can provide some speedup. This utilizes some non-public APIs, but it should be stable. It requires some changes I'm issuing a pull request for (I am typing this while I do not have WiFi) but they are a part of changesets b0a426d61bca and 38d2692e328d.</div>
<div><br></div><div>Here's the code, which will preload some fields and then activate the cache. yt does this itself on the backend when generating spatial fields. By default this will only load 256 grids at a time in advance, however, so there is likely room to tweak for improvements.</div>
<div><br></div><div><div>from yt.mods import *</div><div>pf = load("GasSloshing/sloshing_nomag2_hdf5_plt_cnt_0150")</div><div>#pf = load("IsolatedGalaxy/galaxy0030/galaxy0030")</div><div><br></div><div>
fields = ["dens", "temp"]</div><div>#fields = ["Density", "Temperature"]</div><div><br></div><div>dd = pf.h.all_data()</div><div>preload_fields = dd._determine_fields(fields)</div><div>
for chunk in dd.chunks([], "spatial", preload_fields = preload_fields):</div><div> g = chunk._current_chunk.objs[0]</div><div> with g._activate_cache():</div><div> g[fields[0]]</div></div><div><br></div>
<div>(The Enzo script is still included, as comparison.) When clearing data in between runs, I get best-of-10 runtimes (for FLASH) of roughly 0.8 seconds with preloading and 20 seconds without. For Enzo, I actually see performance *degradation*, with 1.6 seconds with preloading and 0.36 without. However, the Enzo total zones read in was only 83208320 versus 155484160 for the FLASH simulation, so the runtime overhead likely dominates. I suspect in a larger Enzo simulation, the times would be more comparable.</div>
<div><br></div><div>-Matt</div><div><br></div><div>PS Funnily enough, I didn't notice it before, but we're now at 10100 changesets in my personal yt-3.0 repository. Welcome to five digits, everybody!</div><div><br>
</div><div><br></div><div><br>On Thursday, November 14, 2013, James Guillochon wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Hi, I have a histogram routine I wrote to process FLASH data. The routine works in yt 2.x, but gives me an error when call get_data from a grids object. Here's the snippet that fails:<div>
<br></div><div>
<div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><font face="courier new, monospace"> for cnt, g in enumerate(pf.h.grids):<br>
if g.Level > pf.h.max_level - args.undersample: continue<br> if len(g.Children) != 0 and g.Level != pf.h.max_level - args.undersample: continue<br> evals = list()<br> vvals = list()<br>#vvals = g.get_data(args.var).ravel()<br>
for e, ev in enumerate(args.vars):<br> vvals.append(g.get_data(ev).ravel())</font></blockquote></div><div><br></div><div>...and here's the error:</div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<font face="courier new, monospace">vvals.append(g.get_data(ev).ravel())<br></font><font face="courier new, monospace">AttributeError: 'NoneType' object has no attribute 'ravel'</font></blockquote><div><br>
</div><div>I initialized the fields using a loop in the following way:</div><div><br></div><div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<font face="courier new, monospace">myvars = []<br>for e, ev in enumerate(list(set(args.vars) | set(args.excludevars))):<br> myvars += [var(ev)]<br> add_field(ev, function=myvars[e].mesh, take_log=args.log)</font></blockquote>
<div><br></div><div>I understand that how fields behave has changed somewhat in 3.0. Any idea what could be causing the issue?</div></div><div><br></div><div>The whole routine is here: <a href="http://goo.gl/R7FmsK" target="_blank">http://goo.gl/R7FmsK</a></div>
<div><br></div><div>Thanks all!</div><div><br></div>-- <br><div dir="ltr">James Guillochon<br>Einstein Fellow at the Harvard-Smithsonian CfA<br><a href="javascript:_e({}, 'cvml', 'jguillochon@cfa.harvard.edu');" target="_blank">jguillochon@cfa.harvard.edu</a><br>
</div>
</div></div>
</blockquote></div>