[yt-users] add_field or grids.get_data different in yt-3.0 for FLASH data?

[Matthew Turk]

Hi James,

Sorry for the delay in replying.  From what I've checked in yt-2.x, this:

g.get_data(field_name)

is the same as:

g[field_name]

In yt-3.0, we've changed it such that .get_data() no longer returns an
array but instead operates as a mechanism for filling in the arrays in a
grid; this is because it can accept multiple fields (and does a much more
consistent batching of IO) and so the return value is somewhat undefined.

You can and should be able to use this interchangeably with what you've
done before, but I've spent a bit of time just now looking at how the FLASH
frontend works in 3.0 and I think we can provide some speedup.  This
utilizes some non-public APIs, but it should be stable.  It requires some
changes I'm issuing a pull request for (I am typing this while I do not
have WiFi) but they are a part of changesets b0a426d61bca and 38d2692e328d.

Here's the code, which will preload some fields and then activate the
cache.  yt does this itself on the backend when generating spatial fields.
 By default this will only load 256 grids at a time in advance, however, so
there is likely room to tweak for improvements.

from yt.mods import *
pf = load("GasSloshing/sloshing_nomag2_hdf5_plt_cnt_0150")
#pf = load("IsolatedGalaxy/galaxy0030/galaxy0030")

fields = ["dens", "temp"]
#fields = ["Density", "Temperature"]

dd = pf.h.all_data()
preload_fields = dd._determine_fields(fields)
for chunk in dd.chunks([], "spatial", preload_fields = preload_fields):
    g = chunk._current_chunk.objs[0]
    with g._activate_cache():
        g[fields[0]]

(The Enzo script is still included, as comparison.)  When clearing data in
between runs, I get best-of-10 runtimes (for FLASH) of roughly 0.8 seconds
with preloading and 20 seconds without.  For Enzo, I actually see
performance *degradation*, with 1.6 seconds with preloading and 0.36
without.  However, the Enzo total zones read in was only 83208320 versus
155484160 for the FLASH simulation, so the runtime overhead likely
dominates.  I suspect in a larger Enzo simulation, the times would be more
comparable.

-Matt

PS Funnily enough, I didn't notice it before, but we're now at 10100
changesets in my personal yt-3.0 repository.  Welcome to five digits,
everybody!



On Thursday, November 14, 2013, James Guillochon wrote:

> Hi, I have a histogram routine I wrote to process FLASH data. The routine
> works in yt 2.x, but gives me an error when call get_data from a grids
> object. Here's the snippet that fails:
>
>     for cnt, g in enumerate(pf.h.grids):
>>         if g.Level > pf.h.max_level - args.undersample: continue
>>         if len(g.Children) != 0 and g.Level != pf.h.max_level -
>> args.undersample: continue
>>         evals = list()
>>         vvals = list()
>> #vvals = g.get_data(args.var).ravel()
>>         for e, ev in enumerate(args.vars):
>>             vvals.append(g.get_data(ev).ravel())
>
>
> ...and here's the error:
>
> vvals.append(g.get_data(ev).ravel())
>> AttributeError: 'NoneType' object has no attribute 'ravel'
>
>
> I initialized the fields using a loop in the following way:
>
> myvars = []
>> for e, ev in enumerate(list(set(args.vars) | set(args.excludevars))):
>>     myvars += [var(ev)]
>>     add_field(ev, function=myvars[e].mesh, take_log=args.log)
>
>
> I understand that how fields behave has changed somewhat in 3.0. Any idea
> what could be causing the issue?
>
> The whole routine is here: http://goo.gl/R7FmsK
>
> Thanks all!
>
> --
> James Guillochon
> Einstein Fellow at the Harvard-Smithsonian CfA
> jguillochon at cfa.harvard.edu <javascript:_e({}, 'cvml',
> 'jguillochon at cfa.harvard.edu');>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.spacepope.org/pipermail/yt-users-spacepope.org/attachments/20131116/fa81d879/attachment.html>