[yt-users] Caching and speed

[Andrew Myers]

Hi Mark,

In yt-2.X, there was the ability to save expensive fields
<http://yt.readthedocs.org/en/latest/analyzing/creating_derived_fields.html#saving-derived-fields>
to the disk as a sidecar file. You could then work with the field as if it
was a native field like "density". It would still redo the IO for each
projection I think, but with only 1 field. I'm not sure whether this
functionality made it to yt-3.0, but it's worth a try. If it doesn't work,
we can try to restore the capability.

-Andrew

On Mon, Nov 17, 2014 at 10:06 AM, Mark Krumholz <mkrumhol at ucsc.edu> wrote:

> Hi YT Users,
>
> I have a question about speed and caching that I’m hoping someone can help
> with.
>
> I have a fairly large simulation from which I’d like to make projection
> plots in all three cardinal directions and save the results in a flat numpy
> array for later incorporation into an animation. One of the quantities I’d
> like to project is density-weighted temperature, and this is fairly
> expensive to compute because temperature is a derived field whose
> construction requires reading 8 other fields from disk. The computation is
> that temperature = constant times internal energy, and internal energy =
> total energy (1 field) - magnetic energy (3 fields: Bx, By, and Bz) -
> kinetic energy (4 fields: density, px, py, pz). I want to do the projection
> over the entire simulation volume, so it seems like the sensible way to do
> the computation would be to read all the data once, construct the
> temperature, store it in memory, and then project it into the three
> cardinal directions. The size of the data set is not so large that it won’t
> fit in memory on the largest memory nodes to which I have access. However,
> what yt seems to be doing instead is to read the data and compute the
> temperature for the first projection, and then repeat the entire process,
> including the reading and computation, for each of the other two
> directions. This makes the computation a factor of 3 slower than it should
> be (since IO dominates the cost), and, given the size of the data sets
> involved, this is a significant annoyance.
>
> So here’s the question: is there a way to force yt to cache a derived
> field rather than reconstructing it on the fly every time it is needed? Or
> is there some other strategy that can be used to avoid this inefficiency?
> There’s and sample script pasted at https://bpaste.net/show/a58f7ebea9bd that
> demonstrates what I’m trying to do.
>
> Thanks.
>
> --
> Mark Krumholz
>
>
> _______________________________________________
> yt-users mailing list
> yt-users at lists.spacepope.org
> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.spacepope.org/pipermail/yt-users-spacepope.org/attachments/20141117/bf4a581d/attachment.html>