[yt-users] Caching and speed

[Mark Krumholz]

Hi YT Users,

I have a question about speed and caching that I’m hoping someone can help with.

I have a fairly large simulation from which I’d like to make projection plots in all three cardinal directions and save the results in a flat numpy array for later incorporation into an animation. One of the quantities I’d like to project is density-weighted temperature, and this is fairly expensive to compute because temperature is a derived field whose construction requires reading 8 other fields from disk. The computation is that temperature = constant times internal energy, and internal energy = total energy (1 field) - magnetic energy (3 fields: Bx, By, and Bz) - kinetic energy (4 fields: density, px, py, pz). I want to do the projection over the entire simulation volume, so it seems like the sensible way to do the computation would be to read all the data once, construct the temperature, store it in memory, and then project it into the three cardinal directions. The size of the data set is not so large that it won’t fit in memory on the largest memory nodes to which I have access. However, what yt seems to be doing instead is to read the data and compute the temperature for the first projection, and then repeat the entire process, including the reading and computation, for each of the other two directions. This makes the computation a factor of 3 slower than it should be (since IO dominates the cost), and, given the size of the data sets involved, this is a significant annoyance.

So here’s the question: is there a way to force yt to cache a derived field rather than reconstructing it on the fly every time it is needed? Or is there some other strategy that can be used to avoid this inefficiency? There’s and sample script pasted at https://bpaste.net/show/a58f7ebea9bd <https://bpaste.net/show/a58f7ebea9bd> that demonstrates what I’m trying to do.

Thanks.

--
Mark Krumholz

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.spacepope.org/pipermail/yt-users-spacepope.org/attachments/20141117/5ce32130/attachment.htm>