[yt-users] yt memory usage

[Matthew Turk]

Hi Stella,

4500^3 ... definitely not optimal!  I chatted with Jeff and got a copy
of the code you guys worked on, and I've made some modifications (in
consultation with Jeff) that I think will help you out.  I've posted
it here:

http://paste.enzotools.org/show/89/

(you can download the paste either with "yt_lodgeit.py --download=89"
or "wget http://paste.enzotools.org/raw/89/" but wget might add a few
trailing characters)

It's a bit verbose in the comments, because I wanted to justify most
of what it's doing.  I ran it on one of our datasets and it was fairly
memory-conservative.  What I've done is to figure out which grids we
need to consider; then for each of those (iterated over from
highest-to-lowest resolution), we generate a covering grid with a
single ghost zone (and you can decide whether you want the ghost zones
to be smoothed, with the parameter 'smoothed' at the top of the file,
but I don't think it will matter that much) and then we use that as
the basis for the interpolator.  Then we interpolate to those points
in our desired output that are within the grid (not the covering grid)
and that haven't been handled yet, and mark them as done.  You're left
then with the regridded output.

Unfortunately, I don't actually have a convenient way to plot the
resultant data, so there may be a bug in the final output that I'm
missing.  I've taken a look at the values and they seem reasonable,
but I have to ponder about verification of the results; one option
would be to see how well it agrees with interpolation along
non-orthogonal rays cast along every phi&theta pair.  I think this is
an interesting topic, and what I'd like to do is move this routine
into yt proper -- have, say, a regridder that accepts desire x,y,z and
returns interpolated values.  So if this particular code works for
you, let me know, and we can then integrate it back into the main
codebase.

I hope that helps, but let us know either way!

-Matt

On Fri, Mar 27, 2009 at 10:42 AM, Stella Offner <soffner at berkeley.edu> wrote:
> Hi,
>
> I've been working on setting up yt code to interpolate from an AMR dataset
> to a spherical grid of points. Thanks to help from Jeff, I've used a
> covering grid to extract the data and then TrilinearFieldInterpolator to
> derive the values on the grid of points. The problem is that the covering
> grid allocates the entire data array on the maximum level ~4500^3, which
> doesn't fit into the memory on the Nersc Davinci machine (72 gb max).  I
> really only need the cells adaject to points in the output grid, which is <
> 150^3. Is there a more efficient way of using covering_grid or extracting
> the AMR data that doesn't involve first interpolating the entire region to
> the maximum level? (The code definitely works for smaller grids/lower max
> levels, but I really need to get at the high resolution AMR data).
>
> Thanks for your help!
> Stella
>
> _______________________________________________
> yt-users mailing list
> yt-users at lists.spacepope.org
> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>