[yt-users] getting more information about memory usage

[Matthew Turk]

Hi Nick,

On Wed, Nov 6, 2013 at 5:40 AM, nick moeckel <nickolas1 at gmail.com> wrote:
> Hi there,
>
> I'm trying to analyse some data with a 1024^3 base grid, and it's pushing at
> the memory limits of the machine where the data resides. So two quick
> questions:
>
> should I expect parallel analysis to have a reduced memory footprint? E.g.
> if I need 16Gb of memory to run this analysis serially, will running on 16
> cores get me anywhere near 1Gb/core?

Sort of.

With the current RAMSES front end, here's how the memory is set up and
works.  (I'll also mention below how this can be improved.)  As it
currently stands, when the RAMSES output is parsed, a full octree is
created on each processor.  Each oct consists of 3 64-bit integers and
an allocatable pointer.  (256 bits)  When allocated (i.e., non-leaf)
the pointer is 8, which adds on an additional 448 bits.

When actually doing analysis, the octree is still replicated across
all nodes, *but*, the fields that occupy that octree will not be.  So,
you won't be able to eliminate the overhead, but you will be able to
reduce the additional cost of a given field.

The next iteration should probably involve both index-on-demand (which
is how the ARTIO frontend works, and how the N-body frontends are
being transitioned) and "pinning" octree chunks to individual cores.
Since this is a useful application area for you, I could provide some
suggestions for areas to dig in, or work with you to experiment with
ways of making this happen sooner rather than later, but I'm not sure
I'd be able to dedicate full-time energy to it before the end of the
year.  It is definitely a weak spot and something we want to get
fixed, but there are some other barriers to the 3.0 release that need
to be addressed first.

>
> does yt have a more verbose debug mode or something that I can activate to
> output any details about memory consumption that might aid me in the
> delicate and adversarial dance that this cluster and I are engaged in?
>

Yup.  There are a few handy functions:

get_memory_usage()

which will return in MB, and there's also the contextmanager
memory_checker.  This sets up a periodic interval over which memory
usage is reported, defaulting to every 15 seconds.  You'd do this
like:

with memory_checker():
    do_something()
    another_time()

and then when it leaves the indented block, the memory checker terminates.

Hope that helps,

-Matt

> thanks,
> Nick
>
> _______________________________________________
> yt-users mailing list
> yt-users at lists.spacepope.org
> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>