[yt-dev] memory usage with --parallel and periodic radius

[Michael Kuhlen]

> With the last good changeset the total memory usage never gets above
~8.5GB (estimated from top).

Sorry, forgot to say: the last good changeset is 41358eecdad5.


On Wed, Feb 27, 2013 at 7:58 PM, Michael Kuhlen <mqk at astro.berkeley.edu>wrote:

> Hi all and Nathan specifically
>
> I've found that the changes having to do with periodic fields are causing
> excessive memory usage for me when doing parallel halo profiling, to the
> point where I can no longer do analysis that used to work fine on my 24GB
> memory workstation.
>
> A simple HaloProfiler script (http://paste.yt-project.org/show/3211/),
> run in parallel on 8 processors on the halo catalog obtained from one of my
> cosmology simulations, currently almost immediate runs out of memory, while
> it used to complete without problems.
>
> I tried to use hg bisect to find the problematic changeset but had to skip
> the testing of several revisions because of runtime yt errors, so in the
> end bisect only gave me a range of potentially bad revisions:
> http://paste.yt-project.org/show/3210/. As you can see, they're all
> related to the periodic radius mods from about a month ago.
>
> I tried replicating this with the standard Enzo_64 example dataset from
> http://yt-project.org/data/, but I guess that one is too small to produce
> this problem on my machine. If you want to see the problem for yourself,
> then you can download this tarball (
> http://astro.berkeley.edu/~mqk/transfer/RD0003.tar 4.5GB) and run this
> script (http://paste.yt-project.org/show/3211/) on it, like so:
>
> $ mpirun -np 8 python ./profile_halos.py --parallel
>
> When I run this with the current tip (ccfe34e70803) the total memory usage
> grows to >24GB. With the last good changeset the total memory usage never
> gets above ~8.5GB (estimated from top).
>
> It'd be great if we could find out what the problem is and fix it, because
> this is a major performance regression for me, that, as I already said, is
> seriously impacting my ability to do analysis with the current yt tip. Let
> me know if I should file a BB issue about this, and/or if there's some way
> I can assist with fixing this.
>
> Cheers,
> Mike
>
> --
> *********************************************************************
> *                                                                   *
> *  Dr. Michael Kuhlen              Theoretical Astrophysics Center  *
> *  email: mqk at astro.berkeley.edu   UC Berkeley                      *
> *  cell phone: (831) 588-1468      B-116 Hearst Field Annex # 3411  *
> *  skype username: mikekuhlen      Berkeley, CA 94720               *
> *                                                                   *
> *********************************************************************
>



-- 
*********************************************************************
*                                                                   *
*  Dr. Michael Kuhlen              Theoretical Astrophysics Center  *
*  email: mqk at astro.berkeley.edu   UC Berkeley                      *
*  cell phone: (831) 588-1468      B-116 Hearst Field Annex # 3411  *
*  skype username: mikekuhlen      Berkeley, CA 94720               *
*                                                                   *
*********************************************************************
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.spacepope.org/pipermail/yt-dev-spacepope.org/attachments/20130227/d592a2b5/attachment.html>