[yt-users] cloud in cell mesh construction for particle data

[Brendan Griffen]

Thanks very much for your help. I only used ~64GB of memory for 1024^3
particles on a 512^3 mesh (using covering_grid). This essentially cut
memory usage in half by my guess.


[image: Inline image 1]
(I know this could be made to look better!, vmin,vmax etc.)

BG


On Wed, Jun 11, 2014 at 10:56 AM, Matthew Turk <matthewturk at gmail.com>
wrote:

> Hi Brendan,
>
> (Snipped most of the rest of the message.)
>
> Alright, I've issued a pull request that wraps in a few changes that
> for me dropped memory usage considerably.
>
>
> https://bitbucket.org/yt_analysis/yt/pull-request/945/reduce-memory-usage-for-particle-octrees
>
> There are other things that need to be done; the most prominent is
> that we absolutely need to figure out how to substride on octrees.
> Basically, what we're constrained by is that the way the derived
> fields work, the size of the data chunk being operated on in spatial
> mode (i.e., Nzones) is the fundamental size by which we allocate
> memory during the processing of a field.  So if we need five fields to
> generate the sixth one, we have to have those Nfields resident in
> memory during the generation.  As it stands, the octree system is not
> flexible enough to do substrides on these, which means we have  pages
> of that size getting allocated, which is where the big increase in
> memory during the computation of "index","cell_volume" takes place.
>
> Fixing this overlaps with moving to forest-of-octrees, so I am going
> to try to up the priority of that.
>
> -Matt
>
> On Tue, Jun 10, 2014 at 3:00 PM, Brendan Griffen
> <brendan.f.griffen at gmail.com> wrote:
> > Thanks very much Matt! I've improved things at my end as well with my
> script
> > (non-yt related). Hopefully the confluence of the two changes will
> enabled a
> > successful run.
> >
> > Thanks.
> >
> > Brendan
> >
> >
> > On Tue, Jun 10, 2014 at 3:57 PM, Matthew Turk <matthewturk at gmail.com>
> wrote:
> >>
> >> Hi Brendan,
> >>
> >> I think I've found the problem -- volume is making too many copies.
> >> I'm working on a fix.
> >>
> _______________________________________________
> yt-users mailing list
> yt-users at lists.spacepope.org
> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.spacepope.org/pipermail/yt-users-spacepope.org/attachments/20140611/d4216f00/attachment.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: dim512-z0.000.png
Type: image/png
Size: 506504 bytes
Desc: not available
URL: <http://lists.spacepope.org/pipermail/yt-users-spacepope.org/attachments/20140611/d4216f00/attachment.png>