[yt-users] cloud in cell mesh construction for particle data

Matthew Turk matthewturk at gmail.com
Wed Jun 11 07:56:21 PDT 2014


Hi Brendan,

(Snipped most of the rest of the message.)

Alright, I've issued a pull request that wraps in a few changes that
for me dropped memory usage considerably.

https://bitbucket.org/yt_analysis/yt/pull-request/945/reduce-memory-usage-for-particle-octrees

There are other things that need to be done; the most prominent is
that we absolutely need to figure out how to substride on octrees.
Basically, what we're constrained by is that the way the derived
fields work, the size of the data chunk being operated on in spatial
mode (i.e., Nzones) is the fundamental size by which we allocate
memory during the processing of a field.  So if we need five fields to
generate the sixth one, we have to have those Nfields resident in
memory during the generation.  As it stands, the octree system is not
flexible enough to do substrides on these, which means we have  pages
of that size getting allocated, which is where the big increase in
memory during the computation of "index","cell_volume" takes place.

Fixing this overlaps with moving to forest-of-octrees, so I am going
to try to up the priority of that.

-Matt

On Tue, Jun 10, 2014 at 3:00 PM, Brendan Griffen
<brendan.f.griffen at gmail.com> wrote:
> Thanks very much Matt! I've improved things at my end as well with my script
> (non-yt related). Hopefully the confluence of the two changes will enabled a
> successful run.
>
> Thanks.
>
> Brendan
>
>
> On Tue, Jun 10, 2014 at 3:57 PM, Matthew Turk <matthewturk at gmail.com> wrote:
>>
>> Hi Brendan,
>>
>> I think I've found the problem -- volume is making too many copies.
>> I'm working on a fix.
>>



More information about the yt-users mailing list