[Yt-dev] quick question on particle IO

[Matthew Turk]

Hi Stephen,

On Wed, Oct 19, 2011 at 1:18 PM, Stephen Skory <s at skory.us> wrote:
> Hi Matt,
>
> I've gone through what you started and I fixed the bugs. This diff
> below produces identical results to the old way. Could you run this on
> your dataset you used and see if I haven't ballooned the memory? I was
> using a 40^3 dataset (it's fast!) but it's hard to see big changes in
> memory using that. If we like this, I'll remove all the prints, and
> apply it to my yt fork, and ask Geoffrey to try it out. If he sees no
> big problems, I'll do a pull request.
>
> http://paste.enzotools.org/show/1876/
>
>
>>  * Initializing ParallelHOPHaloFinder with copies (by dividing) of
>> position and mass fields
>
> I replaced those with na.multiply and na.divide, which should be doing
> it in-place on the arrays.
>
>>  * Copying position, mass into the fKD object without removing them
>> from the halo finder (is there any reason they can't simply be moved
>> in and removed from the halo finder, then copied back out?)
>
> No, it should be possible, and I did that in the new diff. I don't
> know if it's helped.
>
>>  * Rearrange = True is the default, which I believe copies the entire
>> kD-tree inside fKD?
>
> That's correct, it rearranges the position data so that the nearest
> neighbor searches are faster, and makes a copy to do this.
>
>> It also looks like some of this is because the fKD tree requires
>> position to be (N,3) for memory access speed.
>
> Yes, why I want a C/C++ kdtree someday that is either as fast at this
> fortran one, or nearly as fast. I have yet to find it.
>
>> Unfortunately, it's not entirely clear to me how we then copy back out
>> of the Forthon kD-tree at the right time; I don't quite now the inner
>> workings of pHOP.  But I think this is valid, as I don't believe at
>> any point information is lost.
>
> I think I've found the right place, just before the fKD object is
> deleted and the tree cleared.

Your patch drops my memory usage as well.  If you think this works, I
say go for it!

Thanks, Stephen!

-Matt

>
> --
> Stephen Skory
> s at skory.us
> http://stephenskory.com/
> 510.621.3687 (google voice)
> _______________________________________________
> Yt-dev mailing list
> Yt-dev at lists.spacepope.org
> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>