[Yt-dev] Particles

[Matthew Turk]

Hi everyone,

I've pushed some changes to the memory-optimization branch in the
primary yt hg repo that include the first generation of this particle
handler.  No dataspace stuff yet, but it currently works for all the
rectangular prism objects and it speeds up the IO as well as reducing
the memory overhead.  (Factors of 2 and 3, respectively, in my simple
tests.)  Likely this is due to fragmentation issues and problems with
releasing back to the OS, but whatever the cause, the new method is
faster and cheaper.

These changes will be made more robust (and derived particle fields
will be added) next week and backported to the trunk.  Stephen's been
working on an amazing new parallel HOP implementation that will take
advantage of the particle IO changes.

-Matt

On Thu, Sep 10, 2009 at 12:15 PM, Matthew Turk <matthewturk at gmail.com> wrote:
> Hi guys,
>
> Thanks for the feedback.  I think we'll go with the example y'all
> liked.  I think we can get going on this in a couple weeks or a month
> at the outside, but it should be made a priority.  I also agree, this
> should not be exclusive for dataspace access, but instead that should
> just be the most optimized way of accessing the data.
>
> Thanks!
>
> -Matt
>
> PS I've stuck up a new front page at yt.enzotools.org.  If anybody has
> any images for the gallery, send them my way...
>
> On Thu, Sep 10, 2009 at 4:27 AM, John Wise <john.h.wise at nasa.gov> wrote:
>> Hi,
>> I like the first example the best because usually the user has some idea of
>> the volume they're interested in, such as looking for stars or tracer
>> particles in a particular halo or searching for all of the stars in the
>> simulation.
>> The dataspace framework is pretty cool because it removes the intensive need
>> of searching through the *entire* particle list for only (in most cases) a
>> handful of particles.  Of course because this feature is new, we'd still
>> have to support searching for certain particles if the dataspace doesn't
>> exist.
>> Cheers,
>> John
>> On 10 Sep 2009, at 00:11, Britton Smith wrote:
>>
>> I think something like the first example you gave would be the best.  It's
>> definitely more in keeping with the data_object[field] notation for getting
>> grid data.  In a very similar vein, I find it much more intuitive to think
>> of asking for the particles or grid data associated with a particular object
>> than say start with some larger structure for handling particles for the
>> entire pf, and give me only the ones in some sphere or other object (this is
>> my interpretation of the second example you gave).
>>
>> Additionally, I think particle data-space stuff you done in Enzo is great.
>> It seems silly to have to read in an entire set of particles only to filter
>> most of them back out, like with star particles.  This will also come in
>> handy for other people who are working with tracer particles.
>>
>> Britton
>>
>> On Wed, Sep 9, 2009 at 8:47 PM, Matthew Turk <matthewturk at gmail.com> wrote:
>>>
>>> Hi guys,
>>>
>>> I've noticed several of you making commits about particles.  (John,
>>> Britton, looking at you guys.)  In recent versions of Enzo, I stuck in
>>> a dataspace supplement that pre-computes dataspaces for different
>>> particle types in each grid, which should makes things really cheap to
>>> read *only* star particles, or particles of a different type.  I've
>>> put in support for this in the yt-hg repo, but the biggest problem
>>> with it was that I couldn't figure out the right way to address it --
>>> so I'd like to put out there a question to you guys.
>>>
>>> In an ideal world, if you have some data object:
>>>
>>> data_obj
>>>
>>> what is the best way to be able to address particles of different
>>> types, and only get those particles?  (Let's pretend either that IO is
>>> cheap, or we have the dataspace hack.)
>>>
>>> For instance,
>>>
>>> data_object.particles[type]["x"] # for instance
>>>
>>> or maybe,
>>>
>>> pf.h.particles[type].sphere(...)
>>>
>>> or something?  What do you think?  Any off-the-wall ideas?
>>>
>>> -Matt
>>> _______________________________________________
>>> Yt-dev mailing list
>>> Yt-dev at lists.spacepope.org
>>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>>
>> <ATT00001.txt>
>>
>> _______________________________________________
>> Yt-dev mailing list
>> Yt-dev at lists.spacepope.org
>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>>
>>
>