[yt-dev] io chunks in yt-3.0

[Britton Smith]

Hi Matt,

Thanks for the informative reply.  I'm definitely up for looking into this
this week.

Britton


On Tue, May 13, 2014 at 7:56 AM, Matthew Turk <matthewturk at gmail.com> wrote:

> On Mon, May 12, 2014 at 2:26 PM, Britton Smith <brittonsmith at gmail.com>
> wrote:
> > Hi Matt,
> >
> > Sure, I'm doing the following with the rockstar_halos data on
> > yt-project.org/data:
> >
> > pf = load("rockstar_halos/halos_0.0.bin")
> > dd = pf.all_data()
> > print dd["particle_mass"].size
> > for chunk in dd.chunks([], "io"):
> >     print chunk["particle_mass"].size
> >
> > This dataset has two files, but running this, I only get one chunk with
> the
> > same size as the all_data container.
>
> I've looked into it, and this is my fault for how the particle
> datasets are currently set up.  I knew this a little bit ago, but it's
> been long enough since I last looked at it that I've forgotten.  Right
> now, this is the *biggest* problem with particle datasets.  There is
> no support for sub-selecting data components as is.  There are two
> things:
>
> 1) If you select a small region, it will not read all the data files
> -- it knows the ones to read.
> 2) There's only one chunk, regardless of what you select.
>
> This is because right now, everything lives inside the single global
> octree.  This is something I worked on last fall, but didn't finish.
> Essentially, what needs to happen is to convert the single global
> octree to a forest of octrees, similar to how RAMSES, ART, ARTIO are
> set up in yt.  The reason it's somewhat harder for particle datasets
> is that many of the operations we do on particle datasets
> (specifically on SPH) require neighboring particles, so the selection
> needs to be somewhat fuzzier to ensure boundary particles are
> included.
>
> This is something that's essentially ready to go, but when we started
> pushing for 3.0, I put it off on my todo list until 3.1.  It's going
> to be a major blocker not too long from now -- we're already hitting
> that point -- so the sooner we get to 3.0 the sooner we can do this.
> In NYC this week I can talk with you in person and maybe we can figure
> out a game plan for trying to implement this, if you want to have a
> go.
>
> -Matt
>
> >
> > Britton
> >
> >
> > On Mon, May 12, 2014 at 2:19 PM, Matthew Turk <matthewturk at gmail.com>
> wrote:
> >>
> >> Hi Britton,
> >>
> >> On Mon, May 12, 2014 at 2:16 PM, Britton Smith <brittonsmith at gmail.com>
> >> wrote:
> >> > Hey all,
> >> >
> >> > I'm working on changing how HaloCatalog objects loop over halos from a
> >> > model
> >> > where every processor has to hold the entire halo list to a model in
> >> > which
> >> > we loop over "io" chunks.  This will make the HaloCatalog scale much
> >> > better
> >> > to extremely large catalogs.  My understanding was that an "io" chunk
> >> > was
> >> > essentially one file on disk.  However, when I try to get io chunks
> from
> >> > any
> >> > one of the various halo catalog frontends, there seems to only ever
> be a
> >> > single chunk that contains everything, regardless of how many files on
> >> > disk
> >> > the data is spread over.  Is there a way to change this so that an io
> >> > chunk
> >> > represents the data from a single file?
> >>
> >> That sounds like a bug; IO should be single file at a time.  Can you
> >> show me how you're checking?
> >>
> >> -Matt
> >>
> >> >
> >> > Britton
> >> >
> >> > _______________________________________________
> >> > yt-dev mailing list
> >> > yt-dev at lists.spacepope.org
> >> > http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
> >> >
> >> _______________________________________________
> >> yt-dev mailing list
> >> yt-dev at lists.spacepope.org
> >> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
> >
> >
> >
> > _______________________________________________
> > yt-dev mailing list
> > yt-dev at lists.spacepope.org
> > http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
> >
> _______________________________________________
> yt-dev mailing list
> yt-dev at lists.spacepope.org
> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.spacepope.org/pipermail/yt-dev-spacepope.org/attachments/20140513/e32f123f/attachment.htm>