[yt-dev] io chunks in yt-3.0

[Britton Smith]

Hi Matt,

Sure, I'm doing the following with the rockstar_halos data on
yt-project.org/data:

pf = load("rockstar_halos/halos_0.0.bin")
dd = pf.all_data()
print dd["particle_mass"].size
for chunk in dd.chunks([], "io"):
    print chunk["particle_mass"].size

This dataset has two files, but running this, I only get one chunk with the
same size as the all_data container.

Britton


On Mon, May 12, 2014 at 2:19 PM, Matthew Turk <matthewturk at gmail.com> wrote:

> Hi Britton,
>
> On Mon, May 12, 2014 at 2:16 PM, Britton Smith <brittonsmith at gmail.com>
> wrote:
> > Hey all,
> >
> > I'm working on changing how HaloCatalog objects loop over halos from a
> model
> > where every processor has to hold the entire halo list to a model in
> which
> > we loop over "io" chunks.  This will make the HaloCatalog scale much
> better
> > to extremely large catalogs.  My understanding was that an "io" chunk was
> > essentially one file on disk.  However, when I try to get io chunks from
> any
> > one of the various halo catalog frontends, there seems to only ever be a
> > single chunk that contains everything, regardless of how many files on
> disk
> > the data is spread over.  Is there a way to change this so that an io
> chunk
> > represents the data from a single file?
>
> That sounds like a bug; IO should be single file at a time.  Can you
> show me how you're checking?
>
> -Matt
>
> >
> > Britton
> >
> > _______________________________________________
> > yt-dev mailing list
> > yt-dev at lists.spacepope.org
> > http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
> >
> _______________________________________________
> yt-dev mailing list
> yt-dev at lists.spacepope.org
> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.spacepope.org/pipermail/yt-dev-spacepope.org/attachments/20140512/01522130/attachment.htm>