[Yt-dev] Projection speed improvement patch

[Matthew Turk]

> ibrun mpi4py my_script.py --parallel
>
> This is when I get the errors I posted.  I'm not sure whether this is worth
> debugging on yt's side because I think ranger is at fault here.
>
> Sorry about the false alarm!  But I guess it's good to note this.

Oh, hm, interesting -- thanks for the resolution!

I played around a bit, and in my tests, the H5Pcreate call actually
added a substantial amount of time to the running time.  And
preloading the entire hierarchy popped the memory usage up by quite a
bit.

One thing that I'm finding disturbing is that even though I am
preloading, somehow the ReadData (which does a set at a time) gets
called for the weight field.  This shouldn't be happening, so I'm
going to try to track this down in smaller runs...  The
ReadMultipleDataSets routine performs pretty well, so if I can get rid
of the ReadData, the overall improvement should be pretty good.  (For
reference, on the mirage/lustre filesystem on Triton, reading a single
field one-at-a-time from the grids on processor 1 for the light cone
took 40 seconds.  Reading six fields, only calling H5Fopen once per
cpu file, took 30 seconds.  Yowza.  I think those 40 seconds are
unnecessary and can and should be eliminated.)

-Matt