[Yt-dev] Fw: X1024 jobs on Ranger

[Stephen Skory]

Matt,

> Not sure what to tell you.  However, I would like to note that HOP
> does not in any way pool file access.  Projections did -- they do not
> any more as of yesterday or the day before -- but HOP does not.  You
> can try to implement this via the preload command, which you will also
> see inside the profiling module.  We could improve IO by making the
> DataQueue object aware of which grids will be accessed and then doing
> pool-on-demand, where you'd call 'preload', it'd know which grids to
> pool access to, and then when *one* is accessed, all the others in
> that CPU file would also get pulled.

I'll take a look to see if I can figure it out.

I told him for now that I'd stay away unless I can come up with something better. I guess the MDS server doesn't like it when you read 440,000*(1+padding)^3 grids...

 _______________________________________________________
sskory at physics.ucsd.edu           o__  Stephen Skory
http://physics.ucsd.edu/~sskory/ _.>/ _Graduate Student
________________________________(_)_\(_)_______________