Sorry, hit send too early, and google inbox doesn't have "undo"!<br><br>So if you change that line, then at least the load_uniform_grid() call for me takes about 0.02 seconds. <div><br></div><div>If you could test that out and report back, that would be awesome. Once you make that change, you'll need to go to the root yt directory, like path/src/yt-hg probably, then type "python setup.py develop". </div><div><br></div><div>I'll be flying for the next few hours, but if that all works, let us know, and i can submit a pull request to fix it up.</div><div><br></div><div>Cheers,</div><div>Sam</div><br><div class="gmail_quote">On Sun Nov 23 2014 at 12:55:25 PM Sam Skillman <<a href="mailto:samskillman@gmail.com">samskillman@gmail.com</a>> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi Stuart,<br><br>I just tried to do the same thing on a fake 8GB file and am seeing similar things. After killing the job, it exited while doing this:<div><div>^CTraceback (most recent call last):</div><div> File "test_load.py", line 11, in <module></div><div> ds = yt.load_uniform_grid(data, (1024, 1024, 1024))</div><div> File "/home/skillman/local/yt-x86_64/src/yt-3.0/yt/frontends/stream/data_structures.py", line 605, in load_uniform_grid</div><div> field_units, data = unitify_data(data)</div><div> File "/home/skillman/local/yt-x86_64/src/yt-3.0/yt/frontends/stream/data_structures.py", line 494, in unitify_data</div><div> data = dict((field, np.array(val)) for field, val in data.iteritems())</div><div> File "/home/skillman/local/yt-x86_64/src/yt-3.0/yt/frontends/stream/data_structures.py", line 494, in <genexpr></div><div> data = dict((field, np.array(val)) for field, val in data.iteritems())</div><div>KeyboardInterrupt</div></div><div><br></div><div>Looking at line 605 in yt/frontends/stream/data_structures.py, it was choking when trying to "unitify_data". However, if you go in and change line 494 from:</div><div><span style="font-size:13.63636302948px;line-height:17.9999980926514px"> data = dict((field, np.array(val)) for field, val in data.iteritems())</span><br></div><div><span style="font-size:13.63636302948px;line-height:17.9999980926514px">to </span></div><div><span style="font-size:13.63636302948px;line-height:17.9999980926514px"> data = dict((field, val) for field, val in data.iteritems())</span><span style="font-size:13.63636302948px;line-height:17.9999980926514px"><br></span></div><div><span style="font-size:13.63636302948px;line-height:17.9999980926514px"><br></span></div><div><span style="font-size:13.63636302948px;line-height:17.9999980926514px"><br></span></div><br><div class="gmail_quote">On Fri Nov 21 2014 at 3:07:49 PM Stuart Levy <<a href="mailto:salevy@illinois.edu" target="_blank">salevy@illinois.edu</a>> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">
OK, so I have experimented, though not much.<br>
<br>
First, sheepishly: I was off by a decimal place in the original file
size. It's a unigrid with 1.5e10, not 1.5e11 voxels - big enough
to be a nuisance but not heroic.<br>
<br>
Second: load_uniform_grid() on a big numpy.memmap()'ed file, even a
modest 8GB fraction of the full grid, takes a long time - many tens
of minutes? I ran out of time slice before it finished even doing
that. Note this was just calling load_uniform_grid(), not any
attempt at calculation yet.<br>
<br>
Speculation: something sweeps through the memory, causes a page
fault, sweeps a bit more, another page fault, etc. So there'd be
many small I/O calls triggered sequentially, wasting lots of time.
Could that be? If so, then I'm wondering if it could be possible to
discover which portions of the array will be in each node's domain,
and prefetch those in bulk first, using a few very efficient huge
I/O calls (maybe via madvise()).<br>
<br>
Either that, or if I can do my own domain decomposition up front and
*tell* the AMRKDTree which nodes own which slabs of grid, then I
could just read() them in - also efficiently - and let yt do any
further decomposition, maybe.<br>
<br>
Does either route make sense? Is there code I should look at?<br>
<br>
Thanks as ever</div><div bgcolor="#FFFFFF" text="#000000"><br>
<br>
Stuart</div><div bgcolor="#FFFFFF" text="#000000"><br>
<br>
<div>On 11/7/14 1:33 PM, Sam Skillman wrote:<br>
</div>
<blockquote type="cite">
Yep, the volume rendering should build the AMRKDTree itself, and
*should* automatically decompose the giant brick into Np pieces.
As for memory, you may need to (eek) allow for yt casting to
64-bit floats for the data, but you'll have to just experiment a
bit.<br>
<div><br>
</div>
<div>Sam</div>
<br>
<div class="gmail_quote">On Fri Nov 07 2014 at 11:15:13 AM Stuart
Levy <<a href="mailto:salevy@illinois.edu" target="_blank">salevy@illinois.edu</a>>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000"> Thank you, Sam! I
think this makes sense. Except, in case (1), do I need to
do something to bring the AMRKDTree into the picture? Or
are you telling me that it is automatically constructed
whenever you load_uniform_grid(), or volume-render it?<br>
<br>
I think the available nodes have 64GB, so to load the whole
~600GB might take at least 32 nodes or 1024 cores.<br>
<br>
Will let you know how it goes!</div>
<div bgcolor="#FFFFFF" text="#000000"><br>
<br>
<div>On 11/7/14 11:08 AM, Sam Skillman wrote:<br>
</div>
<blockquote type="cite"> Ack, my calculation of 256-512
cores is probably low... feel free to push up much higher.<br>
<br>
<div class="gmail_quote">On Fri Nov 07 2014 at 9:03:51 AM
Sam Skillman <<a href="mailto:samskillman@gmail.com" target="_blank">samskillman@gmail.com</a>>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi
Stuart,
<div><br>
</div>
<div>
<div class="gmail_quote">On Thu Nov 06 2014 at
8:36:28 AM Stuart Levy <<a href="mailto:salevy@illinois.edu" target="_blank">salevy@illinois.edu</a>>
wrote:<br>
</div>
</div>
<div>
<div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hello all,<br>
<br>
We're hoping to use yt parallel volume rendering
on a very large generic<br>
brick - it's a simple rectangular unigrid slab,
but containing something<br>
like 1.5e11 points, so much too large for
load_uniform_grid() to load<br>
into memory in a single machine.<br>
</blockquote>
<div><br>
</div>
</div>
</div>
<div>
<div class="gmail_quote">
<div>Are you loading directly using something like
numpy.fromfile? If so, I think the easiest
method would be to replace that with a np.memmap
(<a href="http://docs.scipy.org/doc/numpy/reference/generated/numpy.memmap.html" target="_blank">http://docs.scipy.org/doc/<u></u>numpy/reference/generated/<u></u>numpy.memmap.html</a>).
Once that is loaded, you should be able to use
load_uniform_grid.</div>
<div><br>
</div>
<div>At that point, there are two possible routes
that both may or may not work well. </div>
<div><br>
</div>
<div>1) Just try rendering with ~256-512 cores,
and the AMRKDTree should try to geometrically
split the grid before performing and I/O. </div>
<div>or</div>
<div>2) Use load_uniform_grid with the keyword
nprocs=N ( for this size simulation, you
probably need something like 256-1024 processors
depending on the memory per core). This should
do the equivalent thing to (1), but it may hit
the I/O here instead of in the kd-tree.</div>
<div><br>
</div>
<div>I *think* (1) should be your best option, but
I haven't tried rendering this large of a
single-grid output.</div>
<div><br>
</div>
<div>When you build the camera option, definitely
start out using the keyword "no_ghost=True", as
this will extrapolate rather than interpolate
from boundary grids to the vertices. The
rendering quality won't be quite as good but for
unigrid simulations there isn't a tremendous
difference. </div>
<div><br>
</div>
<div>Let us know how that goes! I'd be very
excited to see images from such a large sim...</div>
</div>
</div>
<div>
<div class="gmail_quote">
<div><br>
</div>
<div>Sam </div>
</div>
</div>
<div>
<div class="gmail_quote">
<div> <br>
</div>
<div> </div>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> <br>
I imagine it wouldn't be hard to do the domain
decomposition by hand,<br>
loading a different chunk of grid into each MPI
process. But then<br>
what? What would it take to invoke the volume
renderer on each piece<br>
and composite them together? Would it help if
the chunks were stored<br>
in a KDTree? Is there some example (one of the
existing data loaders?)<br>
which I could follow?<br>
______________________________<u></u>_________________<br>
yt-users mailing list<br>
<a href="mailto:yt-users@lists.spacepope.org" target="_blank">yt-users@lists.spacepope.org</a><br>
<a href="http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org" target="_blank">http://lists.spacepope.org/<u></u>listinfo.cgi/yt-users-<u></u>spacepope.org</a><br>
</blockquote>
</div>
</div>
</blockquote>
</div>
<br>
<fieldset></fieldset>
<br>
<pre>______________________________<u></u>_________________
yt-users mailing list
<a href="mailto:yt-users@lists.spacepope.org" target="_blank">yt-users@lists.spacepope.org</a>
<a href="http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org" target="_blank">http://lists.spacepope.org/<u></u>listinfo.cgi/yt-users-<u></u>spacepope.org</a>
</pre>
</blockquote>
<br>
</div>
______________________________<u></u>_________________<br>
yt-users mailing list<br>
<a href="mailto:yt-users@lists.spacepope.org" target="_blank">yt-users@lists.spacepope.org</a><br>
<a href="http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org" target="_blank">http://lists.spacepope.org/<u></u>listinfo.cgi/yt-users-<u></u>spacepope.org</a><br>
</blockquote>
</div>
<br>
<fieldset></fieldset>
<br>
<pre>______________________________<u></u>_________________
yt-users mailing list
<a href="mailto:yt-users@lists.spacepope.org" target="_blank">yt-users@lists.spacepope.org</a>
<a href="http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org" target="_blank">http://lists.spacepope.org/<u></u>listinfo.cgi/yt-users-<u></u>spacepope.org</a>
</pre>
</blockquote>
<br>
</div>
______________________________<u></u><u></u>_________________<br>
yt-users mailing list<br>
<a href="mailto:yt-users@lists.spacepope.org" target="_blank">yt-users@lists.spacepope.org</a><br>
<a href="http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org" target="_blank">http://lists.spacepope.org/<u></u>lis<u></u>tinfo.cgi/yt-users-<u></u>spacepope.<u></u>org</a><br>
</blockquote></div></blockquote></div>