Yep, the volume rendering should build the AMRKDTree itself, and *should* automatically decompose the giant brick into Np pieces. As for memory, you may need to (eek) allow for yt casting to 64-bit floats for the data, but you'll have to just experiment a bit.<br><div><br></div><div>Sam</div><br><div class="gmail_quote">On Fri Nov 07 2014 at 11:15:13 AM Stuart Levy <<a href="mailto:salevy@illinois.edu">salevy@illinois.edu</a>> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">
Thank you, Sam! I think this makes sense. Except, in case (1),
do I need to do something to bring the AMRKDTree into the picture?
Or are you telling me that it is automatically constructed whenever
you load_uniform_grid(), or volume-render it?<br>
<br>
I think the available nodes have 64GB, so to load the whole ~600GB
might take at least 32 nodes or 1024 cores.<br>
<br>
Will let you know how it goes!</div><div bgcolor="#FFFFFF" text="#000000"><br>
<br>
<div>On 11/7/14 11:08 AM, Sam Skillman
wrote:<br>
</div>
<blockquote type="cite">
Ack, my calculation of 256-512 cores is probably low... feel free
to push up much higher.<br>
<br>
<div class="gmail_quote">On Fri Nov 07 2014 at 9:03:51 AM Sam
Skillman <<a href="mailto:samskillman@gmail.com" target="_blank">samskillman@gmail.com</a>>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi Stuart,
<div><br>
</div>
<div>
<div class="gmail_quote">On Thu Nov 06 2014 at 8:36:28 AM
Stuart Levy <<a href="mailto:salevy@illinois.edu" target="_blank">salevy@illinois.edu</a>>
wrote:<br>
</div>
</div>
<div>
<div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hello
all,<br>
<br>
We're hoping to use yt parallel volume rendering on a
very large generic<br>
brick - it's a simple rectangular unigrid slab, but
containing something<br>
like 1.5e11 points, so much too large for
load_uniform_grid() to load<br>
into memory in a single machine.<br>
</blockquote>
<div><br>
</div>
</div>
</div>
<div>
<div class="gmail_quote">
<div>Are you loading directly using something like
numpy.fromfile? If so, I think the easiest method would
be to replace that with a np.memmap (<a href="http://docs.scipy.org/doc/numpy/reference/generated/numpy.memmap.html" target="_blank">http://docs.scipy.org/doc/numpy/reference/generated/numpy.memmap.html</a>).
Once that is loaded, you should be able to use
load_uniform_grid.</div>
<div><br>
</div>
<div>At that point, there are two possible routes that
both may or may not work well. </div>
<div><br>
</div>
<div>1) Just try rendering with ~256-512 cores, and the
AMRKDTree should try to geometrically split the grid
before performing and I/O. </div>
<div>or</div>
<div>2) Use load_uniform_grid with the keyword nprocs=N (
for this size simulation, you probably need something
like 256-1024 processors depending on the memory per
core). This should do the equivalent thing to (1), but
it may hit the I/O here instead of in the kd-tree.</div>
<div><br>
</div>
<div>I *think* (1) should be your best option, but I
haven't tried rendering this large of a single-grid
output.</div>
<div><br>
</div>
<div>When you build the camera option, definitely start
out using the keyword "no_ghost=True", as this will
extrapolate rather than interpolate from boundary grids
to the vertices. The rendering quality won't be quite as
good but for unigrid simulations there isn't a
tremendous difference. </div>
<div><br>
</div>
<div>Let us know how that goes! I'd be very excited to
see images from such a large sim...</div>
</div>
</div>
<div>
<div class="gmail_quote">
<div><br>
</div>
<div>Sam </div>
</div>
</div>
<div>
<div class="gmail_quote">
<div> <br>
</div>
<div> </div>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
I imagine it wouldn't be hard to do the domain
decomposition by hand,<br>
loading a different chunk of grid into each MPI
process. But then<br>
what? What would it take to invoke the volume renderer
on each piece<br>
and composite them together? Would it help if the
chunks were stored<br>
in a KDTree? Is there some example (one of the
existing data loaders?)<br>
which I could follow?<br>
_______________________________________________<br>
yt-users mailing list<br>
<a href="mailto:yt-users@lists.spacepope.org" target="_blank">yt-users@lists.spacepope.org</a><br>
<a href="http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org" target="_blank">http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org</a><br>
</blockquote>
</div>
</div>
</blockquote>
</div>
<br>
<fieldset></fieldset>
<br>
<pre>_______________________________________________
yt-users mailing list
<a href="mailto:yt-users@lists.spacepope.org" target="_blank">yt-users@lists.spacepope.org</a>
<a href="http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org" target="_blank">http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org</a>
</pre>
</blockquote>
<br>
</div>
______________________________<u></u>_________________<br>
yt-users mailing list<br>
<a href="mailto:yt-users@lists.spacepope.org" target="_blank">yt-users@lists.spacepope.org</a><br>
<a href="http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org" target="_blank">http://lists.spacepope.org/<u></u>listinfo.cgi/yt-users-<u></u>spacepope.org</a><br>
</blockquote></div>