<html><head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
</head>
<body bgcolor="#FFFFFF" text="#000000">
Sam: yes, that fix works great!<br>
<br>
<br>
<div class="moz-cite-prefix">On 11/23/14 2:58 PM, Sam Skillman
wrote:<br>
</div>
<blockquote cite="mid:CAB4=1s35d-YFSAt=_kwBJJ_wmNfTqDMQw206rNz_NLk=ATFVow@mail.gmail.com" type="cite">
Sorry, hit send too early, and google inbox doesn't have "undo"!<br>
<br>
So if you change that line, then at least the load_uniform_grid()
call for me takes about 0.02 seconds.
<div><br>
</div>
<div>If you could test that out and report back, that would be
awesome. Once you make that change, you'll need to go to the
root yt directory, like path/src/yt-hg probably, then type
"python setup.py develop". </div>
<div><br>
</div>
<div>I'll be flying for the next few hours, but if that all works,
let us know, and i can submit a pull request to fix it up.</div>
<div><br>
</div>
<div>Cheers,</div>
<div>Sam</div>
<br>
<div class="gmail_quote">On Sun Nov 23 2014 at 12:55:25 PM Sam
Skillman <<a moz-do-not-send="true" href="mailto:samskillman@gmail.com">samskillman@gmail.com</a>>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">Hi Stuart,<br>
<br>
I just tried to do the same thing on a fake 8GB file and am
seeing similar things. After killing the job, it exited while
doing this:
<div>
<div>^CTraceback (most recent call last):</div>
<div> File "test_load.py", line 11, in <module></div>
<div> ds = yt.load_uniform_grid(data, (1024, 1024, 1024))</div>
<div> File
"/home/skillman/local/yt-x86_64/src/yt-3.0/yt/frontends/stream/data_structures.py",
line 605, in load_uniform_grid</div>
<div> field_units, data = unitify_data(data)</div>
<div> File
"/home/skillman/local/yt-x86_64/src/yt-3.0/yt/frontends/stream/data_structures.py",
line 494, in unitify_data</div>
<div> data = dict((field, np.array(val)) for field, val
in data.iteritems())</div>
<div> File
"/home/skillman/local/yt-x86_64/src/yt-3.0/yt/frontends/stream/data_structures.py",
line 494, in <genexpr></div>
<div> data = dict((field, np.array(val)) for field, val
in data.iteritems())</div>
<div>KeyboardInterrupt</div>
</div>
<div><br>
</div>
<div>Looking at line 605 in
yt/frontends/stream/data_structures.py, it was choking when
trying to "unitify_data". However, if you go in and change
line 494 from:</div>
<div><span style="font-size:13.63636302948px;line-height:17.9999980926514px"> data
= dict((field, np.array(val)) for field, val in
data.iteritems())</span><br>
</div>
<div><span style="font-size:13.63636302948px;line-height:17.9999980926514px">to </span></div>
<div><span style="font-size:13.63636302948px;line-height:17.9999980926514px"> data
= dict((field, val) for field, val in data.iteritems())</span><span style="font-size:13.63636302948px;line-height:17.9999980926514px"><br>
</span></div>
<div><span style="font-size:13.63636302948px;line-height:17.9999980926514px"><br>
</span></div>
<div><span style="font-size:13.63636302948px;line-height:17.9999980926514px"><br>
</span></div>
<br>
<div class="gmail_quote">On Fri Nov 21 2014 at 3:07:49 PM
Stuart Levy <<a moz-do-not-send="true" href="mailto:salevy@illinois.edu" target="_blank">salevy@illinois.edu</a>>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000"> OK, so I have
experimented, though not much.<br>
<br>
First, sheepishly: I was off by a decimal place in the
original file size. It's a unigrid with 1.5e10, not
1.5e11 voxels - big enough to be a nuisance but not
heroic.<br>
<br>
Second: load_uniform_grid() on a big numpy.memmap()'ed
file, even a modest 8GB fraction of the full grid, takes
a long time - many tens of minutes? I ran out of time
slice before it finished even doing that. Note this
was just calling load_uniform_grid(), not any attempt at
calculation yet.<br>
<br>
Speculation: something sweeps through the memory, causes
a page fault, sweeps a bit more, another page fault,
etc. So there'd be many small I/O calls triggered
sequentially, wasting lots of time. Could that be? If
so, then I'm wondering if it could be possible to
discover which portions of the array will be in each
node's domain, and prefetch those in bulk first, using a
few very efficient huge I/O calls (maybe via madvise()).<br>
<br>
Either that, or if I can do my own domain decomposition
up front and *tell* the AMRKDTree which nodes own which
slabs of grid, then I could just read() them in - also
efficiently - and let yt do any further decomposition,
maybe.<br>
<br>
Does either route make sense? Is there code I should
look at?<br>
<br>
Thanks as ever</div>
<div bgcolor="#FFFFFF" text="#000000"><br>
<br>
Stuart</div>
<div bgcolor="#FFFFFF" text="#000000"><br>
<br>
<div>On 11/7/14 1:33 PM, Sam Skillman wrote:<br>
</div>
<blockquote type="cite"> Yep, the volume rendering
should build the AMRKDTree itself, and *should*
automatically decompose the giant brick into Np
pieces. As for memory, you may need to (eek) allow for
yt casting to 64-bit floats for the data, but you'll
have to just experiment a bit.<br>
<div><br>
</div>
<div>Sam</div>
<br>
<div class="gmail_quote">On Fri Nov 07 2014 at
11:15:13 AM Stuart Levy <<a moz-do-not-send="true" href="mailto:salevy@illinois.edu" target="_blank">salevy@illinois.edu</a>>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000"> Thank you,
Sam! I think this makes sense. Except, in
case (1), do I need to do something to bring the
AMRKDTree into the picture? Or are you telling
me that it is automatically constructed whenever
you load_uniform_grid(), or volume-render it?<br>
<br>
I think the available nodes have 64GB, so to
load the whole ~600GB might take at least 32
nodes or 1024 cores.<br>
<br>
Will let you know how it goes!</div>
<div bgcolor="#FFFFFF" text="#000000"><br>
<br>
<div>On 11/7/14 11:08 AM, Sam Skillman wrote:<br>
</div>
<blockquote type="cite"> Ack, my calculation of
256-512 cores is probably low... feel free to
push up much higher.<br>
<br>
<div class="gmail_quote">On Fri Nov 07 2014 at
9:03:51 AM Sam Skillman <<a moz-do-not-send="true" href="mailto:samskillman@gmail.com" target="_blank">samskillman@gmail.com</a>>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px
#ccc solid;padding-left:1ex">Hi Stuart,
<div><br>
</div>
<div>
<div class="gmail_quote">On Thu Nov 06
2014 at 8:36:28 AM Stuart Levy <<a moz-do-not-send="true" href="mailto:salevy@illinois.edu" target="_blank">salevy@illinois.edu</a>>
wrote:<br>
</div>
</div>
<div>
<div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc
solid;padding-left:1ex">Hello all,<br>
<br>
We're hoping to use yt parallel
volume rendering on a very large
generic<br>
brick - it's a simple rectangular
unigrid slab, but containing
something<br>
like 1.5e11 points, so much too
large for load_uniform_grid() to
load<br>
into memory in a single machine.<br>
</blockquote>
<div><br>
</div>
</div>
</div>
<div>
<div class="gmail_quote">
<div>Are you loading directly using
something like numpy.fromfile? If
so, I think the easiest method would
be to replace that with a np.memmap
(<a moz-do-not-send="true" href="http://docs.scipy.org/doc/numpy/reference/generated/numpy.memmap.html" target="_blank">http://docs.scipy.org/doc/numpy/reference/generated/numpy.memmap.html</a>).
Once that is loaded, you should be
able to use load_uniform_grid.</div>
<div><br>
</div>
<div>At that point, there are two
possible routes that both may or may
not work well. </div>
<div><br>
</div>
<div>1) Just try rendering with
~256-512 cores, and the AMRKDTree
should try to geometrically split
the grid before performing and I/O. </div>
<div>or</div>
<div>2) Use load_uniform_grid with the
keyword nprocs=N ( for this size
simulation, you probably need
something like 256-1024 processors
depending on the memory per core).
This should do the equivalent thing
to (1), but it may hit the I/O here
instead of in the kd-tree.</div>
<div><br>
</div>
<div>I *think* (1) should be your best
option, but I haven't tried
rendering this large of a
single-grid output.</div>
<div><br>
</div>
<div>When you build the camera option,
definitely start out using the
keyword "no_ghost=True", as this
will extrapolate rather than
interpolate from boundary grids to
the vertices. The rendering quality
won't be quite as good but for
unigrid simulations there isn't a
tremendous difference. </div>
<div><br>
</div>
<div>Let us know how that goes! I'd
be very excited to see images from
such a large sim...</div>
</div>
</div>
<div>
<div class="gmail_quote">
<div><br>
</div>
<div>Sam </div>
</div>
</div>
<div>
<div class="gmail_quote">
<div> <br>
</div>
<div> </div>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc
solid;padding-left:1ex"> <br>
I imagine it wouldn't be hard to do
the domain decomposition by hand,<br>
loading a different chunk of grid
into each MPI process. But then<br>
what? What would it take to invoke
the volume renderer on each piece<br>
and composite them together? Would
it help if the chunks were stored<br>
in a KDTree? Is there some example
(one of the existing data loaders?)<br>
which I could follow?<br>
_______________________________________________<br>
yt-users mailing list<br>
<a moz-do-not-send="true" href="mailto:yt-users@lists.spacepope.org" target="_blank">yt-users@lists.spacepope.org</a><br>
<a moz-do-not-send="true" href="http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org" target="_blank">http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org</a><br>
</blockquote>
</div>
</div>
</blockquote>
</div>
<br>
<fieldset></fieldset>
<br>
<pre>_______________________________________________
yt-users mailing list
<a moz-do-not-send="true" href="mailto:yt-users@lists.spacepope.org" target="_blank">yt-users@lists.spacepope.org</a>
<a moz-do-not-send="true" href="http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org" target="_blank">http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org</a>
</pre>
</blockquote>
<br>
</div>
_______________________________________________<br>
yt-users mailing list<br>
<a moz-do-not-send="true" href="mailto:yt-users@lists.spacepope.org" target="_blank">yt-users@lists.spacepope.org</a><br>
<a moz-do-not-send="true" href="http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org" target="_blank">http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org</a><br>
</blockquote>
</div>
<br>
<fieldset></fieldset>
<br>
<pre>_______________________________________________
yt-users mailing list
<a moz-do-not-send="true" href="mailto:yt-users@lists.spacepope.org" target="_blank">yt-users@lists.spacepope.org</a>
<a moz-do-not-send="true" href="http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org" target="_blank">http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org</a>
</pre>
</blockquote>
<br>
</div>
_______________________________________________<br>
yt-users mailing list<br>
<a moz-do-not-send="true" href="mailto:yt-users@lists.spacepope.org" target="_blank">yt-users@lists.spacepope.org</a><br>
<a moz-do-not-send="true" href="http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org" target="_blank">http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org</a><br>
</blockquote>
</div>
</blockquote>
</div>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
yt-users mailing list
<a class="moz-txt-link-abbreviated" href="mailto:yt-users@lists.spacepope.org">yt-users@lists.spacepope.org</a>
<a class="moz-txt-link-freetext" href="http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org">http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org</a>
</pre>
</blockquote>
<br>
</body>
</html>