[yt-users] cloud in cell mesh construction for particle data
Brendan Griffen
brendan.f.griffen at gmail.com
Mon Jun 9 07:42:25 PDT 2014
HI Nathan,
I just increased n_particles = 1e9 (roughly what I have) for the example
you gave me and it does indeed crash due to memory.
Line # Mem usage Increment Line Contents
================================================
6 92.121 MiB 0.000 MiB @profile
7 def test():
8 92.121 MiB 0.000 MiB n_particles = 1e9
9
10 22980.336 MiB 22888.215 MiB ppx, ppy, ppz =
1e6*np.random.normal(size=[3, n_particles])
11
12 30609.734 MiB 7629.398 MiB ppm = np.ones(n_particles)
13
14 30609.734 MiB 0.000 MiB data = {'particle_position_x': ppx,
15 30609.734 MiB 0.000 MiB 'particle_position_y': ppy,
16 30609.734 MiB 0.000 MiB 'particle_position_z': ppz,
17 30609.734 MiB 0.000 MiB 'particle_mass': ppm,
18 30609.734 MiB 0.000 MiB 'number_of_particles':
n_particles}
19
20 30609.738 MiB 0.004 MiB bbox = 1.1*np.array([[min(ppx),
max(ppx)], [min(ppy), max(ppy)], [min(ppy), max(ppy)]])
21
22 30610.027 MiB 0.289 MiB ds = yt.load_uniform_grid(data,
[256, 256, 256], length_unit=parsec, mass_unit=1e8*Msun, bbox=bbox)
23
24 30614.352 MiB 4.324 MiB grid_object = ds.index.grids[0]
25
26 uniform_array =
grid_object['deposit', 'all_cic']
27
28 print uniform_array.max()
29 print uniform_array.shape
30
31 plt.imshow(uniform_array[:,:,128].v)
32
33 plt.savefig('test.png')
Traceback (most recent call last):
File
"/nfs/blank/h4231/bgriffen/data/lib/yt-x86_64/lib/python2.7/runpy.py", line
162, in _run_module_as_main
"__main__", fname, loader, pkg_name)
File
"/nfs/blank/h4231/bgriffen/data/lib/yt-x86_64/lib/python2.7/runpy.py", line
72, in _run_code
exec code in run_globals
File
"/bigbang/data/bgriffen/lib/memory_profiler/lib/python/memory_profiler.py",
line 821, in <module>
execfile(__file__, ns, ns)
File "profilecic.py", line 36, in <module>
test()
File
"/bigbang/data/bgriffen/lib/memory_profiler/lib/python/memory_profiler.py",
line 424, in f
result = func(*args, **kwds)
File "profilecic.py", line 28, in test
print uniform_array.max()
File "profilecic.py", line 28, in test
print uniform_array.max()
File
"/bigbang/data/bgriffen/lib/memory_profiler/lib/python/memory_profiler.py",
line 470, in trace_memory_usage
mem = _get_memory(-1)
File
"/bigbang/data/bgriffen/lib/memory_profiler/lib/python/memory_profiler.py",
line 69, in _get_memory
stdout=subprocess.PIPE
File
"/nfs/blank/h4231/bgriffen/data/lib/yt-x86_64/lib/python2.7/subprocess.py",
line 709, in __init__
errread, errwrite)
File
"/nfs/blank/h4231/bgriffen/data/lib/yt-x86_64/lib/python2.7/subprocess.py",
line 1222, in _execute_child
self.pid = os.fork()
OSError: [Errno 12] Cannot allocate memory
[yt-x86_64] bigbang%
So even the "memory efficient" run can't be run on 1024^3 (ndim = 256) on
128GB machine. Though this because of the way the profiler works.
Brendan
On Mon, Jun 9, 2014 at 9:20 AM, Matthew Turk <matthewturk at gmail.com> wrote:
> > /bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/units/yt_array.pyc in
> > convert_to_units(self, units)
> > 366
> > 367 self.units = new_units
> > --> 368 self *= conversion_factor
> > 369 return self
> > 370
> >
> > /bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/units/yt_array.pyc in
> > __imul__(self, other)
> > 667 """ See __mul__. """
> > 668 oth = sanitize_units_mul(self, other)
> > --> 669 return np.multiply(self, oth, out=self)
> > 670
> > 671 def __div__(self, right_object):
> >
> > /bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/units/yt_array.pyc in
> > __array_wrap__(self, out_arr, context)
> > 966 # casting to YTArray avoids creating a YTQuantity
> > with size > 1
> > 967 return YTArray(np.array(out_arr, unit))
> > --> 968 return ret_class(np.array(out_arr), unit)
> > 969
> > 970
> >
> > MemoryError:
> >
>
> Nathan, any idea why this is copying? We shouldn't be copying here.
> _______________________________________________
> yt-users mailing list
> yt-users at lists.spacepope.org
> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.spacepope.org/pipermail/yt-users-spacepope.org/attachments/20140609/37374244/attachment.htm>
More information about the yt-users
mailing list