[yt-users] cloud in cell mesh construction for particle data

Brendan Griffen brendan.f.griffen at gmail.com
Mon Jun 9 07:42:25 PDT 2014


HI Nathan,

I just increased n_particles = 1e9 (roughly what I have) for the example
you gave me and it does indeed crash due to memory.

Line #    Mem usage    Increment   Line Contents
================================================
     6   92.121 MiB    0.000 MiB   @profile
     7                             def test():
     8   92.121 MiB    0.000 MiB       n_particles = 1e9
     9
    10 22980.336 MiB 22888.215 MiB       ppx, ppy, ppz =
1e6*np.random.normal(size=[3, n_particles])
    11
    12 30609.734 MiB 7629.398 MiB       ppm = np.ones(n_particles)
    13
    14 30609.734 MiB    0.000 MiB       data = {'particle_position_x': ppx,
    15 30609.734 MiB    0.000 MiB               'particle_position_y': ppy,
    16 30609.734 MiB    0.000 MiB               'particle_position_z': ppz,
    17 30609.734 MiB    0.000 MiB               'particle_mass': ppm,
    18 30609.734 MiB    0.000 MiB               'number_of_particles':
n_particles}
    19
    20 30609.738 MiB    0.004 MiB       bbox = 1.1*np.array([[min(ppx),
max(ppx)], [min(ppy), max(ppy)], [min(ppy), max(ppy)]])
    21
    22 30610.027 MiB    0.289 MiB       ds = yt.load_uniform_grid(data,
[256, 256, 256], length_unit=parsec, mass_unit=1e8*Msun, bbox=bbox)
    23
    24 30614.352 MiB    4.324 MiB       grid_object = ds.index.grids[0]
    25
    26                                 uniform_array =
grid_object['deposit', 'all_cic']
    27
    28                                 print uniform_array.max()
    29                                 print uniform_array.shape
    30
    31                                 plt.imshow(uniform_array[:,:,128].v)
    32
    33                                 plt.savefig('test.png')


Traceback (most recent call last):
  File
"/nfs/blank/h4231/bgriffen/data/lib/yt-x86_64/lib/python2.7/runpy.py", line
162, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File
"/nfs/blank/h4231/bgriffen/data/lib/yt-x86_64/lib/python2.7/runpy.py", line
72, in _run_code
    exec code in run_globals
  File
"/bigbang/data/bgriffen/lib/memory_profiler/lib/python/memory_profiler.py",
line 821, in <module>
    execfile(__file__, ns, ns)
  File "profilecic.py", line 36, in <module>
    test()
  File
"/bigbang/data/bgriffen/lib/memory_profiler/lib/python/memory_profiler.py",
line 424, in f
    result = func(*args, **kwds)
  File "profilecic.py", line 28, in test
    print uniform_array.max()
  File "profilecic.py", line 28, in test
    print uniform_array.max()
  File
"/bigbang/data/bgriffen/lib/memory_profiler/lib/python/memory_profiler.py",
line 470, in trace_memory_usage
    mem = _get_memory(-1)
  File
"/bigbang/data/bgriffen/lib/memory_profiler/lib/python/memory_profiler.py",
line 69, in _get_memory
    stdout=subprocess.PIPE
  File
"/nfs/blank/h4231/bgriffen/data/lib/yt-x86_64/lib/python2.7/subprocess.py",
line 709, in __init__
    errread, errwrite)
  File
"/nfs/blank/h4231/bgriffen/data/lib/yt-x86_64/lib/python2.7/subprocess.py",
line 1222, in _execute_child
    self.pid = os.fork()
OSError: [Errno 12] Cannot allocate memory
[yt-x86_64] bigbang%

So even the "memory efficient" run can't be run on 1024^3 (ndim = 256) on
128GB machine. Though this because of the way the profiler works.

Brendan


On Mon, Jun 9, 2014 at 9:20 AM, Matthew Turk <matthewturk at gmail.com> wrote:

> > /bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/units/yt_array.pyc in
> > convert_to_units(self, units)
> >     366
> >     367         self.units = new_units
> > --> 368         self *= conversion_factor
> >     369         return self
> >     370
> >
> > /bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/units/yt_array.pyc in
> > __imul__(self, other)
> >     667         """ See __mul__. """
> >     668         oth = sanitize_units_mul(self, other)
> > --> 669         return np.multiply(self, oth, out=self)
> >     670
> >     671     def __div__(self, right_object):
> >
> > /bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/units/yt_array.pyc in
> > __array_wrap__(self, out_arr, context)
> >     966                 # casting to YTArray avoids creating a YTQuantity
> > with size > 1
> >     967                 return YTArray(np.array(out_arr, unit))
> > --> 968             return ret_class(np.array(out_arr), unit)
> >     969
> >     970
> >
> > MemoryError:
> >
>
> Nathan, any idea why this is copying?  We shouldn't be copying here.
> _______________________________________________
> yt-users mailing list
> yt-users at lists.spacepope.org
> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.spacepope.org/pipermail/yt-users-spacepope.org/attachments/20140609/37374244/attachment.htm>


More information about the yt-users mailing list