[yt-users] memory error for gadget visualization

Matthew Turk matthewturk at gmail.com
Mon Apr 13 08:19:37 PDT 2015


Hi Junhwan,

This is a fundamental failing with the way yt 3.0 and 3.1 deals with
particles.  I have reworked this in an open pull request, which is
experimental (but nearing completion, prior to review) if you would
like to try it out; it should make it both feasible and considerably
faster.  Let me know off list if you're willing to try it out.

-Matt

On Thu, Apr 9, 2015 at 2:54 PM, Junhwan Choi (최준환)
<choi.junhwan at gmail.com> wrote:
> Hi yt users and developer,
>
> I tried to make a simple visualization for 244/h cMpc box with 512^3
> DM and gas gadget simulation with following script
> ==================================
> import matplotlib as matplotlib
> matplotlib.use('Agg')
> import matplotlib.pylab as pylab
> import numpy as np
> import yt
> import yt.units as units
>
> index = 0
> text_dict = {'color':'white', 'size':20}
> fname = ('../snap_vbc_%03d.hdf5' % index)
> unit_base = {'UnitLength_in_cm'         : 3.08568e+21, 'UnitMass_in_g'
>            :   1.989e+43, 'UnitVelocity_in_cm_per_s' :      100000}
> bbox_lim = 244000 #kpc
> bbox = [[0,bbox_lim], [0,bbox_lim], [0,bbox_lim]]
>
> ds = yt.load(fname,unit_base=unit_base,bounding_box=bbox)
> cen_center=np.array([122000,122000,122000])
>
> pc = yt.ProjectionPlot(ds, "z", ('deposit', 'PartType0_density'),
> width = (244.0, 'Mpccm/h'))
> pc.annotate_text((0.8,0.9),'z=%4.2f' % ds.current_redshift,
> data_coords=False, text_args = text_dict)
> pc.save("all_view_%04d" % index)
>
> pc = yt.ProjectionPlot(ds, "z", ('deposit', 'PartType1_density'),
> width = (244.0, 'Mpccm/h'))
> pc.annotate_text((0.8,0.9),'z=%4.2f' % ds.current_redshift,
> data_coords=False, text_args = text_dict)
> pc.save("all_view_%04d" % index)
>
> pc = yt.ProjectionPlot(ds, "z", ('deposit', 'all_density'), width =
> (244.0, 'Mpccm/h'))
> pc.annotate_text((0.8,0.9),'z=%4.2f' % ds.current_redshift,
> data_coords=False, text_args = text_dict)
> pc.save("all_view_%04d" % index)
>
> pc = yt.ProjectionPlot(ds, "z", ('gas', 'density'), width = (244.0, 'Mpccm/h'))
> pc.annotate_text((0.8,0.9),'z=%4.2f' % ds.current_redshift,
> data_coords=False, text_args = text_dict)
> pc.save("all_view_%04d" % index)
> ===================================
>
> It initially works an make couple figures but it crash with following
> error message:
> .......
> Traceback (most recent call last):
>   File "viz.py", line 56, in <module>
>     viz4(index)
>   File "viz.py", line 50, in viz4
>     pc = yt.ProjectionPlot(ds, "z", ('gas', 'density'), width =
> (244.0, 'Mpccm/h'))
>   File "/work/00916/tg459470/common/anaconda/lib/python2.7/site-packages/yt-3.2dev-py2.7-linux-x86_64.egg/yt/visualization/plot_window.py",
> line 1228, in __init__
>     field_parameters = field_parameters, method = method)
>   File "/work/00916/tg459470/common/anaconda/lib/python2.7/site-packages/yt-3.2dev-py2.7-linux-x86_64.egg/yt/data_objects/construction_data_containers.py",
> line 250, in __init__
>     self.get_data(field)
>   File "/work/00916/tg459470/common/anaconda/lib/python2.7/site-packages/yt-3.2dev-py2.7-linux-x86_64.egg/yt/data_objects/construction_data_containers.py",
> line 295, in get_data
>     chunk_fields, "io", local_only = True)):
>   File "/work/00916/tg459470/common/anaconda/lib/python2.7/site-packages/yt-3.2dev-py2.7-linux-x86_64.egg/yt/utilities/parallel_tools/parallel_analysis_interface.py",
> line 507, in para
> llel_objects
>     for obj_id, obj in oiter:
>   File "/work/00916/tg459470/common/anaconda/lib/python2.7/site-packages/yt-3.2dev-py2.7-linux-x86_64.egg/yt/data_objects/data_containers.py",
> line 592, in chunks
>     self.get_data(fields)
>   File "/work/00916/tg459470/common/anaconda/lib/python2.7/site-packages/yt-3.2dev-py2.7-linux-x86_64.egg/yt/data_objects/data_containers.py",
> line 693, in get_data
>     self._generate_fields(fields_to_generate)
>   File "/work/00916/tg459470/common/anaconda/lib/python2.7/site-packages/yt-3.2dev-py2.7-linux-x86_64.egg/yt/data_objects/data_containers.py",
> line 713, in _generate_fields
>     fd = self._generate_field(field)
>   File "/work/00916/tg459470/common/anaconda/lib/python2.7/site-packages/yt-3.2dev-py2.7-linux-x86_64.egg/yt/data_objects/data_containers.py",
> line 285, in _generate_field
>     tr = self._generate_fluid_field(field)
>   File "/work/00916/tg459470/common/anaconda/lib/python2.7/site-packages/yt-3.2dev-py2.7-linux-x86_64.egg/yt/data_objects/data_containers.py",
> line 303, in _generate_fluid_field
>     rv = self._generate_spatial_fluid(field, ngt_exception.ghost_zones)
>   File "/work/00916/tg459470/common/anaconda/lib/python2.7/site-packages/yt-3.2dev-py2.7-linux-x86_64.egg/yt/data_objects/data_containers.py",
> line 319, in _generate_spatial_fluid
>     ind += o.select(self.selector, self[field], rv, ind)
>   File "/work/00916/tg459470/common/anaconda/lib/python2.7/site-packages/yt-3.2dev-py2.7-linux-x86_64.egg/yt/data_objects/data_containers.py",
> line 248, in __getitem__
>     self.get_data(f)
>   File "/work/00916/tg459470/common/anaconda/lib/python2.7/site-packages/yt-3.2dev-py2.7-linux-x86_64.egg/yt/data_objects/data_containers.py",
> line 693, in get_data
>     self._generate_fields(fields_to_generate)
>   File "/work/00916/tg459470/common/anaconda/lib/python2.7/site-packages/yt-3.2dev-py2.7-linux-x86_64.egg/yt/data_objects/data_containers.py",
> line 713, in _generate_fields
>     fd = self._generate_field(field)
>   File "/work/00916/tg459470/common/anaconda/lib/python2.7/site-packages/yt-3.2dev-py2.7-linux-x86_64.egg/yt/data_objects/data_containers.py",
> line 285, in _generate_field
>     tr = self._generate_fluid_field(field)
>   File "/work/00916/tg459470/common/anaconda/lib/python2.7/site-packages/yt-3.2dev-py2.7-linux-x86_64.egg/yt/data_objects/data_containers.py",
> line 305, in _generate_fluid_field
>     rv = finfo(gen_obj)
>   File "/work/00916/tg459470/common/anaconda/lib/python2.7/site-packages/yt-3.2dev-py2.7-linux-x86_64.egg/yt/fields/derived_field.py",
> line 184, in __call__
>     dd = self._function(self, data)
>   File "/work/00916/tg459470/common/anaconda/lib/python2.7/site-packages/yt-3.2dev-py2.7-linux-x86_64.egg/yt/fields/particle_fields.py",
> line 958, in _vol_weight
>     dens = data[ptype, density_name].in_cgs()
>   File "/work/00916/tg459470/common/anaconda/lib/python2.7/site-packages/yt-3.2dev-py2.7-linux-x86_64.egg/yt/data_objects/octree_subset.py",
> line 77, in __getitem__
>     tr = super(OctreeSubset, self).__getitem__(key)
>   File "/work/00916/tg459470/common/anaconda/lib/python2.7/site-packages/yt-3.2dev-py2.7-linux-x86_64.egg/yt/data_objects/data_containers.py",
> line 248, in __getitem__
>     self.get_data(f)
>   File "/work/00916/tg459470/common/anaconda/lib/python2.7/site-packages/yt-3.2dev-py2.7-linux-x86_64.egg/yt/data_objects/data_containers.py",
> line 687, in get_data
>     particles, self, self._current_chunk)
>   File "/work/00916/tg459470/common/anaconda/lib/python2.7/site-packages/yt-3.2dev-py2.7-linux-x86_64.egg/yt/geometry/geometry_handler.py",
> line 234, in _read_particle_fields
>     fields_to_read)
>   File "/work/00916/tg459470/common/anaconda/lib/python2.7/site-packages/yt-3.2dev-py2.7-linux-x86_64.egg/yt/utilities/io_handler.py",
> line 177, in _read_particle_selection
>     for field_r, vals in self._read_particle_fields(chunks, ptf, selector):
>   File "/work/00916/tg459470/common/anaconda/lib/python2.7/site-packages/yt-3.2dev-py2.7-linux-x86_64.egg/yt/frontends/owls/io.py",
> line 97, in _read_particle_fields
>     coords = g["Coordinates"][:].astype("float64")
> MemoryError
>
> It looks there is a memory shortage.
> I use the stampede normal node which has ~32GB and I do not think
> 512^3 gas and DM particles require this amount memory.
> [But, I successfully made a visualization with 256^3 in 100/h cMpc run.]
> Is there any my possible mistake that leaks the memory?
> If so, could you inform me to overcome this memory issue.
>
> Thank you in advance,
> Junhwan Choi
> _______________________________________________
> yt-users mailing list
> yt-users at lists.spacepope.org
> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________
yt-users mailing list
yt-users at lists.spacepope.org
http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org




More information about the yt-users mailing list