[yt-users] Athena phase plot problem

John ZuHone jzuhone at gmail.com
Tue Dec 18 13:41:08 PST 2012


Hi Stephanie,

Well, this particular thing is definitely a memory issue. It looks like it failed when trying to open the x-coordinate dataset for all_data(). By itself, this is about 7 GB, given the domain dimensions that you gave. By the way, do those look right (1048,1048,950)?

By the way, is this uniform grid or does this have SMR? I don't think that we've verified that SMR works. 

I'm sorry I'm not able to be more definitive yet--let me think on it a bit more. 

Best,

John

On Dec 18, 2012, at 4:27 PM, Stephanie Tonnesen <stonnes at gmail.com> wrote:

> Hi John,
> 
> The weighted average quantity call of specific_scalar[0] worked.
> 
> The domain info:
> 
> pf.domain_right_edge
> array([ 3.2,  3.2,  3.8])
> >>> pf.domain_dimensions
> array([1048, 1048,  950])
> 
> And then using id0 and id1:
> 
>  pf2.domain_right_edge
> array([-1.349618, -2.259542, -1.145263])
> >>> pf2.domain_dimensions
> array([303, 154, 140])
> 
> When I combined the vtk files into a single 25 GB file, I get an error when creating the PlotCollection:
> 
>  pc = PlotCollection(pf)
> Traceback (most recent call last):
>   File "<stdin>", line 1, in <module>
>   File "/u/stonnes/Installs/yt-x86_64/src/yt-hg/yt/visualization/plot_collection.py", line 121, in __init__
>     v,self.c = pf.h.find_max("Density") # @todo: ensure no caching
>   File "/u/stonnes/Installs/yt-x86_64/src/yt-hg/yt/data_objects/object_finding_mixin.py", line 61, in find_max
>     mg, mc, mv, pos = self.find_max_cell_location(field, finest_levels)
>   File "/u/stonnes/Installs/yt-x86_64/src/yt-hg/yt/data_objects/object_finding_mixin.py", line 74, in find_max_cell_location
>     source.quantities["MaxLocation"]( field, lazy_reader=True)
>   File "/u/stonnes/Installs/yt-x86_64/src/yt-hg/yt/data_objects/derived_quantities.py", line 92, in __call__
>     return self._call_func_lazy(args, kwargs)
>   File "/u/stonnes/Installs/yt-x86_64/src/yt-hg/yt/data_objects/derived_quantities.py", line 99, in _call_func_lazy
>     rv = self.func(GridChildMaskWrapper(g, self._data_source), *args, **kwargs)
>   File "/u/stonnes/Installs/yt-x86_64/src/yt-hg/yt/data_objects/derived_quantities.py", line 649, in _MaxLocation
>     mx, my, mz = [data[ax][maxi] for ax in 'xyz']
>   File "/u/stonnes/Installs/yt-x86_64/src/yt-hg/yt/data_objects/derived_quantities.py", line 60, in __getitem__
>     data = self.data_source._get_data_from_grid(self.grid, item)
>   File "/u/stonnes/Installs/yt-x86_64/src/yt-hg/yt/data_objects/data_containers.py", line 93, in save_state
>     tr = func(self, grid, field, *args, **kwargs)
>   File "/u/stonnes/Installs/yt-x86_64/src/yt-hg/yt/data_objects/data_containers.py", line 2630, in _get_data_from_grid
>     tr = grid[field]
>   File "/u/stonnes/Installs/yt-x86_64/src/yt-hg/yt/data_objects/grid_patch.py", line 157, in __getitem__
>     self.get_data(key)
>   File "/u/stonnes/Installs/yt-x86_64/src/yt-hg/yt/data_objects/grid_patch.py", line 200, in get_data
>     self._generate_field(field)
>   File "/u/stonnes/Installs/yt-x86_64/src/yt-hg/yt/data_objects/grid_patch.py", line 145, in _generate_field
>     self[field] = self.pf.field_info[field](self)
>   File "/u/stonnes/Installs/yt-x86_64/src/yt-hg/yt/data_objects/field_info_container.py", line 378, in __call__
>     dd = self._function(self, data)
>   File "/u/stonnes/Installs/yt-x86_64/src/yt-hg/yt/data_objects/universal_fields.py", line 94, in _coordX
>     return (np.ones(data.ActiveDimensions, dtype='float64')
>   File "/u/stonnes/Installs/yt-x86_64/lib/python2.7/site-packages/numpy/core/numeric.py", line 1819, in ones
>     a = empty(shape, dtype, order)
> MemoryError
> 
> 
> On Tue, Dec 18, 2012 at 1:49 PM, John ZuHone <jzuhone at gmail.com> wrote:
> Hi Stephanie,
> 
> I doubt that 25 GB is too big.
> 
> Can you also try a weighted average quantity of specific_scalar[0] itself?
> 
> When you use only id0 and id1, you are potentially also changing domain_right_edge, since for Athena yt has to go through all of the id*/* to find the largest right edge for any grid to determine domain_right_edge. 
> 
> Can you figure out what pf.domain_right_edge and pf.domain_dimensions are for the two different cases (where you have all the data and also where you have just id0 and id1)?
> 
> I'm not sure at the moment what that would tell us, but I'm curious about it.
> 
> As another possible way to get some clues, I am aware that Athena has a tool that joins the vtk files into one. Is it possible to run that and then try the same script on the merged file?
> 
> I'm also wondering if Matt and Sam have any thoughts on this.
> 
> Best,
> 
> John
> 
> On Dec 18, 2012, at 1:21 PM, Stephanie Tonnesen <stonnes at gmail.com> wrote:
> 
>> Hi John,
>> 
>> So, what I mean by "expected" variables are basically all variables BUT specific_scalar[0].  When I read in the vtk file, you can see that the bottom line says 
>> 
>> yt : [INFO     ] 2012-12-13 23:27:13,214 Adding unknown field specific_scalar[0] to list of fields
>> 
>> 
>> 
>> Yes, I can do any of the "quantities" commands, for example:
>> 
>> alld = pf.h.all_data()
>> tracerp25 = alld.cut_region(["grid['specific_scalar[0]'] > 0.25", "grid['z'] > 0.01"])
>> radmean = tracerp25.quantities["WeightedAverageQuantity"]("Radius","Ones")
>> 
>> Slicing also works:
>> 
>> p = SlicePlot(pf,"x","specific_scalar[0]")
>> 
>> Now I am starting to worry that it is because output is so big--each vtk file is 72 MB, so a full output is almost 25 GB.  When I just use two outputs (moving them so yt only sees id0 and id1 folders) the phase plot works.  But, after running the add_phase_object command on the whole data set for more than 1.5 hours I still have no result and my computer keeps freezing--although top shows that only 42% of memory is being used.  
>> 
>> Is it possible that my output is just too big?
>> 
>> Stephanie
>> 
>> 
>> On Tue, Dec 18, 2012 at 11:00 AM, John Zuhone <jzuhone at gmail.com> wrote:
>> Hi Stephanie,
>> 
>> Ok, thanks for checking that. 
>> 
>> I guess I'm not sure what you mean by the "expected" set of variables. Do you mean the ones that yt defines, or the ones that are in the field list? It looks like "velocity_z" works for one call and doesn't for another. 
>> 
>> Are you able to do anything else with "specific_scalar[0]"? Like make a slice, or a projection of some sort? Or maybe take the weighted average of it in a sphere. I'd be interested in seeing if the problem was restricted to phase plotting. 
>> 
>> How big is the dataset, by the way? Is there something that fails in the same way that the both of us can test on? 
>> 
>> Best,
>> 
>> John
>> 
>> 
>> On Tue, Dec 18, 2012 at 10:53 AM, Stephanie Tonnesen <stonnes at gmail.com> wrote:
>> 
>> Hi,
>> 
>> Yup.  Just to be sure I updated again and reattempted to make the phase_object and python was Killed again.
>> 
>> Thanks!
>> 
>> Stephanie
>> 
>> 
>> 
>> On Tue, Dec 18, 2012 at 12:35 AM, John ZuHone <jzuhone at gmail.com> wrote:
>> Hi Stephanie,
>> 
>> Just want to make sure that you're using the latest version of the yt development tree--it contains some important Athena fixes that were put in a few weeks back.
>> 
>> If so, then we'll work from there to figure out what's gone wrong. 
>> 
>> Best,
>> 
>> John
>> 
>> On Dec 17, 2012, at 5:44 PM, Stephanie Tonnesen <stonnes at gmail.com> wrote:
>> 
>>> Hi yt-users,
>>> 
>>> I am trying to make phase_objects of Athena data.  I am able to make phase_objects if I use the "expected" set of variables (will show below), but kill my kernel or am kicked out of python if I try and use an "unknown field" read in from a set of vtk files.  
>>> 
>>> 
>>> 
>>> from yt.mods import *
>>> 
>>> pf = load("id0/rps.0080.vtk")
>>> 
>>> yt : [INFO     ] 2012-12-13 23:27:08,678 Temporarily setting domain_right_edge = -domain_left_edge. This will be corrected automatically if it is not the case.
>>> yt : [INFO     ] 2012-12-13 23:27:08,974 Parameters: current_time              = 40.00096
>>> yt : [INFO     ] 2012-12-13 23:27:08,974 Parameters: domain_dimensions         = [1048 1048  655]
>>> yt : [INFO     ] 2012-12-13 23:27:08,975 Parameters: domain_left_edge          = [-3.2 -3.2 -2. ]
>>> yt : [INFO     ] 2012-12-13 23:27:08,976 Parameters: domain_right_edge         = [ 3.2  3.2  2. ]
>>> yt : [INFO     ] 2012-12-13 23:27:08,977 Parameters: cosmological_simulation   = 0.0
>>> yt : [INFO     ] 2012-12-13 23:27:13,214 Adding unknown field specific_scalar[0] to list of fields
>>> 
>>> pf.h.field_list
>>> 
>>> ['density',
>>>  'pressure',
>>>  'velocity_z',
>>>  'velocity_x',
>>>  'velocity_y',
>>>  'specific_scalar[0]']
>>> alld = pf.h.all_data()
>>> tracerp25 = alld.cut_region(["grid['specific_scalar[0]'] > 0.25", "grid['z'] > 0.01"])
>>> 
>>> ##These commands works:
>>> radmean = tracerp25.quantities["WeightedAverageQuantity"]("Radius","Ones")
>>> 
>>> pc = PlotCollectionInteractive(pf)
>>> 
>>> pc.add_phase_object(alld,["z","velocity_z","CellMass"],weight=None)
>>> 
>>> #The calls below fail.  There is no error message.  If I am in iPython,
>>> #the kernel dies.  If I am running on a computer, it freezes and
>>> #eventually I just get the message
>>> #Killed
>>> 
>>> pc.add_phase_object(alld,["z","specific_scalar[0]","CellMass"],weight=None)
>>> pc.add_phase_object(tracerp25,["z","velocity_z","CellMass"],weight=None)
>>> 
>>> #If I read in a single vtk file ("id122/rps-id122.0080.vtk") the above
>>> #commands work
>>> Any help would be much appreciated!  Thanks,
>>> 
>>> Stephanie
>>> _______________________________________________
>>> yt-users mailing list
>>> yt-users at lists.spacepope.org
>>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>> 
>> 
>> _______________________________________________
>> yt-users mailing list
>> yt-users at lists.spacepope.org
>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>> 
>> 
>> 
>> _______________________________________________
>> yt-users mailing list
>> yt-users at lists.spacepope.org
>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>> 
>> 
>> 
>> 
>> -- 
>> John ZuHone
>> 
>> Postdoctoral Researcher
>> NASA/Goddard Space Flight Center
>> 
>> jzuhone at gmail.com
>> john.zuhone at nasa.gov
>> 
>> 
>> _______________________________________________
>> yt-users mailing list
>> yt-users at lists.spacepope.org
>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>> 
>> 
>> _______________________________________________
>> yt-users mailing list
>> yt-users at lists.spacepope.org
>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
> 
> 
> _______________________________________________
> yt-users mailing list
> yt-users at lists.spacepope.org
> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
> 
> 
> _______________________________________________
> yt-users mailing list
> yt-users at lists.spacepope.org
> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.spacepope.org/pipermail/yt-users-spacepope.org/attachments/20121218/9a24f906/attachment.html>


More information about the yt-users mailing list