[yt-users] Athena phase plot problem
Stephanie Tonnesen
stonnes at gmail.com
Fri Dec 21 21:01:11 PST 2012
Hi again,
Okay, I still don't know what is going on, but have run a couple tests to
try and narrow it down. So, I was trying to focus on the fact that using
all 343 grids,
alld = pf.h.all_data()
tracerp25 = alld.cut_region(["grid['specific_scalar[0]'] > 0.25"])
pc.add_phase_object(tracerp25,["z","velocity_z","CellMass"],weight=None)
fails. But using a single grid or 2 grids I *can* add that phase_object.
So I checked to see if it worked if I used the upper 245 grids (removed the
two 7x7 x-y planes at the lowest z values). Did NOT work. But if I ALSO
removed the top 7x7 x-y plane, it works (196 grids). If I only remove the
top 7x7 x-y plane, it does NOT work. If I remove the top and bottom x-y
planes, it also does NOT work.
However, an alld.cut_region(["grid['density'] > 10."]) is always able to
make a phase_object.
Is this the clue that makes is all make sense to anyone?
Stephanie
On Tue, Dec 18, 2012 at 9:36 PM, Stephanie Tonnesen <stonnes at gmail.com>wrote:
> Hi John,
>
> Yup, definitely a memory issue, and yes those dimensions are right. It is
> a uniform grid--I think you are right that yt and SMR don't mix yet.
>
> Thanks for working on this problem with me!
>
> Stephanie
>
>
>
> On Tue, Dec 18, 2012 at 4:41 PM, John ZuHone <jzuhone at gmail.com> wrote:
>
>> Hi Stephanie,
>>
>> Well, this particular thing is definitely a memory issue. It looks like
>> it failed when trying to open the x-coordinate dataset for all_data(). By
>> itself, this is about 7 GB, given the domain dimensions that you gave. By
>> the way, do those look right (1048,1048,950)?
>>
>> By the way, is this uniform grid or does this have SMR? I don't think
>> that we've verified that SMR works.
>>
>> I'm sorry I'm not able to be more definitive yet--let me think on it a
>> bit more.
>>
>> Best,
>>
>> John
>>
>> On Dec 18, 2012, at 4:27 PM, Stephanie Tonnesen <stonnes at gmail.com>
>> wrote:
>>
>> Hi John,
>>
>> The weighted average quantity call of specific_scalar[0] worked.
>>
>> The domain info:
>>
>> pf.domain_right_edge
>> array([ 3.2, 3.2, 3.8])
>> >>> pf.domain_dimensions
>> array([1048, 1048, 950])
>>
>> And then using id0 and id1:
>>
>> pf2.domain_right_edge
>> array([-1.349618, -2.259542, -1.145263])
>> >>> pf2.domain_dimensions
>> array([303, 154, 140])
>>
>> When I combined the vtk files into a single 25 GB file, I get an error
>> when creating the PlotCollection:
>>
>> pc = PlotCollection(pf)
>> Traceback (most recent call last):
>> File "<stdin>", line 1, in <module>
>> File
>> "/u/stonnes/Installs/yt-x86_64/src/yt-hg/yt/visualization/plot_collection.py",
>> line 121, in __init__
>> v,self.c = pf.h.find_max("Density") # @todo: ensure no caching
>> File
>> "/u/stonnes/Installs/yt-x86_64/src/yt-hg/yt/data_objects/object_finding_mixin.py",
>> line 61, in find_max
>> mg, mc, mv, pos = self.find_max_cell_location(field, finest_levels)
>> File
>> "/u/stonnes/Installs/yt-x86_64/src/yt-hg/yt/data_objects/object_finding_mixin.py",
>> line 74, in find_max_cell_location
>> source.quantities["MaxLocation"]( field, lazy_reader=True)
>> File
>> "/u/stonnes/Installs/yt-x86_64/src/yt-hg/yt/data_objects/derived_quantities.py",
>> line 92, in __call__
>> return self._call_func_lazy(args, kwargs)
>> File
>> "/u/stonnes/Installs/yt-x86_64/src/yt-hg/yt/data_objects/derived_quantities.py",
>> line 99, in _call_func_lazy
>> rv = self.func(GridChildMaskWrapper(g, self._data_source), *args,
>> **kwargs)
>> File
>> "/u/stonnes/Installs/yt-x86_64/src/yt-hg/yt/data_objects/derived_quantities.py",
>> line 649, in _MaxLocation
>> mx, my, mz = [data[ax][maxi] for ax in 'xyz']
>> File
>> "/u/stonnes/Installs/yt-x86_64/src/yt-hg/yt/data_objects/derived_quantities.py",
>> line 60, in __getitem__
>> data = self.data_source._get_data_from_grid(self.grid, item)
>> File
>> "/u/stonnes/Installs/yt-x86_64/src/yt-hg/yt/data_objects/data_containers.py",
>> line 93, in save_state
>> tr = func(self, grid, field, *args, **kwargs)
>> File
>> "/u/stonnes/Installs/yt-x86_64/src/yt-hg/yt/data_objects/data_containers.py",
>> line 2630, in _get_data_from_grid
>> tr = grid[field]
>> File
>> "/u/stonnes/Installs/yt-x86_64/src/yt-hg/yt/data_objects/grid_patch.py",
>> line 157, in __getitem__
>> self.get_data(key)
>> File
>> "/u/stonnes/Installs/yt-x86_64/src/yt-hg/yt/data_objects/grid_patch.py",
>> line 200, in get_data
>> self._generate_field(field)
>> File
>> "/u/stonnes/Installs/yt-x86_64/src/yt-hg/yt/data_objects/grid_patch.py",
>> line 145, in _generate_field
>> self[field] = self.pf.field_info[field](self)
>> File
>> "/u/stonnes/Installs/yt-x86_64/src/yt-hg/yt/data_objects/field_info_container.py",
>> line 378, in __call__
>> dd = self._function(self, data)
>> File
>> "/u/stonnes/Installs/yt-x86_64/src/yt-hg/yt/data_objects/universal_fields.py",
>> line 94, in _coordX
>> return (np.ones(data.ActiveDimensions, dtype='float64')
>> File
>> "/u/stonnes/Installs/yt-x86_64/lib/python2.7/site-packages/numpy/core/numeric.py",
>> line 1819, in ones
>> a = empty(shape, dtype, order)
>> MemoryError
>>
>>
>> On Tue, Dec 18, 2012 at 1:49 PM, John ZuHone <jzuhone at gmail.com> wrote:
>>
>>> Hi Stephanie,
>>>
>>> I doubt that 25 GB is too big.
>>>
>>> Can you also try a weighted average quantity of specific_scalar[0]
>>> itself?
>>>
>>> When you use only id0 and id1, you are potentially also changing
>>> domain_right_edge, since for Athena yt has to go through all of the id*/*
>>> to find the largest right edge for any grid to determine domain_right_edge.
>>>
>>> Can you figure out what pf.domain_right_edge and pf.domain_dimensions
>>> are for the two different cases (where you have all the data and also where
>>> you have just id0 and id1)?
>>>
>>> I'm not sure at the moment what that would tell us, but I'm curious
>>> about it.
>>>
>>> As another possible way to get some clues, I am aware that Athena has a
>>> tool that joins the vtk files into one. Is it possible to run that and then
>>> try the same script on the merged file?
>>>
>>> I'm also wondering if Matt and Sam have any thoughts on this.
>>>
>>> Best,
>>>
>>> John
>>>
>>> On Dec 18, 2012, at 1:21 PM, Stephanie Tonnesen <stonnes at gmail.com>
>>> wrote:
>>>
>>> Hi John,
>>>
>>> So, what I mean by "expected" variables are basically all variables BUT
>>> specific_scalar[0]. When I read in the vtk file, you can see that the
>>> bottom line says
>>>
>>> yt : [INFO ] 2012-12-13 23:27:13,214 Adding unknown field specific_scalar[0] to list of fields
>>>
>>>
>>> Yes, I can do any of the "quantities" commands, for example:
>>>
>>> alld = pf.h.all_data()
>>> tracerp25 = alld.cut_region(["grid['specific_scalar[0]'] > 0.25",
>>> "grid['z'] > 0.01"])
>>>
>>> radmean = tracerp25.quantities["WeightedAverageQuantity"]("Radius","Ones")
>>>
>>> Slicing also works:
>>>
>>> p = SlicePlot(pf,"x","specific_scalar[0]")
>>>
>>> Now I am starting to worry that it is because output is so big--each vtk
>>> file is 72 MB, so a full output is almost 25 GB. When I just use two
>>> outputs (moving them so yt only sees id0 and id1 folders) the phase plot
>>> works. But, after running the add_phase_object command on the whole data
>>> set for more than 1.5 hours I still have no result and my computer keeps
>>> freezing--although top shows that only 42% of memory is being used.
>>>
>>> Is it possible that my output is just too big?
>>>
>>> Stephanie
>>>
>>>
>>> On Tue, Dec 18, 2012 at 11:00 AM, John Zuhone <jzuhone at gmail.com> wrote:
>>>
>>>> Hi Stephanie,
>>>>
>>>> Ok, thanks for checking that.
>>>>
>>>> I guess I'm not sure what you mean by the "expected" set of variables.
>>>> Do you mean the ones that yt defines, or the ones that are in the field
>>>> list? It looks like "velocity_z" works for one call and doesn't for
>>>> another.
>>>>
>>>> Are you able to do anything else with "specific_scalar[0]"? Like make a
>>>> slice, or a projection of some sort? Or maybe take the weighted average of
>>>> it in a sphere. I'd be interested in seeing if the problem was restricted
>>>> to phase plotting.
>>>>
>>>> How big is the dataset, by the way? Is there something that fails in
>>>> the same way that the both of us can test on?
>>>>
>>>> Best,
>>>>
>>>> John
>>>>
>>>>
>>>> On Tue, Dec 18, 2012 at 10:53 AM, Stephanie Tonnesen <stonnes at gmail.com
>>>> > wrote:
>>>>
>>>>>
>>>>> Hi,
>>>>>
>>>>> Yup. Just to be sure I updated again and reattempted to make the
>>>>> phase_object and python was Killed again.
>>>>>
>>>>> Thanks!
>>>>>
>>>>> Stephanie
>>>>>
>>>>>
>>>>>
>>>>> On Tue, Dec 18, 2012 at 12:35 AM, John ZuHone <jzuhone at gmail.com>wrote:
>>>>>
>>>>>> Hi Stephanie,
>>>>>>
>>>>>> Just want to make sure that you're using the latest version of the yt
>>>>>> development tree--it contains some important Athena fixes that were put in
>>>>>> a few weeks back.
>>>>>>
>>>>>> If so, then we'll work from there to figure out what's gone wrong.
>>>>>>
>>>>>> Best,
>>>>>>
>>>>>> John
>>>>>>
>>>>>> On Dec 17, 2012, at 5:44 PM, Stephanie Tonnesen <stonnes at gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>> Hi yt-users,
>>>>>>
>>>>>> I am trying to make phase_objects of Athena data. I am able to make
>>>>>> phase_objects if I use the "expected" set of variables (will show below),
>>>>>> but kill my kernel or am kicked out of python if I try and use an "unknown
>>>>>> field" read in from a set of vtk files.
>>>>>>
>>>>>>
>>>>>>
>>>>>> from yt.mods import *
>>>>>>
>>>>>> pf = load("id0/rps.0080.vtk")
>>>>>>
>>>>>> yt : [INFO ] 2012-12-13 23:27:08,678 Temporarily setting domain_right_edge = -domain_left_edge. This will be corrected automatically if it is not the case.
>>>>>> yt : [INFO ] 2012-12-13 23:27:08,974 Parameters: current_time = 40.00096
>>>>>> yt : [INFO ] 2012-12-13 23:27:08,974 Parameters: domain_dimensions = [1048 1048 655]
>>>>>> yt : [INFO ] 2012-12-13 23:27:08,975 Parameters: domain_left_edge = [-3.2 -3.2 -2. ]
>>>>>> yt : [INFO ] 2012-12-13 23:27:08,976 Parameters: domain_right_edge = [ 3.2 3.2 2. ]
>>>>>> yt : [INFO ] 2012-12-13 23:27:08,977 Parameters: cosmological_simulation = 0.0
>>>>>> yt : [INFO ] 2012-12-13 23:27:13,214 Adding unknown field specific_scalar[0] to list of fields
>>>>>>
>>>>>> pf.h.field_list
>>>>>>
>>>>>> ['density',
>>>>>> 'pressure',
>>>>>> 'velocity_z',
>>>>>> 'velocity_x',
>>>>>> 'velocity_y',
>>>>>> 'specific_scalar[0]']
>>>>>>
>>>>>> alld = pf.h.all_data()
>>>>>> tracerp25 = alld.cut_region(["grid['specific_scalar[0]'] > 0.25",
>>>>>> "grid['z'] > 0.01"])
>>>>>>
>>>>>> ##These commands works:
>>>>>>
>>>>>> radmean = tracerp25.quantities["WeightedAverageQuantity"]("Radius","Ones")
>>>>>>
>>>>>> pc = PlotCollectionInteractive(pf)
>>>>>>
>>>>>> pc.add_phase_object(alld,["z","velocity_z","CellMass"],weight=None)
>>>>>>
>>>>>> #The calls below fail. There is no error message. If I am in iPython,
>>>>>> #the kernel dies. If I am running on a computer, it freezes and
>>>>>> #eventually I just get the message
>>>>>> #Killed
>>>>>>
>>>>>> pc.add_phase_object(alld,["z","specific_scalar[0]","CellMass"],weight=None)
>>>>>> pc.add_phase_object(tracerp25,["z","velocity_z","CellMass"],weight=None)
>>>>>>
>>>>>> #If I read in a single vtk file ("id122/rps-id122.0080.vtk") the above
>>>>>> #commands work
>>>>>>
>>>>>> Any help would be much appreciated! Thanks,
>>>>>>
>>>>>> Stephanie
>>>>>> _______________________________________________
>>>>>> yt-users mailing list
>>>>>> yt-users at lists.spacepope.org
>>>>>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>>>>>>
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> yt-users mailing list
>>>>>> yt-users at lists.spacepope.org
>>>>>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>>>>>>
>>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> yt-users mailing list
>>>>> yt-users at lists.spacepope.org
>>>>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> John ZuHone
>>>>
>>>> Postdoctoral Researcher
>>>> NASA/Goddard Space Flight Center
>>>>
>>>> jzuhone at gmail.com
>>>> john.zuhone at nasa.gov
>>>>
>>>>
>>>> _______________________________________________
>>>> yt-users mailing list
>>>> yt-users at lists.spacepope.org
>>>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>>>>
>>>>
>>> _______________________________________________
>>> yt-users mailing list
>>> yt-users at lists.spacepope.org
>>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>>>
>>>
>>>
>>> _______________________________________________
>>> yt-users mailing list
>>> yt-users at lists.spacepope.org
>>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>>>
>>>
>> _______________________________________________
>> yt-users mailing list
>> yt-users at lists.spacepope.org
>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>>
>>
>>
>> _______________________________________________
>> yt-users mailing list
>> yt-users at lists.spacepope.org
>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.spacepope.org/pipermail/yt-users-spacepope.org/attachments/20121222/368e8b9b/attachment.htm>
More information about the yt-users
mailing list