[yt-users] Athena phase plot problem
Stephanie Tonnesen
stonnes at gmail.com
Tue Dec 18 18:36:24 PST 2012
Hi John,
Yup, definitely a memory issue, and yes those dimensions are right. It is
a uniform grid--I think you are right that yt and SMR don't mix yet.
Thanks for working on this problem with me!
Stephanie
On Tue, Dec 18, 2012 at 4:41 PM, John ZuHone <jzuhone at gmail.com> wrote:
> Hi Stephanie,
>
> Well, this particular thing is definitely a memory issue. It looks like it
> failed when trying to open the x-coordinate dataset for all_data(). By
> itself, this is about 7 GB, given the domain dimensions that you gave. By
> the way, do those look right (1048,1048,950)?
>
> By the way, is this uniform grid or does this have SMR? I don't think that
> we've verified that SMR works.
>
> I'm sorry I'm not able to be more definitive yet--let me think on it a bit
> more.
>
> Best,
>
> John
>
> On Dec 18, 2012, at 4:27 PM, Stephanie Tonnesen <stonnes at gmail.com> wrote:
>
> Hi John,
>
> The weighted average quantity call of specific_scalar[0] worked.
>
> The domain info:
>
> pf.domain_right_edge
> array([ 3.2, 3.2, 3.8])
> >>> pf.domain_dimensions
> array([1048, 1048, 950])
>
> And then using id0 and id1:
>
> pf2.domain_right_edge
> array([-1.349618, -2.259542, -1.145263])
> >>> pf2.domain_dimensions
> array([303, 154, 140])
>
> When I combined the vtk files into a single 25 GB file, I get an error
> when creating the PlotCollection:
>
> pc = PlotCollection(pf)
> Traceback (most recent call last):
> File "<stdin>", line 1, in <module>
> File
> "/u/stonnes/Installs/yt-x86_64/src/yt-hg/yt/visualization/plot_collection.py",
> line 121, in __init__
> v,self.c = pf.h.find_max("Density") # @todo: ensure no caching
> File
> "/u/stonnes/Installs/yt-x86_64/src/yt-hg/yt/data_objects/object_finding_mixin.py",
> line 61, in find_max
> mg, mc, mv, pos = self.find_max_cell_location(field, finest_levels)
> File
> "/u/stonnes/Installs/yt-x86_64/src/yt-hg/yt/data_objects/object_finding_mixin.py",
> line 74, in find_max_cell_location
> source.quantities["MaxLocation"]( field, lazy_reader=True)
> File
> "/u/stonnes/Installs/yt-x86_64/src/yt-hg/yt/data_objects/derived_quantities.py",
> line 92, in __call__
> return self._call_func_lazy(args, kwargs)
> File
> "/u/stonnes/Installs/yt-x86_64/src/yt-hg/yt/data_objects/derived_quantities.py",
> line 99, in _call_func_lazy
> rv = self.func(GridChildMaskWrapper(g, self._data_source), *args,
> **kwargs)
> File
> "/u/stonnes/Installs/yt-x86_64/src/yt-hg/yt/data_objects/derived_quantities.py",
> line 649, in _MaxLocation
> mx, my, mz = [data[ax][maxi] for ax in 'xyz']
> File
> "/u/stonnes/Installs/yt-x86_64/src/yt-hg/yt/data_objects/derived_quantities.py",
> line 60, in __getitem__
> data = self.data_source._get_data_from_grid(self.grid, item)
> File
> "/u/stonnes/Installs/yt-x86_64/src/yt-hg/yt/data_objects/data_containers.py",
> line 93, in save_state
> tr = func(self, grid, field, *args, **kwargs)
> File
> "/u/stonnes/Installs/yt-x86_64/src/yt-hg/yt/data_objects/data_containers.py",
> line 2630, in _get_data_from_grid
> tr = grid[field]
> File
> "/u/stonnes/Installs/yt-x86_64/src/yt-hg/yt/data_objects/grid_patch.py",
> line 157, in __getitem__
> self.get_data(key)
> File
> "/u/stonnes/Installs/yt-x86_64/src/yt-hg/yt/data_objects/grid_patch.py",
> line 200, in get_data
> self._generate_field(field)
> File
> "/u/stonnes/Installs/yt-x86_64/src/yt-hg/yt/data_objects/grid_patch.py",
> line 145, in _generate_field
> self[field] = self.pf.field_info[field](self)
> File
> "/u/stonnes/Installs/yt-x86_64/src/yt-hg/yt/data_objects/field_info_container.py",
> line 378, in __call__
> dd = self._function(self, data)
> File
> "/u/stonnes/Installs/yt-x86_64/src/yt-hg/yt/data_objects/universal_fields.py",
> line 94, in _coordX
> return (np.ones(data.ActiveDimensions, dtype='float64')
> File
> "/u/stonnes/Installs/yt-x86_64/lib/python2.7/site-packages/numpy/core/numeric.py",
> line 1819, in ones
> a = empty(shape, dtype, order)
> MemoryError
>
>
> On Tue, Dec 18, 2012 at 1:49 PM, John ZuHone <jzuhone at gmail.com> wrote:
>
>> Hi Stephanie,
>>
>> I doubt that 25 GB is too big.
>>
>> Can you also try a weighted average quantity of specific_scalar[0]
>> itself?
>>
>> When you use only id0 and id1, you are potentially also changing
>> domain_right_edge, since for Athena yt has to go through all of the id*/*
>> to find the largest right edge for any grid to determine domain_right_edge.
>>
>> Can you figure out what pf.domain_right_edge and pf.domain_dimensions are
>> for the two different cases (where you have all the data and also where you
>> have just id0 and id1)?
>>
>> I'm not sure at the moment what that would tell us, but I'm curious about
>> it.
>>
>> As another possible way to get some clues, I am aware that Athena has a
>> tool that joins the vtk files into one. Is it possible to run that and then
>> try the same script on the merged file?
>>
>> I'm also wondering if Matt and Sam have any thoughts on this.
>>
>> Best,
>>
>> John
>>
>> On Dec 18, 2012, at 1:21 PM, Stephanie Tonnesen <stonnes at gmail.com>
>> wrote:
>>
>> Hi John,
>>
>> So, what I mean by "expected" variables are basically all variables BUT
>> specific_scalar[0]. When I read in the vtk file, you can see that the
>> bottom line says
>>
>> yt : [INFO ] 2012-12-13 23:27:13,214 Adding unknown field specific_scalar[0] to list of fields
>>
>>
>> Yes, I can do any of the "quantities" commands, for example:
>>
>> alld = pf.h.all_data()
>> tracerp25 = alld.cut_region(["grid['specific_scalar[0]'] > 0.25",
>> "grid['z'] > 0.01"])
>>
>> radmean = tracerp25.quantities["WeightedAverageQuantity"]("Radius","Ones")
>>
>> Slicing also works:
>>
>> p = SlicePlot(pf,"x","specific_scalar[0]")
>>
>> Now I am starting to worry that it is because output is so big--each vtk
>> file is 72 MB, so a full output is almost 25 GB. When I just use two
>> outputs (moving them so yt only sees id0 and id1 folders) the phase plot
>> works. But, after running the add_phase_object command on the whole data
>> set for more than 1.5 hours I still have no result and my computer keeps
>> freezing--although top shows that only 42% of memory is being used.
>>
>> Is it possible that my output is just too big?
>>
>> Stephanie
>>
>>
>> On Tue, Dec 18, 2012 at 11:00 AM, John Zuhone <jzuhone at gmail.com> wrote:
>>
>>> Hi Stephanie,
>>>
>>> Ok, thanks for checking that.
>>>
>>> I guess I'm not sure what you mean by the "expected" set of variables.
>>> Do you mean the ones that yt defines, or the ones that are in the field
>>> list? It looks like "velocity_z" works for one call and doesn't for
>>> another.
>>>
>>> Are you able to do anything else with "specific_scalar[0]"? Like make a
>>> slice, or a projection of some sort? Or maybe take the weighted average of
>>> it in a sphere. I'd be interested in seeing if the problem was restricted
>>> to phase plotting.
>>>
>>> How big is the dataset, by the way? Is there something that fails in the
>>> same way that the both of us can test on?
>>>
>>> Best,
>>>
>>> John
>>>
>>>
>>> On Tue, Dec 18, 2012 at 10:53 AM, Stephanie Tonnesen <stonnes at gmail.com>wrote:
>>>
>>>>
>>>> Hi,
>>>>
>>>> Yup. Just to be sure I updated again and reattempted to make the
>>>> phase_object and python was Killed again.
>>>>
>>>> Thanks!
>>>>
>>>> Stephanie
>>>>
>>>>
>>>>
>>>> On Tue, Dec 18, 2012 at 12:35 AM, John ZuHone <jzuhone at gmail.com>wrote:
>>>>
>>>>> Hi Stephanie,
>>>>>
>>>>> Just want to make sure that you're using the latest version of the yt
>>>>> development tree--it contains some important Athena fixes that were put in
>>>>> a few weeks back.
>>>>>
>>>>> If so, then we'll work from there to figure out what's gone wrong.
>>>>>
>>>>> Best,
>>>>>
>>>>> John
>>>>>
>>>>> On Dec 17, 2012, at 5:44 PM, Stephanie Tonnesen <stonnes at gmail.com>
>>>>> wrote:
>>>>>
>>>>> Hi yt-users,
>>>>>
>>>>> I am trying to make phase_objects of Athena data. I am able to make
>>>>> phase_objects if I use the "expected" set of variables (will show below),
>>>>> but kill my kernel or am kicked out of python if I try and use an "unknown
>>>>> field" read in from a set of vtk files.
>>>>>
>>>>>
>>>>>
>>>>> from yt.mods import *
>>>>>
>>>>> pf = load("id0/rps.0080.vtk")
>>>>>
>>>>> yt : [INFO ] 2012-12-13 23:27:08,678 Temporarily setting domain_right_edge = -domain_left_edge. This will be corrected automatically if it is not the case.
>>>>> yt : [INFO ] 2012-12-13 23:27:08,974 Parameters: current_time = 40.00096
>>>>> yt : [INFO ] 2012-12-13 23:27:08,974 Parameters: domain_dimensions = [1048 1048 655]
>>>>> yt : [INFO ] 2012-12-13 23:27:08,975 Parameters: domain_left_edge = [-3.2 -3.2 -2. ]
>>>>> yt : [INFO ] 2012-12-13 23:27:08,976 Parameters: domain_right_edge = [ 3.2 3.2 2. ]
>>>>> yt : [INFO ] 2012-12-13 23:27:08,977 Parameters: cosmological_simulation = 0.0
>>>>> yt : [INFO ] 2012-12-13 23:27:13,214 Adding unknown field specific_scalar[0] to list of fields
>>>>>
>>>>> pf.h.field_list
>>>>>
>>>>> ['density',
>>>>> 'pressure',
>>>>> 'velocity_z',
>>>>> 'velocity_x',
>>>>> 'velocity_y',
>>>>> 'specific_scalar[0]']
>>>>>
>>>>> alld = pf.h.all_data()
>>>>> tracerp25 = alld.cut_region(["grid['specific_scalar[0]'] > 0.25",
>>>>> "grid['z'] > 0.01"])
>>>>>
>>>>> ##These commands works:
>>>>>
>>>>> radmean = tracerp25.quantities["WeightedAverageQuantity"]("Radius","Ones")
>>>>>
>>>>> pc = PlotCollectionInteractive(pf)
>>>>>
>>>>> pc.add_phase_object(alld,["z","velocity_z","CellMass"],weight=None)
>>>>>
>>>>> #The calls below fail. There is no error message. If I am in iPython,
>>>>> #the kernel dies. If I am running on a computer, it freezes and
>>>>> #eventually I just get the message
>>>>> #Killed
>>>>>
>>>>> pc.add_phase_object(alld,["z","specific_scalar[0]","CellMass"],weight=None)
>>>>> pc.add_phase_object(tracerp25,["z","velocity_z","CellMass"],weight=None)
>>>>>
>>>>> #If I read in a single vtk file ("id122/rps-id122.0080.vtk") the above
>>>>> #commands work
>>>>>
>>>>> Any help would be much appreciated! Thanks,
>>>>>
>>>>> Stephanie
>>>>> _______________________________________________
>>>>> yt-users mailing list
>>>>> yt-users at lists.spacepope.org
>>>>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>>>>>
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> yt-users mailing list
>>>>> yt-users at lists.spacepope.org
>>>>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>>>>>
>>>>>
>>>>
>>>> _______________________________________________
>>>> yt-users mailing list
>>>> yt-users at lists.spacepope.org
>>>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>>>>
>>>>
>>>
>>>
>>> --
>>> John ZuHone
>>>
>>> Postdoctoral Researcher
>>> NASA/Goddard Space Flight Center
>>>
>>> jzuhone at gmail.com
>>> john.zuhone at nasa.gov
>>>
>>>
>>> _______________________________________________
>>> yt-users mailing list
>>> yt-users at lists.spacepope.org
>>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>>>
>>>
>> _______________________________________________
>> yt-users mailing list
>> yt-users at lists.spacepope.org
>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>>
>>
>>
>> _______________________________________________
>> yt-users mailing list
>> yt-users at lists.spacepope.org
>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>>
>>
> _______________________________________________
> yt-users mailing list
> yt-users at lists.spacepope.org
> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>
>
>
> _______________________________________________
> yt-users mailing list
> yt-users at lists.spacepope.org
> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.spacepope.org/pipermail/yt-users-spacepope.org/attachments/20121218/6746a04b/attachment.htm>
More information about the yt-users
mailing list