[yt-users] Dark Sky Simulations, analysis gone wrong.

ALEXANDROS TSOUROS ph4602 at edu.physics.uoc.gr
Sat Jul 1 19:41:54 PDT 2017 

Your help has been invaluable!

So, carrying the logic one step further, I assume the correct binning 
field for the job is particle_radius.

I will keep trying to load field_list.

Thanks again!

keep up the good work!


On 2017-07-02 04:26, Nathan Goldbaum wrote:
> I was able to get the following script to work locally using yt
> 3.4-dev. Unfortunately I hit a cython type error on yt 3.3.5 when yt
> tries to read the particle radial velocities off disk.
> 
> https://gist.github.com/ngoldbaum/9506a215c8ae5770e0149dccde8ed012
> 
> I think what was happening with your memory error is that you either
> weren't loading the dataset with the .midx10 file or you weren't
> loading it with a sufficiently small bounding box.
> 
> Back to your original question - "radial_velocity" is a field defined
> for gas fields in AMR or SPH datasets. For particle data like
> dark_skies, you're going to need to use the "particle_radial_velocity"
> field.
> 
> Sorry that took a little while to figure out, I haven't touched the
> dark skies data or thingking in years so it took me a while to figure
> out how to properly load the data.
> 
> On Sat, Jul 1, 2017 at 7:50 PM, Nathan Goldbaum
> <nathan12343 at gmail.com> wrote:
> 
>> Is this one of the public dark skies datasets? How are you loading
>> it (a full script one of us can run locally would be perfect)? What
>> yt version are you using?
>> 
>> On Sat, Jul 1, 2017 at 7:15 PM ALEXANDROS TSOUROS
>> <ph4602 at edu.physics.uoc.gr> wrote:
>> 
>>> I am terribly sorry for not being clear!
>>> 
>>> Gadget has nothing to do with the problem, I merely mentioned it
>>> because
>>> when (in a completely unrelated problem) I use datasets derived
>>> from it,
>>> there's no problem using the 'radial_velocity' field. So Gadget is
>>> irrelevant to the query.
>>> 
>>> The memory error comes by merely invoking the datasets via
>>> thingking and
>>> printing field_lists.
>>> 
>>> Are you aware of anything I can do to get around this?
>>> 
>>> Thank you for your time!
>>> 
>>> On 2017-07-02 02:56, Nathan Goldbaum wrote:
>>>> Sorry, I'm confused.
>>>> 
>>>> In your first email you said you were having issues with the
>>> dark
>>>> skies simulation, but it seems you're trying to load it using
>>> the
>>>> Gadget frontend somehow? Or is this a different Gadget dataset?
>>>> 
>>>> As far as I'm aware, Dark Skies is available in the SDF output
>>> format,
>>>> or streaming over HTTP via thingking.
>>>> 
>>>> If it's just a big Gadget dataset, the issue is that it's too
>>> big to
>>>> load given the memory available on your machine. Right now you
>>> need
>>>> enough memory to store the Morton indices for all the particles
>>> (one
>>>> int64 per particle). We are actively working on improving this
>>>> situation by making yt use compressed bitmaps to store the
>>> Morton
>>>> indices, but that work won't be done for several months at
>>> least.
>>>> 
>>>> You should be able to stream the dark skies data over HTTP
>>> though
>>>> without having a ton of memory, there were some clever tricks
>>> applied
>>>> to get around the memory limit I described above.
>>>> 
>>>> The reason I was asking about the field_list for dark skies is
>>> because
>>>> I'd like to see what fields are available. The error message you
>>> saw
>>>> indicates that one or more of the fields necessary to calculate
>>> the
>>>> radial velocity is not available (or yt is not aware of an alias
>>> from
>>>> a field name on disk to yt's internal field name).
>>>> 
>>>> On Sat, Jul 1, 2017 at 6:46 PM ALEXANDROS TSOUROS
>>>> <ph4602 at edu.physics.uoc.gr> wrote:
>>>> 
>>>>> Thanks for your prompt reply!
>>>>> 
>>>>> So this is another problem. I'm using a 64-bit python
>>> installation,
>>>>> yet
>>>>> the output of 'print ds.file_list' is
>>>>> 
>>>>> "
>>>>> (...)
>>>>> Traceback (most recent call last):
>>>>> File "ds.py", line 5, in <module>
>>>>> print ds.field_list
>>>>> File
>>>>> 
>>>> 
>>> 
>> 
> "/home/atsouros/gadget2/yt/yt-conda/src/yt-git/yt/data_objects/static_output.py",
>>>>> line 545, in field_list
>>>>> return self.index.field_list
>>>>> File
>>>>> 
>>>> 
>>> 
>> 
> "/home/atsouros/gadget2/yt/yt-conda/src/yt-git/yt/data_objects/static_output.py",
>>>>> line 503, in index
>>>>> self, dataset_type=self.dataset_type)
>>>>> File
>>>>> 
>>>> 
>>> 
>> 
> "/home/atsouros/gadget2/yt/yt-conda/src/yt-git/yt/geometry/particle_geometry_handler.py",
>>>>> line 39, in __init__
>>>>> super(ParticleIndex, self).__init__(ds, dataset_type)
>>>>> File
>>>>> 
>>>> 
>>> 
>> 
> "/home/atsouros/gadget2/yt/yt-conda/src/yt-git/yt/geometry/geometry_handler.py",
>>>>> line 50, in __init__
>>>>> self._setup_geometry()
>>>>> File
>>>>> 
>>>> 
>>> 
>> 
> "/home/atsouros/gadget2/yt/yt-conda/src/yt-git/yt/geometry/particle_geometry_handler.py",
>>>>> line 50, in _setup_geometry
>>>>> self._initialize_particle_handler()
>>>>> File
>>>>> 
>>>> 
>>> 
>> 
> "/home/atsouros/gadget2/yt/yt-conda/src/yt-git/yt/geometry/particle_geometry_handler.py",
>>>>> line 104, in _initialize_particle_handler
>>>>> self._initialize_indices()
>>>>> File
>>>>> 
>>>> 
>>> 
>> 
> "/home/atsouros/gadget2/yt/yt-conda/src/yt-git/yt/geometry/particle_geometry_handler.py",
>>>>> line 126, in _initialize_indices
>>>>> morton = np.empty(self.total_particles, dtype="uint64")
>>>>> MemoryError
>>>>> "
>>>>> 
>>>>> I would assume that the problem is that it's running out of
>>> ram. In
>>>>> general, I am able to produce the examples provided by Dark
>>> Sky, so
>>>>> I
>>>>> can in general load data from the dataset. But I cannot use
>>> 'print
>>>>> ds.field_list'.
>>>>> 
>>>>> TL;DR
>>>>> 
>>>>> The output is 'MemoryError'.
>>>>> 
>>>>> On 2017-07-02 01:34, Nathan Goldbaum wrote:
>>>>>> What's the output of "print(ds.field_list)" with the loaded
>>>>> dataset?
>>>>>> 
>>>>>> On Sat, Jul 1, 2017 at 4:21 PM ALEXANDROS TSOUROS
>>>>>> <ph4602 at edu.physics.uoc.gr> wrote:
>>>>>> 
>>>>>>> Hello all!
>>>>>>> 
>>>>>>> I am quite new to yt, so please bear with my ignorance!
>>>>>>> 
>>>>>>> I need to perform analysis on the open data of the Dark Sky
>>>>>>> Simulation
>>>>>>> (http://darksky.slac.stanford.edu [1]). In particular I need
>>> to
>>>>> create 1
>>>>>>> dimensional profiles of the radial velocity of dark matter
>>>>> particles
>>>>>>> around halos.
>>>>>>> 
>>>>>>> Attached, you'll find my attempt at producing such a profile.
>>>>>>> 
>>>>>>> When I do this on a GADGET2 simulation I've run, I can use
>>> the
>>>>>>> 'radial_velocity' field without a problem. However, this
>>> script,
>>>>>>> where
>>>>>>> the data is that of the Dark Sky simulation, produces the
>>>>> following
>>>>>>> output :
>>>>>>> 
>>>>>>> '
>>>>>>> (...)
>>>>>>> yt.utilities.exceptions.YTFieldNotFound: Could not find field
>>>>>>> '('all',
>>>>>>> 'radial_velocity')' in ds14_a_1.0000.'
>>>>>>> 
>>>>>>> However, as I understand it, the field 'radial_velocity' is
>>>>> native
>>>>>>> to
>>>>>>> yt. How can I surpass this problem?
>>>>>>> 
>>>>>>> Any help will be appreciated.
>>>>>>> 
>>>>>>> Thank you very much for your time,
>>>>>>> Tsouros Alexandros
>>>>>>> _______________________________________________
>>>>>>> yt-users mailing list
>>>>>>> yt-users at lists.spacepope.org
>>>>>>> 
>>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org [2]
>>>>>> _______________________________________________
>>>>>> yt-users mailing list
>>>>>> yt-users at lists.spacepope.org
>>>>>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>>> [2]
>>>>> _______________________________________________
>>>>> yt-users mailing list
>>>>> yt-users at lists.spacepope.org
>>>>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>>> [2]
>>>> _______________________________________________
>>>> yt-users mailing list
>>>> yt-users at lists.spacepope.org
>>>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>>> [2]
>>> _______________________________________________
>>> yt-users mailing list
>>> yt-users at lists.spacepope.org
>>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org [2]
> 
> 
> 
> Links:
> ------
> [1] http://darksky.slac.stanford.edu
> [2] http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
> _______________________________________________
> yt-users mailing list
> yt-users at lists.spacepope.org
> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org

More information about the yt-users mailing list