[yt-users] Dark Sky Simulations, analysis gone wrong.

Sat Jul 1 18:26:24 PDT 2017

I was able to get the following script to work locally using yt 3.4-dev.
Unfortunately I hit a cython type error on yt 3.3.5 when yt tries to read
the particle radial velocities off disk.

https://gist.github.com/ngoldbaum/9506a215c8ae5770e0149dccde8ed012

I think what was happening with your memory error is that you either
weren't loading the dataset with the .midx10 file or you weren't loading it
with a sufficiently small bounding box.

Back to your original question - "radial_velocity" is a field defined for
gas fields in AMR or SPH datasets. For particle data like dark_skies,
you're going to need to use the "particle_radial_velocity" field.

Sorry that took a little while to figure out, I haven't touched the dark
skies data or thingking in years so it took me a while to figure out how to
properly load the data.

On Sat, Jul 1, 2017 at 7:50 PM, Nathan Goldbaum <nathan12343 at gmail.com>
wrote:

> Is this one of the public dark skies datasets? How are you loading it (a
> full script one of us can run locally would be perfect)? What yt version
> are you using?
>
> On Sat, Jul 1, 2017 at 7:15 PM ALEXANDROS TSOUROS <
> ph4602 at edu.physics.uoc.gr> wrote:
>
>> I am terribly sorry for not being clear!
>>
>> Gadget has nothing to do with the problem, I merely mentioned it because
>> when (in a completely unrelated problem) I use datasets derived from it,
>> there's no problem using the 'radial_velocity' field. So Gadget is
>> irrelevant to the query.
>>
>> The memory error comes by merely invoking the datasets via thingking and
>> printing field_lists.
>>
>> Are you aware of anything I can do to get around this?
>>
>> Thank you for your time!
>>
>> On 2017-07-02 02:56, Nathan Goldbaum wrote:
>> > Sorry, I'm confused.
>> >
>> > In your first email you said you were having issues with the dark
>> > skies simulation, but it seems you're trying to load it using the
>> > Gadget frontend somehow? Or is this a different Gadget dataset?
>> >
>> > As far as I'm aware, Dark Skies is available in the SDF output format,
>> > or streaming over HTTP via thingking.
>> >
>> > If it's just a big Gadget dataset, the issue is that it's too big to
>> > load given the memory available on your machine. Right now you need
>> > enough memory to store the Morton indices for all the particles (one
>> > int64 per particle). We are actively working on improving this
>> > situation by making yt use compressed bitmaps to store the Morton
>> > indices, but that work won't be done for several months at least.
>> >
>> > You should be able to stream the dark skies data over HTTP though
>> > without having a ton of memory, there were some clever tricks applied
>> > to get around the memory limit I described above.
>> >
>> > The reason I was asking about the field_list for dark skies is because
>> > I'd like to see what fields are available. The error message you saw
>> > indicates that one or more of the fields necessary to calculate the
>> > radial velocity is not available (or yt is not aware of an alias from
>> > a field name on disk to yt's internal field name).
>> >
>> > On Sat, Jul 1, 2017 at 6:46 PM ALEXANDROS TSOUROS
>> > <ph4602 at edu.physics.uoc.gr> wrote:
>> >
>> >> Thanks for your prompt reply!
>> >>
>> >> So this is another problem. I'm using a 64-bit python installation,
>> >> yet
>> >> the output of 'print ds.file_list' is
>> >>
>> >> "
>> >> (...)
>> >> Traceback (most recent call last):
>> >> File "ds.py", line 5, in <module>
>> >> print ds.field_list
>> >> File
>> >>
>> > "/home/atsouros/gadget2/yt/yt-conda/src/yt-git/yt/data_objec
>> ts/static_output.py",
>> >> line 545, in field_list
>> >> return self.index.field_list
>> >> File
>> >>
>> > "/home/atsouros/gadget2/yt/yt-conda/src/yt-git/yt/data_objec
>> ts/static_output.py",
>> >> line 503, in index
>> >> self, dataset_type=self.dataset_type)
>> >> File
>> >>
>> > "/home/atsouros/gadget2/yt/yt-conda/src/yt-git/yt/geometry/p
>> article_geometry_handler.py",
>> >> line 39, in __init__
>> >> super(ParticleIndex, self).__init__(ds, dataset_type)
>> >> File
>> >>
>> > "/home/atsouros/gadget2/yt/yt-conda/src/yt-git/yt/geometry/g
>> eometry_handler.py",
>> >> line 50, in __init__
>> >> self._setup_geometry()
>> >> File
>> >>
>> > "/home/atsouros/gadget2/yt/yt-conda/src/yt-git/yt/geometry/p
>> article_geometry_handler.py",
>> >> line 50, in _setup_geometry
>> >> self._initialize_particle_handler()
>> >> File
>> >>
>> > "/home/atsouros/gadget2/yt/yt-conda/src/yt-git/yt/geometry/p
>> article_geometry_handler.py",
>> >> line 104, in _initialize_particle_handler
>> >> self._initialize_indices()
>> >> File
>> >>
>> > "/home/atsouros/gadget2/yt/yt-conda/src/yt-git/yt/geometry/p
>> article_geometry_handler.py",
>> >> line 126, in _initialize_indices
>> >> morton = np.empty(self.total_particles, dtype="uint64")
>> >> MemoryError
>> >> "
>> >>
>> >> I would assume that the problem is that it's running out of ram. In
>> >> general, I am able to produce the examples provided by Dark Sky, so
>> >> I
>> >> can in general load data from the dataset. But I cannot use 'print
>> >> ds.field_list'.
>> >>
>> >> TL;DR
>> >>
>> >> The output is 'MemoryError'.
>> >>
>> >> On 2017-07-02 01:34, Nathan Goldbaum wrote:
>> >>> What's the output of "print(ds.field_list)" with the loaded
>> >> dataset?
>> >>>
>> >>> On Sat, Jul 1, 2017 at 4:21 PM ALEXANDROS TSOUROS
>> >>> <ph4602 at edu.physics.uoc.gr> wrote:
>> >>>
>> >>>> Hello all!
>> >>>>
>> >>>> I am quite new to yt, so please bear with my ignorance!
>> >>>>
>> >>>> I need to perform analysis on the open data of the Dark Sky
>> >>>> Simulation
>> >>>> (http://darksky.slac.stanford.edu). In particular I need to
>> >> create 1
>> >>>> dimensional profiles of the radial velocity of dark matter
>> >> particles
>> >>>> around halos.
>> >>>>
>> >>>> Attached, you'll find my attempt at producing such a profile.
>> >>>>
>> >>>> When I do this on a GADGET2 simulation I've run, I can use the
>> >>>> 'radial_velocity' field without a problem. However, this script,
>> >>>> where
>> >>>> the data is that of the Dark Sky simulation, produces the
>> >> following
>> >>>> output :
>> >>>>
>> >>>> '
>> >>>> (...)
>> >>>> yt.utilities.exceptions.YTFieldNotFound: Could not find field
>> >>>> '('all',
>> >>>> 'radial_velocity')' in ds14_a_1.0000.'
>> >>>>
>> >>>> However, as I understand it, the field 'radial_velocity' is
>> >> native
>> >>>> to
>> >>>> yt. How can I surpass this problem?
>> >>>>
>> >>>> Any help will be appreciated.
>> >>>>
>> >>>> Thank you very much for your time,
>> >>>> Tsouros Alexandros
>> >>>> _______________________________________________
>> >>>> yt-users mailing list
>> >>>> yt-users at lists.spacepope.org
>> >>>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>> >>> _______________________________________________
>> >>> yt-users mailing list
>> >>> yt-users at lists.spacepope.org
>> >>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>> >> _______________________________________________
>> >> yt-users mailing list
>> >> yt-users at lists.spacepope.org
>> >> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>> > _______________________________________________
>> > yt-users mailing list
>> > yt-users at lists.spacepope.org
>> > http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>> _______________________________________________
>> yt-users mailing list
>> yt-users at lists.spacepope.org
>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.spacepope.org/pipermail/yt-users-spacepope.org/attachments/20170701/fd27fee9/attachment.html>