[yt-users] Dark Sky Simulations, analysis gone wrong.

Sat Jul 1 17:50:13 PDT 2017

Is this one of the public dark skies datasets? How are you loading it (a
full script one of us can run locally would be perfect)? What yt version
are you using?

On Sat, Jul 1, 2017 at 7:15 PM ALEXANDROS TSOUROS <ph4602 at edu.physics.uoc.gr>
wrote:

> I am terribly sorry for not being clear!
>
> Gadget has nothing to do with the problem, I merely mentioned it because
> when (in a completely unrelated problem) I use datasets derived from it,
> there's no problem using the 'radial_velocity' field. So Gadget is
> irrelevant to the query.
>
> The memory error comes by merely invoking the datasets via thingking and
> printing field_lists.
>
> Are you aware of anything I can do to get around this?
>
> Thank you for your time!
>
> On 2017-07-02 02:56, Nathan Goldbaum wrote:
> > Sorry, I'm confused.
> >
> > In your first email you said you were having issues with the dark
> > skies simulation, but it seems you're trying to load it using the
> > Gadget frontend somehow? Or is this a different Gadget dataset?
> >
> > As far as I'm aware, Dark Skies is available in the SDF output format,
> > or streaming over HTTP via thingking.
> >
> > If it's just a big Gadget dataset, the issue is that it's too big to
> > load given the memory available on your machine. Right now you need
> > enough memory to store the Morton indices for all the particles (one
> > int64 per particle). We are actively working on improving this
> > situation by making yt use compressed bitmaps to store the Morton
> > indices, but that work won't be done for several months at least.
> >
> > You should be able to stream the dark skies data over HTTP though
> > without having a ton of memory, there were some clever tricks applied
> > to get around the memory limit I described above.
> >
> > The reason I was asking about the field_list for dark skies is because
> > I'd like to see what fields are available. The error message you saw
> > indicates that one or more of the fields necessary to calculate the
> > radial velocity is not available (or yt is not aware of an alias from
> > a field name on disk to yt's internal field name).
> >
> > On Sat, Jul 1, 2017 at 6:46 PM ALEXANDROS TSOUROS
> > <ph4602 at edu.physics.uoc.gr> wrote:
> >
> >> Thanks for your prompt reply!
> >>
> >> So this is another problem. I'm using a 64-bit python installation,
> >> yet
> >> the output of 'print ds.file_list' is
> >>
> >> "
> >> (...)
> >> Traceback (most recent call last):
> >> File "ds.py", line 5, in <module>
> >> print ds.field_list
> >> File
> >>
> > "/home/atsouros/gadget2/yt/yt-conda/src/yt-git/yt/data_
> objects/static_output.py",
> >> line 545, in field_list
> >> return self.index.field_list
> >> File
> >>
> > "/home/atsouros/gadget2/yt/yt-conda/src/yt-git/yt/data_
> objects/static_output.py",
> >> line 503, in index
> >> self, dataset_type=self.dataset_type)
> >> File
> >>
> > "/home/atsouros/gadget2/yt/yt-conda/src/yt-git/yt/geometry/
> particle_geometry_handler.py",
> >> line 39, in __init__
> >> super(ParticleIndex, self).__init__(ds, dataset_type)
> >> File
> >>
> > "/home/atsouros/gadget2/yt/yt-conda/src/yt-git/yt/geometry/
> geometry_handler.py",
> >> line 50, in __init__
> >> self._setup_geometry()
> >> File
> >>
> > "/home/atsouros/gadget2/yt/yt-conda/src/yt-git/yt/geometry/
> particle_geometry_handler.py",
> >> line 50, in _setup_geometry
> >> self._initialize_particle_handler()
> >> File
> >>
> > "/home/atsouros/gadget2/yt/yt-conda/src/yt-git/yt/geometry/
> particle_geometry_handler.py",
> >> line 104, in _initialize_particle_handler
> >> self._initialize_indices()
> >> File
> >>
> > "/home/atsouros/gadget2/yt/yt-conda/src/yt-git/yt/geometry/
> particle_geometry_handler.py",
> >> line 126, in _initialize_indices
> >> morton = np.empty(self.total_particles, dtype="uint64")
> >> MemoryError
> >> "
> >>
> >> I would assume that the problem is that it's running out of ram. In
> >> general, I am able to produce the examples provided by Dark Sky, so
> >> I
> >> can in general load data from the dataset. But I cannot use 'print
> >> ds.field_list'.
> >>
> >> TL;DR
> >>
> >> The output is 'MemoryError'.
> >>
> >> On 2017-07-02 01:34, Nathan Goldbaum wrote:
> >>> What's the output of "print(ds.field_list)" with the loaded
> >> dataset?
> >>>
> >>> On Sat, Jul 1, 2017 at 4:21 PM ALEXANDROS TSOUROS
> >>> <ph4602 at edu.physics.uoc.gr> wrote:
> >>>
> >>>> Hello all!
> >>>>
> >>>> I am quite new to yt, so please bear with my ignorance!
> >>>>
> >>>> I need to perform analysis on the open data of the Dark Sky
> >>>> Simulation
> >>>> (http://darksky.slac.stanford.edu). In particular I need to
> >> create 1
> >>>> dimensional profiles of the radial velocity of dark matter
> >> particles
> >>>> around halos.
> >>>>
> >>>> Attached, you'll find my attempt at producing such a profile.
> >>>>
> >>>> When I do this on a GADGET2 simulation I've run, I can use the
> >>>> 'radial_velocity' field without a problem. However, this script,
> >>>> where
> >>>> the data is that of the Dark Sky simulation, produces the
> >> following
> >>>> output :
> >>>>
> >>>> '
> >>>> (...)
> >>>> yt.utilities.exceptions.YTFieldNotFound: Could not find field
> >>>> '('all',
> >>>> 'radial_velocity')' in ds14_a_1.0000.'
> >>>>
> >>>> However, as I understand it, the field 'radial_velocity' is
> >> native
> >>>> to
> >>>> yt. How can I surpass this problem?
> >>>>
> >>>> Any help will be appreciated.
> >>>>
> >>>> Thank you very much for your time,
> >>>> Tsouros Alexandros
> >>>> _______________________________________________
> >>>> yt-users mailing list
> >>>> yt-users at lists.spacepope.org
> >>>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
> >>> _______________________________________________
> >>> yt-users mailing list
> >>> yt-users at lists.spacepope.org
> >>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
> >> _______________________________________________
> >> yt-users mailing list
> >> yt-users at lists.spacepope.org
> >> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
> > _______________________________________________
> > yt-users mailing list
> > yt-users at lists.spacepope.org
> > http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
> _______________________________________________
> yt-users mailing list
> yt-users at lists.spacepope.org
> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.spacepope.org/pipermail/yt-users-spacepope.org/attachments/20170701/526a2fac/attachment-0001.htm>