[yt-users] Dark Sky Simulations, analysis gone wrong.

[ALEXANDROS TSOUROS]

I am terribly sorry for not being clear!

Gadget has nothing to do with the problem, I merely mentioned it because 
when (in a completely unrelated problem) I use datasets derived from it, 
there's no problem using the 'radial_velocity' field. So Gadget is 
irrelevant to the query.

The memory error comes by merely invoking the datasets via thingking and 
printing field_lists.

Are you aware of anything I can do to get around this?

Thank you for your time!

On 2017-07-02 02:56, Nathan Goldbaum wrote:
> Sorry, I'm confused.
> 
> In your first email you said you were having issues with the dark
> skies simulation, but it seems you're trying to load it using the
> Gadget frontend somehow? Or is this a different Gadget dataset?
> 
> As far as I'm aware, Dark Skies is available in the SDF output format,
> or streaming over HTTP via thingking.
> 
> If it's just a big Gadget dataset, the issue is that it's too big to
> load given the memory available on your machine. Right now you need
> enough memory to store the Morton indices for all the particles (one
> int64 per particle). We are actively working on improving this
> situation by making yt use compressed bitmaps to store the Morton
> indices, but that work won't be done for several months at least.
> 
> You should be able to stream the dark skies data over HTTP though
> without having a ton of memory, there were some clever tricks applied
> to get around the memory limit I described above.
> 
> The reason I was asking about the field_list for dark skies is because
> I'd like to see what fields are available. The error message you saw
> indicates that one or more of the fields necessary to calculate the
> radial velocity is not available (or yt is not aware of an alias from
> a field name on disk to yt's internal field name).
> 
> On Sat, Jul 1, 2017 at 6:46 PM ALEXANDROS TSOUROS
> <ph4602 at edu.physics.uoc.gr> wrote:
> 
>> Thanks for your prompt reply!
>> 
>> So this is another problem. I'm using a 64-bit python installation,
>> yet
>> the output of 'print ds.file_list' is
>> 
>> "
>> (...)
>> Traceback (most recent call last):
>> File "ds.py", line 5, in <module>
>> print ds.field_list
>> File
>> 
> "/home/atsouros/gadget2/yt/yt-conda/src/yt-git/yt/data_objects/static_output.py",
>> line 545, in field_list
>> return self.index.field_list
>> File
>> 
> "/home/atsouros/gadget2/yt/yt-conda/src/yt-git/yt/data_objects/static_output.py",
>> line 503, in index
>> self, dataset_type=self.dataset_type)
>> File
>> 
> "/home/atsouros/gadget2/yt/yt-conda/src/yt-git/yt/geometry/particle_geometry_handler.py",
>> line 39, in __init__
>> super(ParticleIndex, self).__init__(ds, dataset_type)
>> File
>> 
> "/home/atsouros/gadget2/yt/yt-conda/src/yt-git/yt/geometry/geometry_handler.py",
>> line 50, in __init__
>> self._setup_geometry()
>> File
>> 
> "/home/atsouros/gadget2/yt/yt-conda/src/yt-git/yt/geometry/particle_geometry_handler.py",
>> line 50, in _setup_geometry
>> self._initialize_particle_handler()
>> File
>> 
> "/home/atsouros/gadget2/yt/yt-conda/src/yt-git/yt/geometry/particle_geometry_handler.py",
>> line 104, in _initialize_particle_handler
>> self._initialize_indices()
>> File
>> 
> "/home/atsouros/gadget2/yt/yt-conda/src/yt-git/yt/geometry/particle_geometry_handler.py",
>> line 126, in _initialize_indices
>> morton = np.empty(self.total_particles, dtype="uint64")
>> MemoryError
>> "
>> 
>> I would assume that the problem is that it's running out of ram. In
>> general, I am able to produce the examples provided by Dark Sky, so
>> I
>> can in general load data from the dataset. But I cannot use 'print
>> ds.field_list'.
>> 
>> TL;DR
>> 
>> The output is 'MemoryError'.
>> 
>> On 2017-07-02 01:34, Nathan Goldbaum wrote:
>>> What's the output of "print(ds.field_list)" with the loaded
>> dataset?
>>> 
>>> On Sat, Jul 1, 2017 at 4:21 PM ALEXANDROS TSOUROS
>>> <ph4602 at edu.physics.uoc.gr> wrote:
>>> 
>>>> Hello all!
>>>> 
>>>> I am quite new to yt, so please bear with my ignorance!
>>>> 
>>>> I need to perform analysis on the open data of the Dark Sky
>>>> Simulation
>>>> (http://darksky.slac.stanford.edu). In particular I need to
>> create 1
>>>> dimensional profiles of the radial velocity of dark matter
>> particles
>>>> around halos.
>>>> 
>>>> Attached, you'll find my attempt at producing such a profile.
>>>> 
>>>> When I do this on a GADGET2 simulation I've run, I can use the
>>>> 'radial_velocity' field without a problem. However, this script,
>>>> where
>>>> the data is that of the Dark Sky simulation, produces the
>> following
>>>> output :
>>>> 
>>>> '
>>>> (...)
>>>> yt.utilities.exceptions.YTFieldNotFound: Could not find field
>>>> '('all',
>>>> 'radial_velocity')' in ds14_a_1.0000.'
>>>> 
>>>> However, as I understand it, the field 'radial_velocity' is
>> native
>>>> to
>>>> yt. How can I surpass this problem?
>>>> 
>>>> Any help will be appreciated.
>>>> 
>>>> Thank you very much for your time,
>>>> Tsouros Alexandros
>>>> _______________________________________________
>>>> yt-users mailing list
>>>> yt-users at lists.spacepope.org
>>>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>>> _______________________________________________
>>> yt-users mailing list
>>> yt-users at lists.spacepope.org
>>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>> _______________________________________________
>> yt-users mailing list
>> yt-users at lists.spacepope.org
>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
> _______________________________________________
> yt-users mailing list
> yt-users at lists.spacepope.org
> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org