[yt-users] FLASH Time Series Analysis with Particle Files

Jason Galyardt jason.galyardt at gmail.com
Thu Dec 1 07:58:25 PST 2016


Hi Nathan,

Thanks for the explanation. I thought I had tried calling yt.load() with
just the plot files at some point, which is why I was trying to give it the
particle_filename keyword. I tried it on my dataset, and it works. User
error. Apologies.

Jason


On Thu, Dec 1, 2016 at 10:47 AM, <yt-users-request at lists.spacepope.org>
wrote:

> Send yt-users mailing list submissions to
>         yt-users at lists.spacepope.org
>
> To subscribe or unsubscribe via the World Wide Web, visit
>         http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
> or, via email, send a message with subject or body 'help' to
>         yt-users-request at lists.spacepope.org
>
> You can reach the person managing the list at
>         yt-users-owner at lists.spacepope.org
>
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of yt-users digest..."
>
> Today's Topics:
>
>    1. Re: FLASH Time Series Analysis with Particle Files (Alex Hill)
>    2. Re: FLASH Time Series Analysis with Particle Files
>       (Jason Galyardt)
>    3. Re: FLASH Time Series Analysis with Particle Files
>       (Nathan Goldbaum)
>
>
> ---------- Forwarded message ----------
> From: Alex Hill <ashill at haverford.edu>
> To: Discussion of the yt analysis package <yt-users at lists.spacepope.org>
> Cc:
> Date: Wed, 30 Nov 2016 13:59:19 -0500
> Subject: Re: [yt-users] FLASH Time Series Analysis with Particle Files
> Hi Jason,
>
> Yes, I use Flash files with particles routinely now. As John alludes to,
> there are issues when the plot files and particle files don’t match in both
> file name (including the plot/particle file number) and timestamp. I now
> avoid those issues by ensuring that the names are the same; I don’t recall
> if the issue with different names/numbers was solved but could look into it
> again if it would help.
>
> -Alex
>
> > On Nov 30, 2016, at 11:32, John Zuhone <jzuhone at gmail.com> wrote:
> >
> > Yes, my understanding is that this is working, provided that your files
> have the same basename and the times of the files are the same.
> >
> >> On Nov 30, 2016, at 10:32 AM, Nathan Goldbaum <nathan12343 at gmail.com>
> wrote:
> >>
> >> Hi Jason,
> >>
> >> As far as I'm aware this should be working now. Are you having trouble?
> >>
> >> -Nathan
> >>
> >> On Wed, Nov 30, 2016 at 9:16 AM, Jason Galyardt <
> jason.galyardt at gmail.com> wrote:
> >> Dear yt-users,
> >>
> >> In May 2015, there was some traffic on this list concerning errors in
> time series analysis on FLASH simulations that include both plot files and
> particle files. Does anyone have an update on the status of this issue?
> >>
> >> Thanks,
> >> Jason
> >>
> >> ------
> >> Jason Galyardt
> >> University of Georgia
> >>
> >>
> >> _______________________________________________
> >> yt-users mailing list
> >> yt-users at lists.spacepope.org
> >> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
> >>
> >>
> >> _______________________________________________
> >> yt-users mailing list
> >> yt-users at lists.spacepope.org
> >> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
> >
> > _______________________________________________
> > yt-users mailing list
> > yt-users at lists.spacepope.org
> > http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>
>
>
>
> ---------- Forwarded message ----------
> From: Jason Galyardt <jason.galyardt at gmail.com>
> To: Discussion of the yt analysis package <yt-users at lists.spacepope.org>
> Cc:
> Date: Wed, 30 Nov 2016 14:47:06 -0500
> Subject: Re: [yt-users] FLASH Time Series Analysis with Particle Files
> Hi John & Nathan,
>
> Yes, I'm having trouble using the time series analysis machinery with my
> FLASH simulations. I've checked that the plot files and particle files all
> have the same current_time variable, so there's no issue there. I'm using a
> development version of yt with a changeset ID of 5cf30893afdf. This
> changeset includes Nathan's changes supporting cut_region functionality for
> FLASH plot files.  Here's what I see when I try to do a time series
> analysis:
>
> ###########################
>
> In [9]: ts = yt.load("mysim_hdf5_plt_cnt_*",
>     particle_filename="mysim_hdf5_part_*")
>
> In [10]: for ds in ts:
>     print ds.current_time
>    ....:
> ------------------------------------------------------------
> ---------------
> IOError                                   Traceback (most recent call
> last)
> <ipython-input-10-fccd01bb5065> in <module>()
> ----> 1 for ds in ts:
>       2     print ds.current_time
>       3
>
> /home/emrys/yt/yt/data_objects/time_series.pyc in __iter__(self)
>     179         for o in self._pre_outputs:
>     180             if isinstance(o, string_types):
> --> 181                 ds = self._load(o, **self.kwargs)
>     182                 self._setup_function(ds)
>     183                 yield ds
>
> /home/emrys/yt/yt/data_objects/time_series.pyc in _load(self, output_fn,
> **kwargs)
>     396         elif self._mixed_dataset_types:
>     397             return load(output_fn, **kwargs)
> --> 398         ds = load(output_fn, **kwargs)
>     399         self._dataset_cls = ds.__class__
>     400         return ds
>
> /home/emrys/yt/yt/convenience.pyc in load(*args, **kwargs)
>      84     candidates = find_lowest_subclasses(candidates)
>      85     if len(candidates) == 1:
> ---> 86         return candidates[0](*args, **kwargs)
>      87     if len(candidates) == 0:
>      88         if ytcfg.get("yt", "enzo_db") != '' \
>
> /home/emrys/yt/yt/frontends/flash/data_structures.pyc in __init__(self,
> filename, dataset_type, storage_filename, particle_filename,
> units_override, unit_system)
>     209                 self._particle_handle =
> HDF5FileHandler(self.particle_filename)
>     210             except:
> --> 211                 raise IOError(self.particle_filename)
>     212         # Check if the particle file has the same time
>     213         if self._particle_handle != self._handle:
>
> IOError: mysim_hdf5_part_*
>
> ###########################
>
> It looks to me like HDF5FileHandler() just does not like the wildcard.
> I've also tried, unsuccessfully, to feed a list of particle files to
> yt.load(). I thought perhaps that I should be using the DatasetSeries
> framework directly, but I'm not clear on how to pass in the list of
> particle files, as DatasetSeries does not understand the particle_filename
> keyword argument.
>
> Any ideas?
>
> Thanks,
> Jason
>
>
> On Wed, Nov 30, 2016 at 1:57 PM, <yt-users-request at lists.spacepope.org>
> wrote:
>
>> Send yt-users mailing list submissions to
>>         yt-users at lists.spacepope.org
>>
>> To subscribe or unsubscribe via the World Wide Web, visit
>>         http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>> or, via email, send a message with subject or body 'help' to
>>         yt-users-request at lists.spacepope.org
>>
>> You can reach the person managing the list at
>>         yt-users-owner at lists.spacepope.org
>>
>> When replying, please edit your Subject line so it is more specific
>> than "Re: Contents of yt-users digest..."
>>
>> Today's Topics:
>>
>>    1. FLASH Time Series Analysis with Particle Files (Jason Galyardt)
>>    2. Re: FLASH Time Series Analysis with Particle Files
>>       (Nathan Goldbaum)
>>    3. Re: FLASH Time Series Analysis with Particle Files (John Zuhone)
>>
>>
>> ---------- Forwarded message ----------
>> From: Jason Galyardt <jason.galyardt at gmail.com>
>> To: Discussion of the yt analysis package <yt-users at lists.spacepope.org>
>> Cc:
>> Date: Wed, 30 Nov 2016 10:16:44 -0500
>> Subject: [yt-users] FLASH Time Series Analysis with Particle Files
>> Dear yt-users,
>>
>> In May 2015, there was some traffic on this list concerning errors in
>> time series analysis on FLASH simulations that include both plot files and
>> particle files. Does anyone have an update on the status of this issue?
>>
>> Thanks,
>> Jason
>>
>> ------
>> Jason Galyardt
>> University of Georgia
>>
>>
>>
>> ---------- Forwarded message ----------
>> From: Nathan Goldbaum <nathan12343 at gmail.com>
>> To: Discussion of the yt analysis package <yt-users at lists.spacepope.org>
>> Cc:
>> Date: Wed, 30 Nov 2016 09:32:37 -0600
>> Subject: Re: [yt-users] FLASH Time Series Analysis with Particle Files
>> Hi Jason,
>>
>> As far as I'm aware this should be working now. Are you having trouble?
>>
>> -Nathan
>>
>> On Wed, Nov 30, 2016 at 9:16 AM, Jason Galyardt <jason.galyardt at gmail.com
>> > wrote:
>>
>>> Dear yt-users,
>>>
>>> In May 2015, there was some traffic on this list concerning errors in
>>> time series analysis on FLASH simulations that include both plot files and
>>> particle files. Does anyone have an update on the status of this issue?
>>>
>>> Thanks,
>>> Jason
>>>
>>> ------
>>> Jason Galyardt
>>> University of Georgia
>>>
>>>
>>> _______________________________________________
>>> yt-users mailing list
>>> yt-users at lists.spacepope.org
>>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>>>
>>>
>>
>>
>> ---------- Forwarded message ----------
>> From: John Zuhone <jzuhone at gmail.com>
>> To: Discussion of the yt analysis package <yt-users at lists.spacepope.org>
>> Cc:
>> Date: Wed, 30 Nov 2016 11:32:09 -0500
>> Subject: Re: [yt-users] FLASH Time Series Analysis with Particle Files
>> Yes, my understanding is that this is working, provided that your files
>> have the same basename and the times of the files are the same.
>>
>> On Nov 30, 2016, at 10:32 AM, Nathan Goldbaum <nathan12343 at gmail.com>
>> wrote:
>>
>> Hi Jason,
>>
>> As far as I'm aware this should be working now. Are you having trouble?
>>
>> -Nathan
>>
>> On Wed, Nov 30, 2016 at 9:16 AM, Jason Galyardt <jason.galyardt at gmail.com
>> > wrote:
>>
>>> Dear yt-users,
>>>
>>> In May 2015, there was some traffic on this list concerning errors in
>>> time series analysis on FLASH simulations that include both plot files and
>>> particle files. Does anyone have an update on the status of this issue?
>>>
>>> Thanks,
>>> Jason
>>>
>>> ------
>>> Jason Galyardt
>>> University of Georgia
>>>
>>>
>>> _______________________________________________
>>> yt-users mailing list
>>> yt-users at lists.spacepope.org
>>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>>>
>>>
>> _______________________________________________
>> yt-users mailing list
>> yt-users at lists.spacepope.org
>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>>
>>
>>
>> _______________________________________________
>> yt-users mailing list
>> yt-users at lists.spacepope.org
>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>>
>>
>
>
> ---------- Forwarded message ----------
> From: Nathan Goldbaum <nathan12343 at gmail.com>
> To: Discussion of the yt analysis package <yt-users at lists.spacepope.org>
> Cc:
> Date: Thu, 1 Dec 2016 09:46:54 -0600
> Subject: Re: [yt-users] FLASH Time Series Analysis with Particle Files
> Hi Jason,
>
> Thanks for those details.
>
> The FLASH frontend has a capability to automatically detect the presence
> of an accompanying particle file for a given plotfile.  I have a test
> dataset (that I will be uploading to yt-project.org/data shortly for
> future questions about flash datasets that contain both particle and mesh
> fields) that I've written the following test script for:
>
>     import yt
>
>     ts = yt.load('fiducial_1to1_b0_hdf5_plt_cnt_*')
>
>     for ds in ts:
>         print(ds.particle_type_counts)
>
> This time series contains five outputs, with a plotfile and a particlefile
> for each output:
>
>     $ ls
>     fiducial_1to1_b0_hdf5_part_0000    fiducial_1to1_b0_hdf5_plt_cnt_0001
>     fiducial_1to1_b0_hdf5_part_0001    fiducial_1to1_b0_hdf5_plt_cnt_0002
>     fiducial_1to1_b0_hdf5_part_0002    fiducial_1to1_b0_hdf5_plt_cnt_0003
>     fiducial_1to1_b0_hdf5_part_0003    fiducial_1to1_b0_hdf5_plt_cnt_0004
>     fiducial_1to1_b0_hdf5_part_0004    test.py
>     fiducial_1to1_b0_hdf5_plt_cnt_0000
>
> When I run the script, it outputs:
>
>     {'io': 9999998}
>     {'io': 9999998}
>     {'io': 9999998}
>     {'io': 9999998}
>     {'io': 9999998}
>
> As you can see, my script only explicitly loads the plotfile but yt still
> recognizes that there are about 10 million particles in each ouput, since
> the particle file for each plotfile is automatically associated with the
> plotfile. You can see how this functionality is setup in the FLASH frontend
> here:
>
> https://bitbucket.org/yt_analysis/yt/src/yt/yt/frontends/flash/data_
> structures.py?#data_structures.py-202
>
> This requires that the particle file associated with a plotfile has the
> same output number. We used to strictly enforce that the particle file has
> the same simulation time as the plotifle, but this ended up being too
> strict for real-world use cases so now we only print a warning (see around
> like 217 in the file I linked to above).
>
> Is this sufficient for your use cases or do you need to explicitly specify
> the name of the particle file every time you load the dataset? If so, one
> hacky way around this would be to manually associate the particle file with
> each plotfile using the `setup_function` keyword argument for the
> DatasetSeries object. I could also look into making it so that if you pass
> a glob pattern as the `particle_filename` keyword argument to the time
> series object, it does the right thing, but I want to make sure that the
> existing functionality will not work for you before trying that.
>
> Hope that clears things up,
>
> Nathan
>
> On Wed, Nov 30, 2016 at 1:47 PM, Jason Galyardt <jason.galyardt at gmail.com>
> wrote:
>
>> Hi John & Nathan,
>>
>> Yes, I'm having trouble using the time series analysis machinery with my
>> FLASH simulations. I've checked that the plot files and particle files all
>> have the same current_time variable, so there's no issue there. I'm using a
>> development version of yt with a changeset ID of 5cf30893afdf. This
>> changeset includes Nathan's changes supporting cut_region functionality for
>> FLASH plot files.  Here's what I see when I try to do a time series
>> analysis:
>>
>> ###########################
>>
>> In [9]: ts = yt.load("mysim_hdf5_plt_cnt_*",
>>     particle_filename="mysim_hdf5_part_*")
>>
>> In [10]: for ds in ts:
>>     print ds.current_time
>>    ....:
>> ------------------------------------------------------------
>> ---------------
>> IOError                                   Traceback (most recent call
>> last)
>> <ipython-input-10-fccd01bb5065> in <module>()
>> ----> 1 for ds in ts:
>>       2     print ds.current_time
>>       3
>>
>> /home/emrys/yt/yt/data_objects/time_series.pyc in __iter__(self)
>>     179         for o in self._pre_outputs:
>>     180             if isinstance(o, string_types):
>> --> 181                 ds = self._load(o, **self.kwargs)
>>     182                 self._setup_function(ds)
>>     183                 yield ds
>>
>> /home/emrys/yt/yt/data_objects/time_series.pyc in _load(self, output_fn,
>> **kwargs)
>>     396         elif self._mixed_dataset_types:
>>     397             return load(output_fn, **kwargs)
>> --> 398         ds = load(output_fn, **kwargs)
>>     399         self._dataset_cls = ds.__class__
>>     400         return ds
>>
>> /home/emrys/yt/yt/convenience.pyc in load(*args, **kwargs)
>>      84     candidates = find_lowest_subclasses(candidates)
>>      85     if len(candidates) == 1:
>> ---> 86         return candidates[0](*args, **kwargs)
>>      87     if len(candidates) == 0:
>>      88         if ytcfg.get("yt", "enzo_db") != '' \
>>
>> /home/emrys/yt/yt/frontends/flash/data_structures.pyc in __init__(self,
>> filename, dataset_type, storage_filename, particle_filename,
>> units_override, unit_system)
>>     209                 self._particle_handle =
>> HDF5FileHandler(self.particle_filename)
>>     210             except:
>> --> 211                 raise IOError(self.particle_filename)
>>     212         # Check if the particle file has the same time
>>     213         if self._particle_handle != self._handle:
>>
>> IOError: mysim_hdf5_part_*
>>
>> ###########################
>>
>> It looks to me like HDF5FileHandler() just does not like the wildcard.
>> I've also tried, unsuccessfully, to feed a list of particle files to
>> yt.load(). I thought perhaps that I should be using the DatasetSeries
>> framework directly, but I'm not clear on how to pass in the list of
>> particle files, as DatasetSeries does not understand the particle_filename
>> keyword argument.
>>
>> Any ideas?
>>
>> Thanks,
>> Jason
>>
>>
>> On Wed, Nov 30, 2016 at 1:57 PM, <yt-users-request at lists.spacepope.org>
>> wrote:
>>
>>> Send yt-users mailing list submissions to
>>>         yt-users at lists.spacepope.org
>>>
>>> To subscribe or unsubscribe via the World Wide Web, visit
>>>         http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>>> or, via email, send a message with subject or body 'help' to
>>>         yt-users-request at lists.spacepope.org
>>>
>>> You can reach the person managing the list at
>>>         yt-users-owner at lists.spacepope.org
>>>
>>> When replying, please edit your Subject line so it is more specific
>>> than "Re: Contents of yt-users digest..."
>>>
>>> Today's Topics:
>>>
>>>    1. FLASH Time Series Analysis with Particle Files (Jason Galyardt)
>>>    2. Re: FLASH Time Series Analysis with Particle Files
>>>       (Nathan Goldbaum)
>>>    3. Re: FLASH Time Series Analysis with Particle Files (John Zuhone)
>>>
>>>
>>> ---------- Forwarded message ----------
>>> From: Jason Galyardt <jason.galyardt at gmail.com>
>>> To: Discussion of the yt analysis package <yt-users at lists.spacepope.org>
>>> Cc:
>>> Date: Wed, 30 Nov 2016 10:16:44 -0500
>>> Subject: [yt-users] FLASH Time Series Analysis with Particle Files
>>> Dear yt-users,
>>>
>>> In May 2015, there was some traffic on this list concerning errors in
>>> time series analysis on FLASH simulations that include both plot files and
>>> particle files. Does anyone have an update on the status of this issue?
>>>
>>> Thanks,
>>> Jason
>>>
>>> ------
>>> Jason Galyardt
>>> University of Georgia
>>>
>>>
>>>
>>> ---------- Forwarded message ----------
>>> From: Nathan Goldbaum <nathan12343 at gmail.com>
>>> To: Discussion of the yt analysis package <yt-users at lists.spacepope.org>
>>> Cc:
>>> Date: Wed, 30 Nov 2016 09:32:37 -0600
>>> Subject: Re: [yt-users] FLASH Time Series Analysis with Particle Files
>>> Hi Jason,
>>>
>>> As far as I'm aware this should be working now. Are you having trouble?
>>>
>>> -Nathan
>>>
>>> On Wed, Nov 30, 2016 at 9:16 AM, Jason Galyardt <
>>> jason.galyardt at gmail.com> wrote:
>>>
>>>> Dear yt-users,
>>>>
>>>> In May 2015, there was some traffic on this list concerning errors in
>>>> time series analysis on FLASH simulations that include both plot files and
>>>> particle files. Does anyone have an update on the status of this issue?
>>>>
>>>> Thanks,
>>>> Jason
>>>>
>>>> ------
>>>> Jason Galyardt
>>>> University of Georgia
>>>>
>>>>
>>>> _______________________________________________
>>>> yt-users mailing list
>>>> yt-users at lists.spacepope.org
>>>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>>>>
>>>>
>>>
>>>
>>> ---------- Forwarded message ----------
>>> From: John Zuhone <jzuhone at gmail.com>
>>> To: Discussion of the yt analysis package <yt-users at lists.spacepope.org>
>>> Cc:
>>> Date: Wed, 30 Nov 2016 11:32:09 -0500
>>> Subject: Re: [yt-users] FLASH Time Series Analysis with Particle Files
>>> Yes, my understanding is that this is working, provided that your files
>>> have the same basename and the times of the files are the same.
>>>
>>> On Nov 30, 2016, at 10:32 AM, Nathan Goldbaum <nathan12343 at gmail.com>
>>> wrote:
>>>
>>> Hi Jason,
>>>
>>> As far as I'm aware this should be working now. Are you having trouble?
>>>
>>> -Nathan
>>>
>>> On Wed, Nov 30, 2016 at 9:16 AM, Jason Galyardt <
>>> jason.galyardt at gmail.com> wrote:
>>>
>>>> Dear yt-users,
>>>>
>>>> In May 2015, there was some traffic on this list concerning errors in
>>>> time series analysis on FLASH simulations that include both plot files and
>>>> particle files. Does anyone have an update on the status of this issue?
>>>>
>>>> Thanks,
>>>> Jason
>>>>
>>>> ------
>>>> Jason Galyardt
>>>> University of Georgia
>>>>
>>>>
>>>> _______________________________________________
>>>> yt-users mailing list
>>>> yt-users at lists.spacepope.org
>>>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>>>>
>>>>
>>> _______________________________________________
>>> yt-users mailing list
>>> yt-users at lists.spacepope.org
>>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>>>
>>>
>>>
>>> _______________________________________________
>>> yt-users mailing list
>>> yt-users at lists.spacepope.org
>>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>>>
>>>
>>
>> _______________________________________________
>> yt-users mailing list
>> yt-users at lists.spacepope.org
>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>>
>>
>
> _______________________________________________
> yt-users mailing list
> yt-users at lists.spacepope.org
> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.spacepope.org/pipermail/yt-users-spacepope.org/attachments/20161201/af9e9604/attachment.html>


More information about the yt-users mailing list