[yt-users] Storing multiple calculations using yt.parallel_objects

[Joshua Wall]

Britton,

      Indeed, I have noticed in the runs that work was being repeated. I'm
not quite sure how to get around it, since the particle filenames are
different than the plot file names. Perhaps I can do?

for plt_file, part_file in plot, part:

     ds = yt.load(plt_file, particle_filename=part_file)

Thnaks again,

Josh

On Wed, May 27, 2015 at 11:14 AM Britton Smith <brittonsmith at gmail.com>
wrote:

> Hi Josh,
>
> Glad to hear you've got it working.  I have just one comment on your
> script.  I don't think you want to use the i variable to load the datasets
> from the plot list.  Since you are incrementing i by one, each process will
> be loading datasets starting from 0 and going up to the total divided by
> the number of processors.  Instead, since you're iterating over the plot
> list, you can just do:
> ds = yt.load(p, particle_filename=p)
>
> Britton
>
> On Sun, May 24, 2015 at 5:59 PM, Joshua Wall <joshua.e.wall at gmail.com>
> wrote:
>
>> Okay, I *think* I got it sorted out. Here's my solution (for future use
>> if deemed correct!)
>>
>> plot = glob.glob("./SinkMomTest/*plt_cnt_000?")
>> plot.sort()
>> part = glob.glob("./SinkMomTest/*part_000?")
>> part.sort()
>> #print plot       # Testing
>>
>> # Make a list of variables you are collecting to plot/store as
>> # dictionaries.
>>
>> my_storage = {}
>>
>> # Now the parallel loop.
>>
>> i = 0
>>
>> for sto,p in yt.parallel_objects(plot, num_procs, storage = my_storage):
>>
>>     ds = yt.load(plot[i], particle_filename=part[i])
>>     dd = ds.all_data()
>>
>>     if (yt.is_root()):
>>         print "Yo I'm root!"
>>
>>     if (~yt.is_root()):
>>         print "Not root!"
>>
>>     momx                             = dd["total_px"]
>>     momy                             = dd["total_py"]
>>     partmomx             = dd["total_part_px"]
>>     partmomy             = dd["total_part_py"]
>>
>>     sto.result_id = p[-4:]
>>     sto.result    = [momx,momy,partmomx,partmomy]
>>
>>     i=i+1
>>
>>
>> print "Done!"
>>
>> if yt.is_root():
>>
>>                 my_storage =
>> collections.OrderedDict(sorted(my_storage.items()))
>>                 print my_storage
>>
>> Thanks again for the help,
>>
>> Josh
>>
>> On Sun, May 24, 2015 at 6:33 PM Joshua Wall <joshua.e.wall at gmail.com>
>> wrote:
>>
>>> Dear Britton,
>>>
>>>
>>> I tried something like that, but clearly I'm not doing it correctly. I'm
>>> noticing two errors, 1) everything is reporting running on the root proc
>>> (with a "Yo I'm root!") and 2) I'm getting four copies of my_storage back.
>>> Here's the code:
>>>
>>> #########################################################################
>>>
>>> from matplotlib import rc
>>> import matplotlib.pyplot as plt
>>> import yt
>>> from yt import derived_field
>>> import numpy as np
>>> import yt.units as u
>>> import glob
>>>
>>> # Use YT in parallel.
>>> yt.enable_parallelism()
>>>
>>> # Number of procs on the machine
>>> num_procs=4
>>>
>>>
>>> @derived_field(name = "total_KE", units = "erg")
>>> def _KE(field, data):
>>>     return
>>> np.sum(data["kinetic_energy"].in_mks()*data["cell_volume"].in_cgs())
>>>
>>> @derived_field(name = "total_PE", units = "erg")
>>> def _PE(field, data):
>>>     return 0.5*np.sum(data["gpot"].in_mks()*data["cell_mass"].in_cgs())
>>>
>>> @derived_field(name = "total_Eint", units = "erg")
>>> def _Eint(field, data):
>>>     return np.sum(data["eint"]*data["cell_mass"]).in_cgs()
>>>
>>> @derived_field(name = "total_energy", units = "erg")
>>> def _Etotal(field, data):
>>>     return data["total_Eint"] + data["total_PE"] + data["total_KE"]
>>>
>>> @derived_field(name = "total_px", units = "g*cm/s")
>>> def _px(field, data):
>>>     return np.sum(data["cell_mass"]*data["velx"]).in_cgs()
>>>
>>> # Use glob to gather up the data files. Note here we use plt
>>> # and particle files because they are much smaller than
>>> # checkpoint files.
>>>
>>> plot = glob.glob("./SinkMomTest/*plt_cnt_000?")
>>> plot.sort()
>>> part = glob.glob("./SinkMomTest/*part_000?")
>>> part.sort()
>>>
>>> # Make a list of variables you are collecting to plot/store as
>>> # dictionaries.
>>> my_storage = {}
>>>
>>> # Now the parallel loop.
>>>
>>> i = 0
>>>
>>> for sto,p in yt.parallel_objects(plot, num_procs, storage = my_storage):
>>>
>>>     ds = yt.load(plot[i], particle_filename=part[i])
>>>     dd = ds.all_data()
>>>
>>>     if (yt.is_root()):
>>>         print "Yo I'm root!"
>>>     else:
>>>         print "Not root!"
>>>
>>>
>>>     energy = dd["total_energy"]
>>>     mom    = dd["total_px"]
>>>
>>>     sto.result_id = p[-4:]
>>>     sto.result = [energy, mom]
>>>
>>>     i=i+1
>>>
>>>
>>> print "Done!"
>>>
>>> print my_storage
>>>
>>> #########################################################################
>>>
>>> Is this what you mean?
>>>
>>> Kindly,
>>>
>>> Josh
>>>
>>>
>>>
>>> On Sun, May 24, 2015 at 12:19 PM Britton Smith <brittonsmith at gmail.com>
>>> wrote:
>>>
>>>> Hi Josh,
>>>>
>>>> The reason that's not working is that there is still only one storage
>>>> item (the sto variable) per iteration of the loop.  You could keep both
>>>> things if you make sto.result a list object and just append each of those
>>>> things to that list.
>>>>
>>>> Britton
>>>>
>>>> On Sun, May 24, 2015 at 11:12 AM, Joshua Wall <joshua.e.wall at gmail.com>
>>>> wrote:
>>>>
>>>>> Dear users,
>>>>>
>>>>>
>>>>>      This there a simple way to store multiple values at each step
>>>>> using parallel_objects? I'm doing:
>>>>>
>>>>> ########################################################
>>>>> yt.enable_parallelism()
>>>>> num_procs = 4
>>>>> plot = glob.glob(./plot*)
>>>>> my_storage = {}
>>>>>
>>>>> for sto, p in yt.parallel_objects(plots, num_procs, my_storage):
>>>>>
>>>>>   sto.result_id = 'energy' + p
>>>>>   sto.resutl     = dd["total_energy"]
>>>>>
>>>>>   sto.result_id = 'Px' + p
>>>>>   sto.result     = dd["total_mom_x"]
>>>>>
>>>>> ######################################################
>>>>>
>>>>> which is just overwriting my energy with the momentum. I also tried to
>>>>> just set to different dictionaries without passing it to
>>>>> yt.parallel_objects as storage, but of course each proc as only a local
>>>>> copy then.
>>>>>
>>>>> Cordially,
>>>>>
>>>>> Josh
>>>>>
>>>>> _______________________________________________
>>>>> yt-users mailing list
>>>>> yt-users at lists.spacepope.org
>>>>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>>>>>
>>>>>
>>>> _______________________________________________
>>>> yt-users mailing list
>>>> yt-users at lists.spacepope.org
>>>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>>>>
>>>
>> _______________________________________________
>> yt-users mailing list
>> yt-users at lists.spacepope.org
>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>>
>>
> _______________________________________________
> yt-users mailing list
> yt-users at lists.spacepope.org
> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.spacepope.org/pipermail/yt-users-spacepope.org/attachments/20150527/6fd48e1e/attachment.htm>
-------------- next part --------------
_______________________________________________
yt-users mailing list
yt-users at lists.spacepope.org
http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org