[yt-users] Storing multiple calculations using yt.parallel_objects
Britton Smith
brittonsmith at gmail.com
Wed May 27 08:56:14 PDT 2015
Hi Josh,
If you want to loop over two or more things, you can something like this:
list_a = ...
list_b = ...
for sto, (a, b) in yt.parallel_objects(zip(list_a, list_b), num_procs,
storage=my_storage):
print a, b
Britton
On Wed, May 27, 2015 at 10:26 AM, Joshua Wall <joshua.e.wall at gmail.com>
wrote:
> Britton,
>
> Indeed, I have noticed in the runs that work was being repeated. I'm
> not quite sure how to get around it, since the particle filenames are
> different than the plot file names. Perhaps I can do?
>
> for plt_file, part_file in plot, part:
>
> ds = yt.load(plt_file, particle_filename=part_file)
>
> Thnaks again,
>
> Josh
>
> On Wed, May 27, 2015 at 11:14 AM Britton Smith <brittonsmith at gmail.com>
> wrote:
>
>> Hi Josh,
>>
>> Glad to hear you've got it working. I have just one comment on your
>> script. I don't think you want to use the i variable to load the datasets
>> from the plot list. Since you are incrementing i by one, each process will
>> be loading datasets starting from 0 and going up to the total divided by
>> the number of processors. Instead, since you're iterating over the plot
>> list, you can just do:
>> ds = yt.load(p, particle_filename=p)
>>
>> Britton
>>
>> On Sun, May 24, 2015 at 5:59 PM, Joshua Wall <joshua.e.wall at gmail.com>
>> wrote:
>>
>>> Okay, I *think* I got it sorted out. Here's my solution (for future use
>>> if deemed correct!)
>>>
>>> plot = glob.glob("./SinkMomTest/*plt_cnt_000?")
>>> plot.sort()
>>> part = glob.glob("./SinkMomTest/*part_000?")
>>> part.sort()
>>> #print plot # Testing
>>>
>>> # Make a list of variables you are collecting to plot/store as
>>> # dictionaries.
>>>
>>> my_storage = {}
>>>
>>> # Now the parallel loop.
>>>
>>> i = 0
>>>
>>> for sto,p in yt.parallel_objects(plot, num_procs, storage = my_storage):
>>>
>>> ds = yt.load(plot[i], particle_filename=part[i])
>>> dd = ds.all_data()
>>>
>>> if (yt.is_root()):
>>> print "Yo I'm root!"
>>>
>>> if (~yt.is_root()):
>>> print "Not root!"
>>>
>>> momx = dd["total_px"]
>>> momy = dd["total_py"]
>>> partmomx = dd["total_part_px"]
>>> partmomy = dd["total_part_py"]
>>>
>>> sto.result_id = p[-4:]
>>> sto.result = [momx,momy,partmomx,partmomy]
>>>
>>> i=i+1
>>>
>>>
>>> print "Done!"
>>>
>>> if yt.is_root():
>>>
>>> my_storage =
>>> collections.OrderedDict(sorted(my_storage.items()))
>>> print my_storage
>>>
>>> Thanks again for the help,
>>>
>>> Josh
>>>
>>> On Sun, May 24, 2015 at 6:33 PM Joshua Wall <joshua.e.wall at gmail.com>
>>> wrote:
>>>
>>>> Dear Britton,
>>>>
>>>>
>>>> I tried something like that, but clearly I'm not doing it correctly.
>>>> I'm noticing two errors, 1) everything is reporting running on the root
>>>> proc (with a "Yo I'm root!") and 2) I'm getting four copies of my_storage
>>>> back. Here's the code:
>>>>
>>>>
>>>> #########################################################################
>>>>
>>>> from matplotlib import rc
>>>> import matplotlib.pyplot as plt
>>>> import yt
>>>> from yt import derived_field
>>>> import numpy as np
>>>> import yt.units as u
>>>> import glob
>>>>
>>>> # Use YT in parallel.
>>>> yt.enable_parallelism()
>>>>
>>>> # Number of procs on the machine
>>>> num_procs=4
>>>>
>>>>
>>>> @derived_field(name = "total_KE", units = "erg")
>>>> def _KE(field, data):
>>>> return
>>>> np.sum(data["kinetic_energy"].in_mks()*data["cell_volume"].in_cgs())
>>>>
>>>> @derived_field(name = "total_PE", units = "erg")
>>>> def _PE(field, data):
>>>> return 0.5*np.sum(data["gpot"].in_mks()*data["cell_mass"].in_cgs())
>>>>
>>>> @derived_field(name = "total_Eint", units = "erg")
>>>> def _Eint(field, data):
>>>> return np.sum(data["eint"]*data["cell_mass"]).in_cgs()
>>>>
>>>> @derived_field(name = "total_energy", units = "erg")
>>>> def _Etotal(field, data):
>>>> return data["total_Eint"] + data["total_PE"] + data["total_KE"]
>>>>
>>>> @derived_field(name = "total_px", units = "g*cm/s")
>>>> def _px(field, data):
>>>> return np.sum(data["cell_mass"]*data["velx"]).in_cgs()
>>>>
>>>> # Use glob to gather up the data files. Note here we use plt
>>>> # and particle files because they are much smaller than
>>>> # checkpoint files.
>>>>
>>>> plot = glob.glob("./SinkMomTest/*plt_cnt_000?")
>>>> plot.sort()
>>>> part = glob.glob("./SinkMomTest/*part_000?")
>>>> part.sort()
>>>>
>>>> # Make a list of variables you are collecting to plot/store as
>>>> # dictionaries.
>>>> my_storage = {}
>>>>
>>>> # Now the parallel loop.
>>>>
>>>> i = 0
>>>>
>>>> for sto,p in yt.parallel_objects(plot, num_procs, storage = my_storage):
>>>>
>>>> ds = yt.load(plot[i], particle_filename=part[i])
>>>> dd = ds.all_data()
>>>>
>>>> if (yt.is_root()):
>>>> print "Yo I'm root!"
>>>> else:
>>>> print "Not root!"
>>>>
>>>>
>>>> energy = dd["total_energy"]
>>>> mom = dd["total_px"]
>>>>
>>>> sto.result_id = p[-4:]
>>>> sto.result = [energy, mom]
>>>>
>>>> i=i+1
>>>>
>>>>
>>>> print "Done!"
>>>>
>>>> print my_storage
>>>>
>>>>
>>>> #########################################################################
>>>>
>>>> Is this what you mean?
>>>>
>>>> Kindly,
>>>>
>>>> Josh
>>>>
>>>>
>>>>
>>>> On Sun, May 24, 2015 at 12:19 PM Britton Smith <brittonsmith at gmail.com>
>>>> wrote:
>>>>
>>>>> Hi Josh,
>>>>>
>>>>> The reason that's not working is that there is still only one storage
>>>>> item (the sto variable) per iteration of the loop. You could keep both
>>>>> things if you make sto.result a list object and just append each of those
>>>>> things to that list.
>>>>>
>>>>> Britton
>>>>>
>>>>> On Sun, May 24, 2015 at 11:12 AM, Joshua Wall <joshua.e.wall at gmail.com
>>>>> > wrote:
>>>>>
>>>>>> Dear users,
>>>>>>
>>>>>>
>>>>>> This there a simple way to store multiple values at each step
>>>>>> using parallel_objects? I'm doing:
>>>>>>
>>>>>> ########################################################
>>>>>> yt.enable_parallelism()
>>>>>> num_procs = 4
>>>>>> plot = glob.glob(./plot*)
>>>>>> my_storage = {}
>>>>>>
>>>>>> for sto, p in yt.parallel_objects(plots, num_procs, my_storage):
>>>>>>
>>>>>> sto.result_id = 'energy' + p
>>>>>> sto.resutl = dd["total_energy"]
>>>>>>
>>>>>> sto.result_id = 'Px' + p
>>>>>> sto.result = dd["total_mom_x"]
>>>>>>
>>>>>> ######################################################
>>>>>>
>>>>>> which is just overwriting my energy with the momentum. I also tried
>>>>>> to just set to different dictionaries without passing it to
>>>>>> yt.parallel_objects as storage, but of course each proc as only a local
>>>>>> copy then.
>>>>>>
>>>>>> Cordially,
>>>>>>
>>>>>> Josh
>>>>>>
>>>>>> _______________________________________________
>>>>>> yt-users mailing list
>>>>>> yt-users at lists.spacepope.org
>>>>>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>>>>>>
>>>>>>
>>>>> _______________________________________________
>>>>> yt-users mailing list
>>>>> yt-users at lists.spacepope.org
>>>>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>>>>>
>>>>
>>> _______________________________________________
>>> yt-users mailing list
>>> yt-users at lists.spacepope.org
>>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>>>
>>>
>> _______________________________________________
>> yt-users mailing list
>> yt-users at lists.spacepope.org
>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>>
>
> _______________________________________________
> yt-users mailing list
> yt-users at lists.spacepope.org
> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.spacepope.org/pipermail/yt-users-spacepope.org/attachments/20150527/cea0ec46/attachment.html>
-------------- next part --------------
_______________________________________________
yt-users mailing list
yt-users at lists.spacepope.org
http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
More information about the yt-users
mailing list