[yt-users] Storing multiple calculations using yt.parallel_objects

[Britton Smith]

Hi Josh,

Glad to hear you've got it working.  I have just one comment on your
script.  I don't think you want to use the i variable to load the datasets
from the plot list.  Since you are incrementing i by one, each process will
be loading datasets starting from 0 and going up to the total divided by
the number of processors.  Instead, since you're iterating over the plot
list, you can just do:
ds = yt.load(p, particle_filename=p)

Britton

On Sun, May 24, 2015 at 5:59 PM, Joshua Wall <joshua.e.wall at gmail.com>
wrote:

> Okay, I *think* I got it sorted out. Here's my solution (for future use if
> deemed correct!)
>
> plot = glob.glob("./SinkMomTest/*plt_cnt_000?")
> plot.sort()
> part = glob.glob("./SinkMomTest/*part_000?")
> part.sort()
> #print plot       # Testing
>
> # Make a list of variables you are collecting to plot/store as
> # dictionaries.
>
> my_storage = {}
>
> # Now the parallel loop.
>
> i = 0
>
> for sto,p in yt.parallel_objects(plot, num_procs, storage = my_storage):
>
>     ds = yt.load(plot[i], particle_filename=part[i])
>     dd = ds.all_data()
>
>     if (yt.is_root()):
>         print "Yo I'm root!"
>
>     if (~yt.is_root()):
>         print "Not root!"
>
>     momx                             = dd["total_px"]
>     momy                             = dd["total_py"]
>     partmomx             = dd["total_part_px"]
>     partmomy             = dd["total_part_py"]
>
>     sto.result_id = p[-4:]
>     sto.result    = [momx,momy,partmomx,partmomy]
>
>     i=i+1
>
>
> print "Done!"
>
> if yt.is_root():
>
>                 my_storage =
> collections.OrderedDict(sorted(my_storage.items()))
>                 print my_storage
>
> Thanks again for the help,
>
> Josh
>
> On Sun, May 24, 2015 at 6:33 PM Joshua Wall <joshua.e.wall at gmail.com>
> wrote:
>
>> Dear Britton,
>>
>>
>> I tried something like that, but clearly I'm not doing it correctly. I'm
>> noticing two errors, 1) everything is reporting running on the root proc
>> (with a "Yo I'm root!") and 2) I'm getting four copies of my_storage back.
>> Here's the code:
>>
>> #########################################################################
>>
>> from matplotlib import rc
>> import matplotlib.pyplot as plt
>> import yt
>> from yt import derived_field
>> import numpy as np
>> import yt.units as u
>> import glob
>>
>> # Use YT in parallel.
>> yt.enable_parallelism()
>>
>> # Number of procs on the machine
>> num_procs=4
>>
>>
>> @derived_field(name = "total_KE", units = "erg")
>> def _KE(field, data):
>>     return
>> np.sum(data["kinetic_energy"].in_mks()*data["cell_volume"].in_cgs())
>>
>> @derived_field(name = "total_PE", units = "erg")
>> def _PE(field, data):
>>     return 0.5*np.sum(data["gpot"].in_mks()*data["cell_mass"].in_cgs())
>>
>> @derived_field(name = "total_Eint", units = "erg")
>> def _Eint(field, data):
>>     return np.sum(data["eint"]*data["cell_mass"]).in_cgs()
>>
>> @derived_field(name = "total_energy", units = "erg")
>> def _Etotal(field, data):
>>     return data["total_Eint"] + data["total_PE"] + data["total_KE"]
>>
>> @derived_field(name = "total_px", units = "g*cm/s")
>> def _px(field, data):
>>     return np.sum(data["cell_mass"]*data["velx"]).in_cgs()
>>
>> # Use glob to gather up the data files. Note here we use plt
>> # and particle files because they are much smaller than
>> # checkpoint files.
>>
>> plot = glob.glob("./SinkMomTest/*plt_cnt_000?")
>> plot.sort()
>> part = glob.glob("./SinkMomTest/*part_000?")
>> part.sort()
>>
>> # Make a list of variables you are collecting to plot/store as
>> # dictionaries.
>> my_storage = {}
>>
>> # Now the parallel loop.
>>
>> i = 0
>>
>> for sto,p in yt.parallel_objects(plot, num_procs, storage = my_storage):
>>
>>     ds = yt.load(plot[i], particle_filename=part[i])
>>     dd = ds.all_data()
>>
>>     if (yt.is_root()):
>>         print "Yo I'm root!"
>>     else:
>>         print "Not root!"
>>
>>
>>     energy = dd["total_energy"]
>>     mom    = dd["total_px"]
>>
>>     sto.result_id = p[-4:]
>>     sto.result = [energy, mom]
>>
>>     i=i+1
>>
>>
>> print "Done!"
>>
>> print my_storage
>>
>> #########################################################################
>>
>> Is this what you mean?
>>
>> Kindly,
>>
>> Josh
>>
>>
>>
>> On Sun, May 24, 2015 at 12:19 PM Britton Smith <brittonsmith at gmail.com>
>> wrote:
>>
>>> Hi Josh,
>>>
>>> The reason that's not working is that there is still only one storage
>>> item (the sto variable) per iteration of the loop.  You could keep both
>>> things if you make sto.result a list object and just append each of those
>>> things to that list.
>>>
>>> Britton
>>>
>>> On Sun, May 24, 2015 at 11:12 AM, Joshua Wall <joshua.e.wall at gmail.com>
>>> wrote:
>>>
>>>> Dear users,
>>>>
>>>>
>>>>      This there a simple way to store multiple values at each step
>>>> using parallel_objects? I'm doing:
>>>>
>>>> ########################################################
>>>> yt.enable_parallelism()
>>>> num_procs = 4
>>>> plot = glob.glob(./plot*)
>>>> my_storage = {}
>>>>
>>>> for sto, p in yt.parallel_objects(plots, num_procs, my_storage):
>>>>
>>>>   sto.result_id = 'energy' + p
>>>>   sto.resutl     = dd["total_energy"]
>>>>
>>>>   sto.result_id = 'Px' + p
>>>>   sto.result     = dd["total_mom_x"]
>>>>
>>>> ######################################################
>>>>
>>>> which is just overwriting my energy with the momentum. I also tried to
>>>> just set to different dictionaries without passing it to
>>>> yt.parallel_objects as storage, but of course each proc as only a local
>>>> copy then.
>>>>
>>>> Cordially,
>>>>
>>>> Josh
>>>>
>>>> _______________________________________________
>>>> yt-users mailing list
>>>> yt-users at lists.spacepope.org
>>>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>>>>
>>>>
>>> _______________________________________________
>>> yt-users mailing list
>>> yt-users at lists.spacepope.org
>>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>>>
>>
> _______________________________________________
> yt-users mailing list
> yt-users at lists.spacepope.org
> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.spacepope.org/pipermail/yt-users-spacepope.org/attachments/20150527/7d6ef38a/attachment.htm>
-------------- next part --------------
_______________________________________________
yt-users mailing list
yt-users at lists.spacepope.org
http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org