[yt-users] Storing multiple calculations using yt.parallel_objects

[Joshua Wall]

Okay, I *think* I got it sorted out. Here's my solution (for future use if
deemed correct!)

plot = glob.glob("./SinkMomTest/*plt_cnt_000?")
plot.sort()
part = glob.glob("./SinkMomTest/*part_000?")
part.sort()
#print plot       # Testing

# Make a list of variables you are collecting to plot/store as
# dictionaries.

my_storage = {}

# Now the parallel loop.

i = 0

for sto,p in yt.parallel_objects(plot, num_procs, storage = my_storage):

    ds = yt.load(plot[i], particle_filename=part[i])
    dd = ds.all_data()

    if (yt.is_root()):
        print "Yo I'm root!"

    if (~yt.is_root()):
        print "Not root!"

    momx                             = dd["total_px"]
    momy                             = dd["total_py"]
    partmomx             = dd["total_part_px"]
    partmomy             = dd["total_part_py"]

    sto.result_id = p[-4:]
    sto.result    = [momx,momy,partmomx,partmomy]

    i=i+1


print "Done!"

if yt.is_root():

                my_storage =
collections.OrderedDict(sorted(my_storage.items()))
                print my_storage

Thanks again for the help,

Josh

On Sun, May 24, 2015 at 6:33 PM Joshua Wall <joshua.e.wall at gmail.com> wrote:

> Dear Britton,
>
>
> I tried something like that, but clearly I'm not doing it correctly. I'm
> noticing two errors, 1) everything is reporting running on the root proc
> (with a "Yo I'm root!") and 2) I'm getting four copies of my_storage back.
> Here's the code:
>
> #########################################################################
>
> from matplotlib import rc
> import matplotlib.pyplot as plt
> import yt
> from yt import derived_field
> import numpy as np
> import yt.units as u
> import glob
>
> # Use YT in parallel.
> yt.enable_parallelism()
>
> # Number of procs on the machine
> num_procs=4
>
>
> @derived_field(name = "total_KE", units = "erg")
> def _KE(field, data):
>     return
> np.sum(data["kinetic_energy"].in_mks()*data["cell_volume"].in_cgs())
>
> @derived_field(name = "total_PE", units = "erg")
> def _PE(field, data):
>     return 0.5*np.sum(data["gpot"].in_mks()*data["cell_mass"].in_cgs())
>
> @derived_field(name = "total_Eint", units = "erg")
> def _Eint(field, data):
>     return np.sum(data["eint"]*data["cell_mass"]).in_cgs()
>
> @derived_field(name = "total_energy", units = "erg")
> def _Etotal(field, data):
>     return data["total_Eint"] + data["total_PE"] + data["total_KE"]
>
> @derived_field(name = "total_px", units = "g*cm/s")
> def _px(field, data):
>     return np.sum(data["cell_mass"]*data["velx"]).in_cgs()
>
> # Use glob to gather up the data files. Note here we use plt
> # and particle files because they are much smaller than
> # checkpoint files.
>
> plot = glob.glob("./SinkMomTest/*plt_cnt_000?")
> plot.sort()
> part = glob.glob("./SinkMomTest/*part_000?")
> part.sort()
>
> # Make a list of variables you are collecting to plot/store as
> # dictionaries.
> my_storage = {}
>
> # Now the parallel loop.
>
> i = 0
>
> for sto,p in yt.parallel_objects(plot, num_procs, storage = my_storage):
>
>     ds = yt.load(plot[i], particle_filename=part[i])
>     dd = ds.all_data()
>
>     if (yt.is_root()):
>         print "Yo I'm root!"
>     else:
>         print "Not root!"
>
>
>     energy = dd["total_energy"]
>     mom    = dd["total_px"]
>
>     sto.result_id = p[-4:]
>     sto.result = [energy, mom]
>
>     i=i+1
>
>
> print "Done!"
>
> print my_storage
>
> #########################################################################
>
> Is this what you mean?
>
> Kindly,
>
> Josh
>
>
>
> On Sun, May 24, 2015 at 12:19 PM Britton Smith <brittonsmith at gmail.com>
> wrote:
>
>> Hi Josh,
>>
>> The reason that's not working is that there is still only one storage
>> item (the sto variable) per iteration of the loop.  You could keep both
>> things if you make sto.result a list object and just append each of those
>> things to that list.
>>
>> Britton
>>
>> On Sun, May 24, 2015 at 11:12 AM, Joshua Wall <joshua.e.wall at gmail.com>
>> wrote:
>>
>>> Dear users,
>>>
>>>
>>>      This there a simple way to store multiple values at each step using
>>> parallel_objects? I'm doing:
>>>
>>> ########################################################
>>> yt.enable_parallelism()
>>> num_procs = 4
>>> plot = glob.glob(./plot*)
>>> my_storage = {}
>>>
>>> for sto, p in yt.parallel_objects(plots, num_procs, my_storage):
>>>
>>>   sto.result_id = 'energy' + p
>>>   sto.resutl     = dd["total_energy"]
>>>
>>>   sto.result_id = 'Px' + p
>>>   sto.result     = dd["total_mom_x"]
>>>
>>> ######################################################
>>>
>>> which is just overwriting my energy with the momentum. I also tried to
>>> just set to different dictionaries without passing it to
>>> yt.parallel_objects as storage, but of course each proc as only a local
>>> copy then.
>>>
>>> Cordially,
>>>
>>> Josh
>>>
>>> _______________________________________________
>>> yt-users mailing list
>>> yt-users at lists.spacepope.org
>>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>>>
>>>
>> _______________________________________________
>> yt-users mailing list
>> yt-users at lists.spacepope.org
>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.spacepope.org/pipermail/yt-users-spacepope.org/attachments/20150524/5035c3bd/attachment.htm>
-------------- next part --------------
_______________________________________________
yt-users mailing list
yt-users at lists.spacepope.org
http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org