<meta http-equiv="Content-Type" content="text/html; charset=utf-8"><div dir="ltr"><div><div><div><div><div>Britton,<br><br></div> Indeed, I have noticed in the runs that work was being repeated. I'm not quite sure how to get around it, since the particle filenames are different than the plot file names. Perhaps I can do?<br><br></div>for plt_file, part_file in plot, part:<br><br></div> ds = yt.load(plt_file, particle_filename=part_file)<br><br></div>Thnaks again,<br><br></div>Josh<br></div><br><div class="gmail_quote"><div dir="ltr">On Wed, May 27, 2015 at 11:14 AM Britton Smith <<a href="mailto:brittonsmith@gmail.com">brittonsmith@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Hi Josh,<div><br></div><div>Glad to hear you've got it working. I have just one comment on your script. I don't think you want to use the i variable to load the datasets from the plot list. Since you are incrementing i by one, each process will be loading datasets starting from 0 and going up to the total divided by the number of processors. Instead, since you're iterating over the plot list, you can just do:</div><div><span style="color:rgb(80,0,80);font-size:12.8000001907349px">ds = yt.load(p, particle_filename=p)</span><br></div></div><div dir="ltr"><div><span style="color:rgb(80,0,80);font-size:12.8000001907349px"><br></span></div><div><span style="color:rgb(80,0,80);font-size:12.8000001907349px">Britton</span></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Sun, May 24, 2015 at 5:59 PM, Joshua Wall <span dir="ltr"><<a href="mailto:joshua.e.wall@gmail.com" target="_blank">joshua.e.wall@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><div>Okay, I *think* I got it sorted out. Here's my solution (for future use if deemed correct!)<span><br><br>plot = glob.glob("./SinkMomTest/*plt_cnt_000?")<br>plot.sort()<br>part = glob.glob("./SinkMomTest/*part_000?")<br>part.sort()<br></span>#print plot # Testing<span><br><br># Make a list of variables you are collecting to plot/store as<br># dictionaries. <br><br>my_storage = {}<br><br># Now the parallel loop.<br><br>i = 0<br><br>for sto,p in yt.parallel_objects(plot, num_procs, storage = my_storage):<br><br> ds = yt.load(plot[i], particle_filename=part[i]) <br> dd = ds.all_data()<br> <br> if (yt.is_root()):<br> print "Yo I'm root!"<br> <br></span> if (~yt.is_root()):<br> print "Not root!"<br> <br> momx = dd["total_px"]<br> momy = dd["total_py"]<br> partmomx = dd["total_part_px"]<br> partmomy = dd["total_part_py"]<br> <br> sto.result_id = p[-4:]<br> sto.result = [momx,momy,partmomx,partmomy]<br> <br> i=i+1<br> <br> <br>print "Done!"<br><br>if yt.is_root():<br><br> my_storage = collections.OrderedDict(sorted(my_storage.items()))<br> print my_storage<br><br></div>Thanks again for the help,<br><br></div>Josh<br></div><div><div><br><div class="gmail_quote">On Sun, May 24, 2015 at 6:33 PM Joshua Wall <<a href="mailto:joshua.e.wall@gmail.com" target="_blank">joshua.e.wall@gmail.com</a>> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><div>Dear Britton,<br><br></div><div><br>I tried something like that, but clearly I'm not doing it correctly. I'm noticing two errors, 1) everything is reporting running on the root proc (with a "Yo I'm root!") and 2) I'm getting four copies of my_storage back. Here's the code:<br><br>#########################################################################<br><br>from matplotlib import rc<br>import matplotlib.pyplot as plt<br>import yt<br>from yt import derived_field<br>import numpy as np<br>import yt.units as u<br>import glob<br><br># Use YT in parallel.<br>yt.enable_parallelism()<br><br># Number of procs on the machine<br>num_procs=4<br><br><br>@derived_field(name = "total_KE", units = "erg")<br>def _KE(field, data):<br> return np.sum(data["kinetic_energy"].in_mks()*data["cell_volume"].in_cgs())<br><br>@derived_field(name = "total_PE", units = "erg")<br>def _PE(field, data):<br> return 0.5*np.sum(data["gpot"].in_mks()*data["cell_mass"].in_cgs())<br><br>@derived_field(name = "total_Eint", units = "erg")<br>def _Eint(field, data):<br> return np.sum(data["eint"]*data["cell_mass"]).in_cgs()<br><br>@derived_field(name = "total_energy", units = "erg")<br>def _Etotal(field, data):<br> return data["total_Eint"] + data["total_PE"] + data["total_KE"]<br><br>@derived_field(name = "total_px", units = "g*cm/s")<br>def _px(field, data):<br> return np.sum(data["cell_mass"]*data["velx"]).in_cgs()<br><br># Use glob to gather up the data files. Note here we use plt<br># and particle files because they are much smaller than <br># checkpoint files.<br><br>plot = glob.glob("./SinkMomTest/*plt_cnt_000?")<br>plot.sort()<br>part = glob.glob("./SinkMomTest/*part_000?")<br>part.sort()<br><br># Make a list of variables you are collecting to plot/store as<br># dictionaries. <br>my_storage = {}<br><br># Now the parallel loop.<br><br>i = 0<br><br>for sto,p in yt.parallel_objects(plot, num_procs, storage = my_storage):<br><br> ds = yt.load(plot[i], particle_filename=part[i])<br> dd = ds.all_data()<br> <br> if (yt.is_root()):<br> print "Yo I'm root!"<br> else:<br> print "Not root!"<br> <br><br> energy = dd["total_energy"]<br> mom = dd["total_px"]<br> <br> sto.result_id = p[-4:]<br> sto.result = [energy, mom]<br> <br> i=i+1<br> <br> <br>print "Done!"<br><br>print my_storage<br><br>#########################################################################<br><br></div>Is this what you mean?<br><br></div><div>Kindly,<br></div><div><br></div>Josh<br><div><div><div><br><br></div></div></div></div><br><div class="gmail_quote">On Sun, May 24, 2015 at 12:19 PM Britton Smith <<a href="mailto:brittonsmith@gmail.com" target="_blank">brittonsmith@gmail.com</a>> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Hi Josh,<div><br></div><div>The reason that's not working is that there is still only one storage item (the sto variable) per iteration of the loop. You could keep both things if you make sto.result a list object and just append each of those things to that list.</div><div><br></div><div>Britton</div></div><div class="gmail_extra"><br><div class="gmail_quote"></div></div><div class="gmail_extra"><div class="gmail_quote">On Sun, May 24, 2015 at 11:12 AM, Joshua Wall <span dir="ltr"><<a href="mailto:joshua.e.wall@gmail.com" target="_blank">joshua.e.wall@gmail.com</a>></span> wrote:<br></div></div><div class="gmail_extra"><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><div><div><div><div><div><div><div><div><div>Dear users,<br><br><br></div> This there a simple way to store multiple values at each step using parallel_objects? I'm doing:<br><br>########################################################<br></div><div>yt.enable_parallelism()<br>num_procs = 4<br></div><div>plot = glob.glob(./plot*)<br></div>my_storage = {}<br><br></div>for sto, p in yt.parallel_objects(plots, num_procs, my_storage):<br><br></div> sto.result_id = 'energy' + p<br></div> sto.resutl = dd["total_energy"]<br><br></div> sto.result_id = 'Px' + p<br></div> sto.result = dd["total_mom_x"]<br><br>######################################################<br><br></div>which is just overwriting my energy with the momentum. I also tried to just set to different dictionaries without passing it to yt.parallel_objects as storage, but of course each proc as only a local copy then.<br><br></div>Cordially,<br></div><div><br></div>Josh<br></div>
<br></blockquote></div></div><div class="gmail_extra"><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">_______________________________________________<br>
yt-users mailing list<br>
<a href="mailto:yt-users@lists.spacepope.org" target="_blank">yt-users@lists.spacepope.org</a><br>
<a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.spacepope.org_listinfo.cgi_yt-2Dusers-2Dspacepope.org&d=AwMFaQ&c=8hUWFZcy2Z-Za5rBPlktOQ&r=hgcBC3x6dKFoTrmFmMYYbKNfiHZlGLKliIidd1LwmHI&m=JsX-lo9MJWzwaxPbUuNKHvYhL4Dz7u9m3XIuxY0ZWuQ&s=uI54esC-Lg2pwyEmZWpABSSHHiKqejcExKlSaIbsOLc&e=" target="_blank">http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org</a><br>
<br></blockquote></div><br></div>
_______________________________________________<br>
yt-users mailing list<br>
<a href="mailto:yt-users@lists.spacepope.org" target="_blank">yt-users@lists.spacepope.org</a><br>
<a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.spacepope.org_listinfo.cgi_yt-2Dusers-2Dspacepope.org&d=AwMFaQ&c=8hUWFZcy2Z-Za5rBPlktOQ&r=hgcBC3x6dKFoTrmFmMYYbKNfiHZlGLKliIidd1LwmHI&m=JsX-lo9MJWzwaxPbUuNKHvYhL4Dz7u9m3XIuxY0ZWuQ&s=uI54esC-Lg2pwyEmZWpABSSHHiKqejcExKlSaIbsOLc&e=" target="_blank">http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org</a><br>
</blockquote></div></blockquote></div>
</div></div><br>_______________________________________________<br>
yt-users mailing list<br>
<a href="mailto:yt-users@lists.spacepope.org" target="_blank">yt-users@lists.spacepope.org</a><br>
<a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.spacepope.org_listinfo.cgi_yt-2Dusers-2Dspacepope.org&d=AwMFaQ&c=8hUWFZcy2Z-Za5rBPlktOQ&r=hgcBC3x6dKFoTrmFmMYYbKNfiHZlGLKliIidd1LwmHI&m=JsX-lo9MJWzwaxPbUuNKHvYhL4Dz7u9m3XIuxY0ZWuQ&s=uI54esC-Lg2pwyEmZWpABSSHHiKqejcExKlSaIbsOLc&e=" target="_blank">http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org</a><br>
<br></blockquote></div><br></div>
_______________________________________________<br>
yt-users mailing list<br>
<a href="mailto:yt-users@lists.spacepope.org" target="_blank">yt-users@lists.spacepope.org</a><br>
<a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.spacepope.org_listinfo.cgi_yt-2Dusers-2Dspacepope.org&d=AwMFaQ&c=8hUWFZcy2Z-Za5rBPlktOQ&r=hgcBC3x6dKFoTrmFmMYYbKNfiHZlGLKliIidd1LwmHI&m=JsX-lo9MJWzwaxPbUuNKHvYhL4Dz7u9m3XIuxY0ZWuQ&s=uI54esC-Lg2pwyEmZWpABSSHHiKqejcExKlSaIbsOLc&e=" target="_blank">http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org</a><br>
</blockquote></div>