<div dir="ltr">Sam: Yep, I am definitely sending the infiniband-disabling flags and running it as follows<div><div><br></div><div>mpirun -np 48 --mca btl ^openib python halos.py --parallel<br></div><div><br></div><div>Cameron: I had originally switched to rockstar from HOP because I couldn't get it to run on kraken due to (very clear) out of memory issues. I suppose that since I've switched supercomputers anyway, I might as well play around with HOP again.</div>
</div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Mon, Nov 4, 2013 at 5:57 PM, Sam Skillman <span dir="ltr"><<a href="mailto:samskillman@gmail.com" target="_blank">samskillman@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div>Cameron -- My experience has been that HOP/FOF are slow on datasets this large, and PHOP uses a large amount of memory. Anyways, regardless of that we should figure out what is going on with rockstar given its use in the community.<br>
</div>
<div><br></div><div><div>Hilary -- Just to be clear, you are sending the infiniband-disabling flags to this last run, correct? </div></div><div><br></div></div><div class="HOEnZb"><div class="h5"><div class="gmail_extra">
<br><br><div class="gmail_quote">On Mon, Nov 4, 2013 at 4:36 PM, Cameron Hummels <span dir="ltr"><<a href="mailto:chummels@gmail.com" target="_blank">chummels@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Are you tied to using rockstar? I know yt works well with HOP, PHOP, and FOF without many problems...</div>
<div class="gmail_extra"><div><div><br><br><div class="gmail_quote">On Mon, Nov 4, 2013 at 3:50 PM, Hilary Egan <span dir="ltr"><<a href="mailto:hilaryye@gmail.com" target="_blank">hilaryye@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Hi all,<div><br></div><div>I'm still having trouble getting rockstar to run on my dataset. I've moved my data to a different super computer that uses the mpirun command rather than kraken's aprun so that I could make sure there aren't any infiniband issues, but Im still seeing similar issues. I've also determined I can run the halo finder just fine on a smaller test dataset, which is leading me to believe that its some sort of memory issue, but I can't quite figure out how I would go about fixing it. I've tried playing with the number of readers and the number of nodes I'm running on, to no avail. For reference, the dataset is a 1024^3 unigrid enzo run. If anyone has any suggestions, I'd love to hear them!</div>
<div><br></div><div>Thanks!</div><div>Hilary</div><div><br></div><div>Script: <a href="http://paste.yt-project.org/show/4025/" target="_blank">http://paste.yt-project.org/show/4025/</a></div><div><br></div><div>Error message: <a href="http://paste.yt-project.org/show/4024/" target="_blank">http://paste.yt-project.org/show/4024/</a></div>
</div><div><div><div class="gmail_extra"><br><br><div class="gmail_quote">On Wed, Oct 30, 2013 at 8:50 AM, John Wise <span dir="ltr"><<a href="mailto:jwise@physics.gatech.edu" target="_blank">jwise@physics.gatech.edu</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi Hilary,<div><br>
<br>
On 10/29/2013 12:30 PM, Hilary Egan wrote:<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
I'm quite confused on a number of points related to running the rockstar<br>
halo finder, so I hope its alright that I put all these questions into<br>
this one email!<br>
</blockquote>
<br></div>
It's no problem at all to include all of your questions in a single email. It's probably better this way!<div><br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
1. I can't seem to run the rockstar halo finder at all without getting<br>
this error followed by a segmentation fault and crash.<br>
<br>
[Warning] Network IO Failure (PID XXXXXX): Connection reset by peer<br>
[Network] Packet receive retry count at: 1<br>
<br>
It sort of seems like this issue<br>
(<a href="http://lists.spacepope.org/htdig.cgi/yt-dev-spacepope.org/2012-November/002681.html" target="_blank">http://lists.spacepope.org/<u></u>htdig.cgi/yt-dev-spacepope.<u></u>org/2012-November/002681.html</a>)<br>
but I couldn't really figure out what the resolution was from the<br>
thread. Im attempting to run this on kraken and it doesn't matter if I<br>
use a single compute node or multiple, I get the same error. (I hope<br>
this isn't the infiniband issue the docs warned about, I couldn't figure<br>
out if that is how kraken is connected and I got an error that the<br>
suggested flag doesn't exist so I didn't press the issue.)<br>
</blockquote>
<br></div>
I haven't seen that error before, but I still have to specific to *not* run on infiniband when running Rockstar on a single node. With OpenMPI, you would use "mpirun -n 32 --mca btl ^openib ...", but I haven't done this on kraken with their aprun but hopefully it's easily accompolished!<div>
<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
2. Whenever I finally do get the halo finder to work, I need the results<br>
to be in a form that the merger tree can use. It seems as though the<br>
MergerTree needs the results in the same form as the other halo finders<br>
give, so would getting the halo list and then dumping it as usual be the<br>
appropriate strategy? Ie:<br>
<br>
rh.run()<br>
halo_list = rh.halo_list()<br>
halo_list.dump('MergerHalos')<br>
<br>
2.5. The docs sort of give mixed messages on whether or not I could just<br>
be calling MergerTree with the argument halo_finder_function =<br>
RockstarHaloFinder. At this point I've pretty thoroughly convinced<br>
myself that I can't, but it would be nice if that was clarified. (Just a<br>
thoroughly overwhelmed new user's perspective!)<br>
</blockquote>
<br></div>
I'm not sure whether you can use yt's merger tree code with the Rockstar halos. I haven't tried.<br>
<br>
However, I've used Consistent Trees<br>
<br>
<a href="https://code.google.com/p/consistent-trees/" target="_blank">https://code.google.com/p/<u></u>consistent-trees/</a><br>
<br>
with Rockstar's halo lists, which is also written by Peter Behroozi. I've chosen this route because the algorithm seems to be more physically robust in constructing parent/child relationships and boundness. All of the instructions are in the README of the code, and it's pretty straightforward and fast (probably 5-10 minutes for a 512^3 simulation with 60 outputs) to run.<br>
<br>
I also have a visualizing script for consistent tree's output.<br>
<br>
<a href="https://bitbucket.org/jwise77/rockstar-dot" target="_blank">https://bitbucket.org/jwise77/<u></u>rockstar-dot</a><br>
<br>
>From consistent tree's output, you can use the provided script, <a href="http://halo_trees_to_catalog.pl" target="_blank">halo_trees_to_catalog.pl</a>, (instructions also in the README) to convert the tree output into halo lists.<div>
<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
3. I'm a little confused as to whether or not I have to use a<br>
TimeSeriesData object rather than the usual single time output when<br>
instantiating the halo finder. Under "Rockstar Halo Finding" it uses<br>
TimeSeriesData, unlike the rest of the examples, but under the<br>
subheading "Output Analysis" it just uses pf. The "Output Analysis"<br>
example also doesn't call the run() method, which leads me to believe<br>
something else entirely is going on, but its not quite clear.<br>
</blockquote>
<br></div>
This actually came up recently. It's best to supply a TimeSeriesData object. Here's the link to the email for more details.<br>
<br>
<a href="http://lists.spacepope.org/pipermail/yt-users-spacepope.org/2013-August/003845.html" target="_blank">http://lists.spacepope.org/<u></u>pipermail/yt-users-spacepope.<u></u>org/2013-August/003845.html</a><br>
<br>
Cheers,<br>
John<span><font color="#888888"><br>
<br>
-- <br>
John Wise<br>
Assistant Professor of Physics<br>
Center for Relativistic Astrophysics, Georgia Tech<br>
<a href="http://cosmo.gatech.edu" target="_blank">http://cosmo.gatech.edu</a><br>
______________________________<u></u>_________________<br>
yt-users mailing list<br>
<a href="mailto:yt-users@lists.spacepope.org" target="_blank">yt-users@lists.spacepope.org</a><br>
<a href="http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org" target="_blank">http://lists.spacepope.org/<u></u>listinfo.cgi/yt-users-<u></u>spacepope.org</a><br>
</font></span></blockquote></div><br></div>
</div></div><br>_______________________________________________<br>
yt-users mailing list<br>
<a href="mailto:yt-users@lists.spacepope.org" target="_blank">yt-users@lists.spacepope.org</a><br>
<a href="http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org" target="_blank">http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org</a><br>
<br></blockquote></div><br><br clear="all"><div><br></div></div></div><span><font color="#888888">-- <br>Cameron Hummels<div>Postdoctoral Researcher</div><div>Steward Observatory</div><div>University of Arizona</div>
<div><a href="http://chummels.org" target="_blank">http://chummels.org</a></div>
</font></span></div>
<br>_______________________________________________<br>
yt-users mailing list<br>
<a href="mailto:yt-users@lists.spacepope.org" target="_blank">yt-users@lists.spacepope.org</a><br>
<a href="http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org" target="_blank">http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org</a><br>
<br></blockquote></div><br></div>
</div></div><br>_______________________________________________<br>
yt-users mailing list<br>
<a href="mailto:yt-users@lists.spacepope.org">yt-users@lists.spacepope.org</a><br>
<a href="http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org" target="_blank">http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org</a><br>
<br></blockquote></div><br></div>