[yt-users] Running Rockstar Halo Finder to create a merger tree

Mon Nov 4 17:56:30 PST 2013

I suggest using parallel HOP since it is more scalable than HOP on large
datasets.  With HOP you can get into trouble where the padding takes up too
much overhead.

From
G.S.

On Mon, Nov 4, 2013 at 5:37 PM, Hilary Egan <hilaryye at gmail.com> wrote:

> Sam: Yep, I am definitely sending the infiniband-disabling flags and
> running it as follows
>
> mpirun -np 48 --mca btl ^openib python halos.py --parallel
>
> Cameron: I had originally switched to rockstar from HOP because I couldn't
> get it to run on kraken due to (very clear) out of memory issues. I suppose
> that since I've switched supercomputers anyway, I might as well play around
> with HOP again.
>
>
> On Mon, Nov 4, 2013 at 5:57 PM, Sam Skillman <samskillman at gmail.com>wrote:
>
>> Cameron -- My experience has been that HOP/FOF are slow on datasets this
>> large, and PHOP uses a large amount of memory.  Anyways, regardless of that
>> we should figure out what is going on with rockstar given its use in the
>> community.
>>
>> Hilary -- Just to be clear, you are sending the infiniband-disabling
>> flags to this last run, correct?
>>
>>
>>
>> On Mon, Nov 4, 2013 at 4:36 PM, Cameron Hummels <chummels at gmail.com>wrote:
>>
>>> Are you tied to using rockstar?  I know yt works well with HOP, PHOP,
>>> and FOF without many problems...
>>>
>>>
>>> On Mon, Nov 4, 2013 at 3:50 PM, Hilary Egan <hilaryye at gmail.com> wrote:
>>>
>>>> Hi all,
>>>>
>>>> I'm still having trouble getting rockstar to run on my dataset. I've
>>>> moved my data to a different super computer that uses the mpirun command
>>>> rather than kraken's aprun so that I could make sure there aren't any
>>>> infiniband issues, but Im still seeing similar issues. I've also determined
>>>> I can run the halo finder just fine on a smaller test dataset, which is
>>>> leading me to believe that its some sort of memory issue, but I can't quite
>>>> figure out how I would go about fixing it. I've tried playing with the
>>>> number of readers and the number of nodes I'm running on, to no avail. For
>>>> reference, the dataset is a 1024^3 unigrid enzo run. If anyone has any
>>>> suggestions, I'd love to hear them!
>>>>
>>>> Thanks!
>>>> Hilary
>>>>
>>>> Script: http://paste.yt-project.org/show/4025/
>>>>
>>>> Error message: http://paste.yt-project.org/show/4024/
>>>>
>>>>
>>>> On Wed, Oct 30, 2013 at 8:50 AM, John Wise <jwise at physics.gatech.edu>wrote:
>>>>
>>>>> Hi Hilary,
>>>>>
>>>>>
>>>>> On 10/29/2013 12:30 PM, Hilary Egan wrote:
>>>>>
>>>>>  I'm quite confused on a number of points related to running the
>>>>>> rockstar
>>>>>> halo finder, so I hope its alright that I put all these questions into
>>>>>> this one email!
>>>>>>
>>>>>
>>>>> It's no problem at all to include all of your questions in a single
>>>>> email.  It's probably better this way!
>>>>>
>>>>>
>>>>>  1. I can't seem to run the rockstar halo finder at all without getting
>>>>>> this error followed by a segmentation fault and crash.
>>>>>>
>>>>>> [Warning] Network IO Failure (PID XXXXXX): Connection reset by peer
>>>>>> [Network] Packet receive retry count at: 1
>>>>>>
>>>>>> It sort of seems like this issue
>>>>>> (http://lists.spacepope.org/htdig.cgi/yt-dev-spacepope.
>>>>>> org/2012-November/002681.html)
>>>>>> but I couldn't really figure out what the resolution was from the
>>>>>> thread. Im attempting to run this on kraken and it doesn't matter if I
>>>>>> use a single compute node or multiple, I get the same error. (I hope
>>>>>> this isn't the infiniband issue the docs warned about, I couldn't
>>>>>> figure
>>>>>> out if that is how kraken is connected and I got an error that the
>>>>>> suggested flag doesn't exist so I didn't press the issue.)
>>>>>>
>>>>>
>>>>> I haven't seen that error before, but I still have to specific to
>>>>> *not* run on infiniband when running Rockstar on a single node.  With
>>>>> OpenMPI, you would use "mpirun -n 32 --mca btl ^openib ...", but I haven't
>>>>> done this on kraken with their aprun but hopefully it's easily
>>>>> accompolished!
>>>>>
>>>>>
>>>>>  2. Whenever I finally do get the halo finder to work, I need the
>>>>>> results
>>>>>> to be in a form that the merger tree can use. It seems as though the
>>>>>> MergerTree needs the results in the same form as the other halo
>>>>>> finders
>>>>>> give, so would getting the halo list and then dumping it as usual be
>>>>>> the
>>>>>> appropriate strategy? Ie:
>>>>>>
>>>>>>          rh.run()
>>>>>>          halo_list = rh.halo_list()
>>>>>> halo_list.dump('MergerHalos')
>>>>>>
>>>>>> 2.5. The docs sort of give mixed messages on whether or not I could
>>>>>> just
>>>>>> be calling MergerTree with the argument halo_finder_function =
>>>>>> RockstarHaloFinder. At this point I've pretty thoroughly convinced
>>>>>> myself that I can't, but it would be nice if that was clarified.
>>>>>> (Just a
>>>>>> thoroughly overwhelmed new user's perspective!)
>>>>>>
>>>>>
>>>>> I'm not sure whether you can use yt's merger tree code with the
>>>>> Rockstar halos.  I haven't tried.
>>>>>
>>>>> However, I've used Consistent Trees
>>>>>
>>>>> https://code.google.com/p/consistent-trees/
>>>>>
>>>>> with Rockstar's halo lists, which is also written by Peter Behroozi.
>>>>> I've chosen this route because the algorithm seems to be more physically
>>>>> robust in constructing parent/child relationships and boundness.  All of
>>>>> the instructions are in the README of the code, and it's pretty
>>>>> straightforward and fast (probably 5-10 minutes for a 512^3 simulation with
>>>>> 60 outputs) to run.
>>>>>
>>>>> I also have a visualizing script for consistent tree's output.
>>>>>
>>>>> https://bitbucket.org/jwise77/rockstar-dot
>>>>>
>>>>> From consistent tree's output, you can use the provided script,
>>>>> halo_trees_to_catalog.pl, (instructions also in the README) to
>>>>> convert the tree output into halo lists.
>>>>>
>>>>>
>>>>>  3. I'm a little confused as to whether or not I have to use a
>>>>>> TimeSeriesData object rather than the usual single time output when
>>>>>> instantiating the halo finder. Under "Rockstar Halo Finding" it uses
>>>>>> TimeSeriesData, unlike the rest of the examples, but under the
>>>>>> subheading "Output Analysis" it just uses pf. The "Output Analysis"
>>>>>> example also doesn't call the run() method, which leads me to believe
>>>>>> something else entirely is going on, but its not quite clear.
>>>>>>
>>>>>
>>>>> This actually came up recently.  It's best to supply a TimeSeriesData
>>>>> object.  Here's the link to the email for more details.
>>>>>
>>>>> http://lists.spacepope.org/pipermail/yt-users-spacepope.
>>>>> org/2013-August/003845.html
>>>>>
>>>>> Cheers,
>>>>> John
>>>>>
>>>>> --
>>>>> John Wise
>>>>> Assistant Professor of Physics
>>>>> Center for Relativistic Astrophysics, Georgia Tech
>>>>> http://cosmo.gatech.edu
>>>>> _______________________________________________
>>>>> yt-users mailing list
>>>>> yt-users at lists.spacepope.org
>>>>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> yt-users mailing list
>>>> yt-users at lists.spacepope.org
>>>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>>>>
>>>>
>>>
>>>
>>> --
>>> Cameron Hummels
>>> Postdoctoral Researcher
>>> Steward Observatory
>>> University of Arizona
>>> http://chummels.org
>>>
>>> _______________________________________________
>>> yt-users mailing list
>>> yt-users at lists.spacepope.org
>>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>>>
>>>
>>
>> _______________________________________________
>> yt-users mailing list
>> yt-users at lists.spacepope.org
>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>>
>>
>
> _______________________________________________
> yt-users mailing list
> yt-users at lists.spacepope.org
> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.spacepope.org/pipermail/yt-users-spacepope.org/attachments/20131104/4efe1a51/attachment.html>