<div dir="ltr">I just want to add a little bit of information on why rockstar needs to run in parallel. Natively, rockstar has it's own parallelism in which separate processes are responsible for reading the data (readers), doing the computation and output (writers), and controlling everything (servers). There is always only one server, but the number of readers and writers can be changed with the n_readers and n_writers keywords (which you can add to the finder_kwargs dict in the HaloCatalog). At minimum, you need one server, one reader, and one writer, so rockstar must be run with at least 3 MPI processes.</div>
<div class="gmail_extra"><br><br><div class="gmail_quote">On Tue, Aug 12, 2014 at 3:31 PM, Sam Skillman <span dir="ltr"><<a href="mailto:samskillman@gmail.com" target="_blank">samskillman@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">This worked for me recently:<div><a href="http://paste.yt-project.org/show/5029/" target="_blank">http://paste.yt-project.org/show/5029/</a><br>
</div><div><br></div><div>With mpirun -np 3 python script.py </div><span class="HOEnZb"><font color="#888888"><div><br></div>
<div>Sam</div></font></span></div><div class="HOEnZb"><div class="h5"><div class="gmail_extra"><br><br><div class="gmail_quote">On Tue, Aug 12, 2014 at 7:21 AM, Matthew Turk <span dir="ltr"><<a href="mailto:matthewturk@gmail.com" target="_blank">matthewturk@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div>On Tue, Aug 12, 2014 at 9:15 AM, Cameron Hummels <<a href="mailto:chummels@gmail.com" target="_blank">chummels@gmail.com</a>> wrote:<br>
> Hi Britton, Sam, and David,<br>
><br>
> Thanks for the insights into this problem. I've made it farther, but I'm<br>
> still getting problems. I'll make sure that when we've resolved this, I'll<br>
> write it up and put it in the docs for other people trying to use Rockstar<br>
> with yt, as I think this may be a common occurrance since rockstar is now<br>
> the preferred halo finder with yt above hop and fof. A few comments:<br>
><br>
>> Cameron, you are correct, on Mac systems you need to set DYLD_LIBRARY_PATH<br>
>> instead of LD_LIBRARY_PATH. The fact that this causes vi and other things<br>
>> to fail is apparently an OS X bug (I found some discussions of it on the<br>
>> internet) and has nothing to do with yt. I went through a similar thing and<br>
>> had to resort to having a separate terminal for running rockstar that I<br>
>> didn't try to do anything else in. For what it's worth, this behavior seems<br>
>> to have been fixed. I am on OS X 10.8.5 and no longer have the issue of<br>
>> things crashing in a terminal where I have edited DYLD_LIBRARY_PATH.<br>
><br>
><br>
> That's odd, since I'm also using OS X 10.8.5 on both machines where I'm<br>
> getting this vi failure. And strangely, it appears to be somewhat related<br>
> to yt, according to the failure message when I try to run 'vi' from the<br>
> command-line seen here:<br>
><br>
> <a href="http://paste.yt-project.org/show/5022/" target="_blank">http://paste.yt-project.org/show/5022/</a><br>
<br>
</div>Looks like a conflict between system libpng and the yt stack's libpng.<br>
Might be fixable with an install where libpng is disabled.<br>
<div><div><br>
><br>
> Anyway, as you suggest, I can just run rockstar in a new terminal without<br>
> doing anything else, but I thought you might have seen this error before.<br>
><br>
>><br>
>> On the second issue that you both reported, the problem is the way in<br>
>> which one tells yt to run in parallel depends on how you import yt.<br>
>><br>
>> If you do "from yt.mods import *", the parallelism is enabled by adding<br>
>> the --parallel flag on the command line. However, if you do "import yt",<br>
>> then parallelism is enabled by doing "yt.enable_parallelism()" directly<br>
>> below the import statement within your script.<br>
>><br>
>> I looked at the latest documentation on parallelism<br>
>> (<a href="http://yt-project.org/doc/analyzing/parallel_computation.html" target="_blank">http://yt-project.org/doc/analyzing/parallel_computation.html</a>) and is<br>
>> discusses the use of yt.enable_parallelism(), but not the old way that works<br>
>> with "from yt.mods import *". There should probably be a short discussion<br>
>> added of how this has changed. Additionally, let's make sure that the<br>
>> yt-rockstar docs get any needed additions there as well.<br>
><br>
><br>
> OK, so one needs to run rockstar in parallel. I thought in reading the<br>
> rockstar docs, there was a way of running it with one processor, but OK, I<br>
> can run it in parallel. When I do that on this particular dataset, Enzo_64,<br>
> it fails because there are particles in the simulation of two separate<br>
> masses (stars and DM). I know that is a problem with this version of<br>
> rockstar as described in the docs here:<br>
><br>
> <a href="http://yt-project.org/docs/dev/analyzing/analysis_modules/halo_finders.html#rockstar-halo-finding" target="_blank">http://yt-project.org/docs/dev/analyzing/analysis_modules/halo_finders.html#rockstar-halo-finding</a><br>
><br>
> It also mentions in those docs that in order to deal with simulations with<br>
> multiple particle masses, "In simulations where the highest-resolution<br>
> particles all have the same mass (ie: zoom-in grid based simulations), one<br>
> can set up a particle filter to select the lowest mass particles and perform<br>
> the halo finding only on those."<br>
> I have tried a few things here to make this work. One, as suggested I<br>
> created a particle filter to just look at one mass of particles, and then I<br>
> applied it to the dataset, but I'm not sure how to pass that particle filter<br>
> to the Halo Catalog object. Here is my attempt:<br>
><br>
> <a href="http://paste.yt-project.org/show/5023/" target="_blank">http://paste.yt-project.org/show/5023/</a><br>
><br>
> And the corresponding failure traceback:<br>
><br>
> <a href="http://paste.yt-project.org/show/4IWCE1hZS1KGpTWztokW" target="_blank">http://paste.yt-project.org/show/4IWCE1hZS1KGpTWztokW</a><br>
><br>
> Alternatively, I see that the Halo Catalog object does take a "data_source"<br>
> kwarg, so I made a cut_region of all_data just selecting particles that were<br>
> the most massive, and again, I get the same failure. So I'm curious how to<br>
> use Rockstar to find halos on simulations that have multiple masses. I<br>
> guess I'm just missing something stupid here, as I know there are people who<br>
> have done this in the past. Does anyone have an example script or some hint<br>
> for me here? Here is my script using the cut region to get only the most<br>
> massive particles of a single mass, but it fails the same way as above:<br>
><br>
> <a href="http://paste.yt-project.org/show/5027/" target="_blank">http://paste.yt-project.org/show/5027/</a><br>
><br>
> Alternatively, it seems like the yt docs advocate using rockstar external to<br>
> yt, and then just using the output halos file for doing things (e.g.<br>
> <a href="http://yt-project.org/docs/dev/cookbook/cosmological_analysis.html#plotting-halos" target="_blank">http://yt-project.org/docs/dev/cookbook/cosmological_analysis.html#plotting-halos</a>),<br>
> but as I said, I am running into problems using the version of rockstar that<br>
> comes with yt on enzo (or any hdf5) datasets. Does anyone know how to use<br>
> this standalone rockstar with an enzo dataset to produce such a halo file<br>
> for later use within yt?<br>
<br>
</div></div>Hm, news to me about the docs suggesting that. My understanding is<br>
that we're mostly going in the opposite way -- avoiding using<br>
Rockstar's parallelism completely (to avoid the issues you note) and<br>
instead feeding in FOF groups found by yt. This didn't get finished<br>
in time for the 3.0 release, and I wouldn't recommend trying it right<br>
now.<br>
<br>
For multiple masses, I think your first inclination (using particle<br>
filters) is the best. Seems like it should just be fixable by<br>
allowing the Halo Catalog to accept a particle filter.<br>
<div><div><br>
><br>
> Sorry for the long email, but I've tried lots of things. Thank you very<br>
> much for the help so far!<br>
><br>
> Cameron<br>
><br>
><br>
>><br>
>> Britton<br>
>><br>
>><br>
>> On Tue, Aug 12, 2014 at 11:52 AM, David Sullivan <<a href="mailto:D.Sullivan@sussex.ac.uk" target="_blank">D.Sullivan@sussex.ac.uk</a>><br>
>> wrote:<br>
>>><br>
>>> Hi all,<br>
>>><br>
>>> I also get the same error as Cameron when trying to run rockstar with<br>
>>> mpi:<br>
>>><br>
>>> <a href="http://paste.yt-project.org/show/5019/" target="_blank">http://paste.yt-project.org/show/5019/</a><br>
>>><br>
>>> I’m using a ramses dataset (not sure if this is supported, but couldn’t<br>
>>> see any clear evidence that it isn’t) and the following script:<br>
>>><br>
>>> mpirun -np 4 --mca btl ^openib python run_rockstar.py --parallel<br>
>>> <a href="http://paste.yt-project.org/show/5020/" target="_blank">http://paste.yt-project.org/show/5020/</a><br>
>>><br>
>>> I followed the same install instructions as Cameron, but I’m using a<br>
>>> Linux cluster running CentOS. Appreciate any help!<br>
>>><br>
>>> Cheers,<br>
>>> David<br>
>>><br>
>>> On 12 Aug 2014, at 10:04, <a href="mailto:yt-users-request@lists.spacepope.org" target="_blank">yt-users-request@lists.spacepope.org</a> wrote:<br>
>>><br>
>>> Message: 3<br>
>>> Date: Tue, 12 Aug 2014 02:04:10 -0700<br>
>>> From: Sam Skillman <<a href="mailto:samskillman@gmail.com" target="_blank">samskillman@gmail.com</a>><br>
>>> To: Discussion of the yt analysis package<br>
>>> <<a href="mailto:yt-users@lists.spacepope.org" target="_blank">yt-users@lists.spacepope.org</a>><br>
>>> Subject: Re: [yt-users] Problems using Rockstar in yt<br>
>>> Message-ID:<br>
>>> <CAB4=<a href="mailto:1s1oHK0uPFo88rkSaEa-W6EASMLoyOiEFeniV-xSm6SPaQ@mail.gmail.com" target="_blank">1s1oHK0uPFo88rkSaEa-W6EASMLoyOiEFeniV-xSm6SPaQ@mail.gmail.com</a>><br>
>>> Content-Type: text/plain; charset="utf-8"<br>
>>><br>
>>><br>
>>> Hi Cameron,<br>
>>><br>
>>> Are you running in MPI? You need to run with at least 3 MPI tasks for<br>
>>> Rockstar to work properly.<br>
>>><br>
>>> Sam<br>
>>><br>
>>><br>
>>> On Mon, Aug 11, 2014 at 6:41 PM, Cameron Hummels <<a href="mailto:chummels@gmail.com" target="_blank">chummels@gmail.com</a>><br>
>>> wrote:<br>
>>><br>
>>> Hello,<br>
>>><br>
>>> I'm trying to use Rockstar with the new HaloCatalog functionality in yt<br>
>>> 3.0, but I'm encountering some problems. For reference, this is an OS X<br>
>>> box with the "yt" branch. I've tested it on two OS X systems to the same<br>
>>> result.<br>
>>><br>
>>> I modified my install_script.py to have it install Rockstar as described<br>
>>> here:<br>
>>><br>
>>><br>
>>><br>
>>> <a href="http://yt-project.org/docs/dev/analyzing/analysis_modules/halo_finders.html#rockstar-installation" target="_blank">http://yt-project.org/docs/dev/analyzing/analysis_modules/halo_finders.html#rockstar-installation</a><br>
>>><br>
>>> Then, after rerunning my install script for the `yt` branch (ie yt-3.1<br>
>>> dev) I tried to run the script provided here in order to generate a halo<br>
>>> catalog using HOP:<br>
>>><br>
>>><br>
>>><br>
>>> <a href="http://yt-project.org/docs/dev/analyzing/analysis_modules/halo_transition.html#getting-halo-information" target="_blank">http://yt-project.org/docs/dev/analyzing/analysis_modules/halo_transition.html#getting-halo-information</a><br>
>>><br>
>>> It can create the catalog fine with finder_method='hop', but when I<br>
>>> change<br>
>>> it to finder_method='rockstar', I get failures. This is the traceback:<br>
>>><br>
>>> <a href="http://paste.yt-project.org/show/TE2DW8gc50urchv4JnUk" target="_blank">http://paste.yt-project.org/show/TE2DW8gc50urchv4JnUk</a><br>
>>><br>
>>> In the traceback, it claims that it cannot access librockstar.so, but<br>
>>> this<br>
>>> library exists in the yt-x86_64/lib directory, which is explicitly in<br>
>>> from<br>
>>> my $LD_LIBRARY_PATH. I can seemingly get it to work, if I add that path<br>
>>> to<br>
>>> my $DYLD_LIBRARY_PATH environment variable, but then a bunch of other<br>
>>> things break (like vi). Not sure what to do here.<br>
>>><br>
>>> But assuming there is no vi problem here, even then, when I run the<br>
>>> script<br>
>>> above, I get failures. Here is the traceback:<br>
>>><br>
>>> <a href="http://paste.yt-project.org/show/rCOT1eVT1szR1T6DEUDb" target="_blank">http://paste.yt-project.org/show/rCOT1eVT1szR1T6DEUDb</a><br>
>>><br>
>>> I'm not sure what to do here. I've also tried running rockstar external<br>
>>> to yt on an enzo dataset, but it doesn't appear that the version we<br>
>>> install<br>
>>> as part of yt allows it to work on HDF5 datasets.<br>
>>><br>
>>> Has anyone run into any of these problems before? I'm at a bit of a loss<br>
>>> as to how to proceed on any of these fronts.<br>
>>><br>
>>> Cameron<br>
>>><br>
>>><br>
>>> --<br>
>>> Cameron Hummels<br>
>>> Postdoctoral Researcher<br>
>>> Steward Observatory<br>
>>> University of Arizona<br>
>>> <a href="http://chummels.org" target="_blank">http://chummels.org</a><br>
>>><br>
>>> _______________________________________________<br>
>>> yt-users mailing list<br>
>>> <a href="mailto:yt-users@lists.spacepope.org" target="_blank">yt-users@lists.spacepope.org</a><br>
>>> <a href="http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org" target="_blank">http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org</a><br>
>>><br>
>>><br>
>>><br>
>>> _______________________________________________<br>
>>> yt-users mailing list<br>
>>> <a href="mailto:yt-users@lists.spacepope.org" target="_blank">yt-users@lists.spacepope.org</a><br>
>>> <a href="http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org" target="_blank">http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org</a><br>
>>><br>
>><br>
>><br>
>> _______________________________________________<br>
>> yt-users mailing list<br>
>> <a href="mailto:yt-users@lists.spacepope.org" target="_blank">yt-users@lists.spacepope.org</a><br>
>> <a href="http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org" target="_blank">http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org</a><br>
>><br>
><br>
><br>
><br>
> --<br>
> Cameron Hummels<br>
> Postdoctoral Researcher<br>
> Steward Observatory<br>
> University of Arizona<br>
> <a href="http://chummels.org" target="_blank">http://chummels.org</a><br>
><br>
> _______________________________________________<br>
> yt-users mailing list<br>
> <a href="mailto:yt-users@lists.spacepope.org" target="_blank">yt-users@lists.spacepope.org</a><br>
> <a href="http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org" target="_blank">http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org</a><br>
><br>
_______________________________________________<br>
yt-users mailing list<br>
<a href="mailto:yt-users@lists.spacepope.org" target="_blank">yt-users@lists.spacepope.org</a><br>
<a href="http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org" target="_blank">http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org</a><br>
</div></div></blockquote></div><br></div>
</div></div><br>_______________________________________________<br>
yt-users mailing list<br>
<a href="mailto:yt-users@lists.spacepope.org">yt-users@lists.spacepope.org</a><br>
<a href="http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org" target="_blank">http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org</a><br>
<br></blockquote></div><br></div>