[yt-users] YT: Rockstar Halo Finder

[Matthew Turk]

Hi Devin,

I think the best solution moving forward is to use FOF on subsets of
the data, and feed them into Hilary's new routine -- this should keep
memory at a minimum.  If you mandate the tiles are small enough that
they fit in memory, but not so small they don't contain halos, it
should be fine.

-Matt

On Thu, Jun 11, 2015 at 3:33 PM, Devin Silvia <devin.silvia at gmail.com> wrote:
> To follow up on this, Jacob has been working with me on this dataset and
> since Blue Waters doesn't have any nodes with huge quantities of memory
> we've been trying just about everything we can think of, including running
> on a huge number of nodes or trying a very low processor count per node.
> Unfortunately, none of the configurations we've tried has worked.
>
> Thanks for the information Hilary.  Unfortunately, we don't have access to a
> shared memory machine at the moment :(  If anyone else has thoughts for how
> to make this go, we'd love to hear them.
>
> Thanks!
> Devin
>
> On Thu, Jun 11, 2015 at 4:14 PM, Hilary Egan <hilaryye at gmail.com> wrote:
>>
>> Hi Jacob,
>>
>> While I'm not confident in exact numbers in terms of nodes/ppn settings
>> for your run, I think I can point you in the right direction.
>>
>> * When I used rockstar to find halos for a 1024^3 unigrid enzo I ran into
>> similar OOM errors. I eventually got it to work by running on a shared
>> memory machine (PSC Blacklight to be exact). I'm not sure you're ever going
>> to be able to run it without a machine that can share memory between nodes,
>> because of the way rockstar handles keeping the particles in memory
>> separately from yt.
>>
>> * You also don't need anywhere near that number of processes. If I can
>> recall, I think I only used ~16 processes in the end, once I could request
>> enough memory.
>>
>> Hope that helps a bit! Sorry the current implementation of rockstar in yt
>> is a bit of a pain to work with. It's something we've been working on
>> fixing, so we can keep you posted on when the new infrastructure is ready
>> for users.
>>
>> -Hilary
>>
>> On Thu, Jun 11, 2015 at 1:22 PM, Jacob Kneibel <kneibelj at msu.edu> wrote:
>>>
>>> Hello,
>>>
>>> I've been trying to run the Rockstar Halo finder on a 1536^3 unigrid Enzo
>>> simulation and have been running into a few issues. I am using the Blue
>>> waters machine and when I run the finder with multiple readers, the process
>>> is killed by the OOM killer, and when I run it with one reader there is a
>>> Network IO error.
>>>
>>> My yt version is 3.2-dev, changeset= 8ad62b624390
>>>
>>> Here is the python script I am running:
>>> http://paste.yt-project.org/show/5631/
>>>
>>> and here is the job script I'm submitting to blue waters:
>>> http://paste.yt-project.org/show/5632/
>>>
>>> I've tried different node and ppn settings, with no luck.
>>>
>>> Does anyone have an idea on how to get rockstar to run on the blue waters
>>> system with this large simulation? Or is there a recommended number of
>>> readers based on the number of nodes I set?
>>>
>>> Thank you for any help you can give,
>>>
>>> Jacob Kneibel
>>>
>>>
>>> _______________________________________________
>>> yt-users mailing list
>>> yt-users at lists.spacepope.org
>>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>>
>>
>>
>> _______________________________________________
>> yt-users mailing list
>> yt-users at lists.spacepope.org
>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>>
>
>
>
> --
> Devin W. Silvia
> NSF Astronomy and Astrophysics Postdoctoral Fellow
> Department of Physics and Astronomy
> Michigan State University
> www.devinsilvia.com
>
> _______________________________________________
> yt-users mailing list
> yt-users at lists.spacepope.org
> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>