[Yt-dev] nickel instruction for using parallel HOP (not on Kraken)

[Stephen Skory]

Hi all,

Matt asked me to outline how to get parallel HOP (in the hg devel version) working. In easy to follow list format:

0. Build yt like you did already (you're on this list, no?).

1. Get the latest hg, choose the 'yt' branch, which is the default.

2. Install Forthon in the standard python way.
http://hifweb.lbl.gov/Forthon/

3. Go into yt-devel/yt/extensions/kdtree and I think 'make' will work on most machines. If not, let me know. It should make 'fKDpy.so' in that same dir.

4. Go to the top level of yt-devel, and type 'python setup.py install' like usual, but then do 'cp -r yt/extensions/kdtree ../../lib/python2.6/site-packages/yt-version-num-sys-etc/yt/extensions/'. I haven't added the kdtree stuff to setup utils yet, so this step isn't automated.

5. One runs parallel HOP very similarly to old HOP:

h = yt.lagos.HaloFinding.parallelHF(pf, threshold=160.0, safety=1.5, \
dm_only=False,resize=True, fancy_padding=True, rearrange=True)

- safety - is a padding multiplier that adjusts the size of the padding around each subvolume. 1.5 is usually sufficient, and 1.35 probably is. There's no reason to try going lower from 1.5 unless memory is a concern.
- dm_only - I currently have things a little messed up in HaloFinding.py in the dev, so this needs to be set unless you have a ['creation_time'] field.
- resize - Load balancing. This is a recursive bisection that subdivides the domains like a kd tree, but it can do arbitrary numbers of cuts, so you can run with any number of tasks. Generally, I turn it on for datasets smaller than 300 Mpc, and for ones larger, the particles are already pretty evenly balanced and it doesn't help.
- fancy_padding - Each subvolume has six different faces, and six different values for the padding, one for each face. There is really no reason to turn this off.
- rearrange - The fortran kdtree can make an internal copy of the point data which increases the speed of the searches by almost 20%. It however increases the memory by 3 fields, so if memory is a concern, turn this off.

6. The usual halo convenience functions work as well, but the output is a bit different.

h.write_out('dist-chain.out') - Just like the normal one.
h.write_particle_lists("chain") - One hdf5 file per task, that contains the particle data for all the haloes on that task.
h.write_particle_lists_txt("chain") - One text file that lists which tasks own a piece of a halo. Since haloes can exist on more than one task, each halo may have several entries on it's line.

7. There is a rough memory scaling law: it takes 1 MB of RAM per 4,500 particles, with rearrange turned on, regardless of task count. This does not include the yt hierarchy costs, so if you have a dataset like L7, which uses 1GB per task, you will need to take that into account, too.

Good luck!

 _______________________________________________________
sskory at physics.ucsd.edu           o__  Stephen Skory
http://physics.ucsd.edu/~sskory/ _.>/ _Graduate Student
________________________________(_)_\(_)_______________