[yt-svn] commit/yt: 15 new changesets
commits-noreply at bitbucket.org
commits-noreply at bitbucket.org
Fri Jul 18 14:05:37 PDT 2014
15 new commits in yt:
https://bitbucket.org/yt_analysis/yt/commits/4be1ed732c05/
Changeset: 4be1ed732c05
Branch: yt-3.0
User: hegan
Date: 2014-07-17 17:47:45
Summary: adding translation of halo tasks from yt-2.x->yt-3.0 in docs
Affected #: 2 files
diff -r 2bccbc8de4e2b93aed82fff99f0dafd9308e1691 -r 4be1ed732c05c6eae4e9a2e2aac05291e65b1c62 doc/source/analyzing/analysis_modules/halo_catalogs.rst
--- a/doc/source/analyzing/analysis_modules/halo_catalogs.rst
+++ b/doc/source/analyzing/analysis_modules/halo_catalogs.rst
@@ -7,6 +7,8 @@
together into a single framework. This framework is substantially
different from the limited framework included in yt-2.x and is only
backwards compatible in that output from old halo finders may be loaded.
+For a direct translation of various halo analysis tasks using yt-2.x
+to yt-3.0 please see :ref:`halo_translation'.
A catalog of halos can be created from any initial dataset given to halo
catalog through data_pf. These halos can be found using friends-of-friends,
diff -r 2bccbc8de4e2b93aed82fff99f0dafd9308e1691 -r 4be1ed732c05c6eae4e9a2e2aac05291e65b1c62 doc/source/analyzing/analysis_modules/halo_translation.rst
--- /dev/null
+++ b/doc/source/analyzing/analysis_modules/halo_translation.rst
@@ -0,0 +1,7 @@
+
+Getting up to Speed with Halo Analysis in yt-3.0
+================================================
+
+If you're used to halo analysis in yt-2.x, heres a guide to
+how to update your analysis pipeline to take advantage of
+the new halo catalog infrastructure.
https://bitbucket.org/yt_analysis/yt/commits/95be98a1282e/
Changeset: 95be98a1282e
Branch: yt-3.0
User: hegan
Date: 2014-07-17 18:09:55
Summary: Reorganized halo finder file, removed most of the now irrelevant/incorrect info
Affected #: 2 files
diff -r 4be1ed732c05c6eae4e9a2e2aac05291e65b1c62 -r 95be98a1282eab1c60b5acd3967b359fd66126f3 doc/source/analyzing/analysis_modules/halo_finders.rst
--- /dev/null
+++ b/doc/source/analyzing/analysis_modules/halo_finders.rst
@@ -0,0 +1,224 @@
+.. _halo_finding:
+
+Halo Finding
+============
+.. sectionauthor:: Stephen Skory <sskory at physics.ucsd.edu>
+
+There are four methods of finding particle haloes in yt. The
+recommended and default method is called HOP, a method described
+in `Eisenstein and Hut (1998)
+<http://adsabs.harvard.edu/abs/1998ApJ...498..137E>`_. A basic
+friends-of-friends (e.g. `Efstathiou et al. (1985)
+<http://adsabs.harvard.edu/abs/1985ApJS...57..241E>`_) halo
+finder is also implemented. Finally Rockstar (`Behroozi et a.
+(2011) <http://adsabs.harvard.edu/abs/2011arXiv1110.4372B>`_) is
+a 6D-phase space halo finder developed by Peter Behroozi that
+excels in finding subhalos and substrcture, but does not allow
+multiple particle masses.
+
+HOP
+---
+
+The version of HOP used in yt is an upgraded version of the
+`publicly available HOP code
+<http://cmb.as.arizona.edu/~eisenste/hop/hop.html>`_. Support
+for 64-bit floats and integers has been added, as well as
+parallel analysis through spatial decomposition. HOP builds
+groups in this fashion:
+
+ 1. Estimates the local density at each particle using a
+ smoothing kernel.
+ 2. Builds chains of linked particles by 'hopping' from one
+ particle to its densest neighbor. A particle which is
+ its own densest neighbor is the end of the chain.
+ 3. All chains that share the same densest particle are
+ grouped together.
+ 4. Groups are included, linked together, or discarded
+ depending on the user-supplied over density
+ threshold parameter. The default is 160.0.
+
+Please see the `HOP method paper
+<http://adsabs.harvard.edu/abs/1998ApJ...498..137E>`_ for
+full details.
+
+Friends-of-Friends
+------------------
+
+The version of FoF in yt is based on the `publicly available
+FoF code <http://www-hpcc.astro.washington.edu/tools/fof.html>`_
+from the University of Washington. Like HOP, FoF supports
+parallel analysis through spatial decomposition. FoF is much
+simpler than HOP:
+
+ 1. From the total number of particles, and the volume of the
+ region, the average inter-particle spacing is calculated.
+ 2. Pairs of particles closer together than some fraction of
+ the average inter-particle spacing (the default is 0.2)
+ are linked together. Particles can be paired with more
+ than one other particle.
+ 3. The final groups are formed the networks of particles linked
+ together by friends, hence the name.
+
+.. warning:: The FoF halo finder in yt is not thoroughly tested!
+ It is probably fine to use, but you are strongly encouraged
+ to check your results against the data for errors.
+
+Rockstar Halo Finding
+---------------------
+.. sectionauthor:: Matthew Turk <matthewturk at gmail.com>
+.. sectionauthor:: Christopher Erick Moody<cemoody at ucsc.edu>
+.. sectionauthor:: Stephen Skory <s at skory.us>
+
+Rockstar uses an adaptive hierarchical refinement of friends-of-friends
+groups in six phase-space dimensions and one time dimension, which
+allows for robust (grid-independent, shape-independent, and noise-
+resilient) tracking of substructure. The code is prepackaged with yt,
+but also `separately available <http://code.google.com/p/rockstar>`_. The lead
+developer is Peter Behroozi, and the methods are described in `Behroozi
+et al. 2011 <http://rockstar.googlecode.com/files/rockstar_ap101911.pdf>`_.
+
+.. note:: At the moment, Rockstar does not support multiple particle masses,
+ instead using a fixed particle mass. This will not affect most dark matter
+ simulations, but does make it less useful for finding halos from the stellar
+ mass.
+
+To run the Rockstar Halo finding, you must launch python with MPI and
+parallelization enabled. While Rockstar itself does not require MPI to run,
+the MPI libraries allow yt to distribute particle information across multiple
+nodes.
+
+.. warning:: At the moment, running Rockstar inside of yt on multiple compute nodes
+ connected by an Infiniband network can be problematic. Therefore, for now
+ we recommend forcing the use of the non-Infiniband network (e.g. Ethernet)
+ using this flag: ``--mca btl ^openib``.
+ For example, here is how Rockstar might be called using 24 cores:
+ ``mpirun -n 24 --mca btl ^openib python ./run_rockstar.py --parallel``.
+
+Designing the python script itself is straightforward:
+
+.. code-block:: python
+
+ from yt.mods import *
+ from yt.analysis_modules.halo_finding.rockstar.api import RockstarHaloFinder
+
+ #find all of our simulation files
+ files = glob.glob("Enzo_64/DD*/\*index")
+ #hopefully the file name order is chronological
+ files.sort()
+ ts = DatasetSeries.from_filenames(files[:])
+ rh = RockstarHaloFinder(ts)
+ rh.run()
+
+The script above configures the Halo finder, launches a server process which
+disseminates run information and coordinates writer-reader processes.
+Afterwards, it launches reader and writer tasks, filling the available MPI
+slots, which alternately read particle information and analyze for halo
+content.
+
+The RockstarHaloFinder class has these options:
+ * ``dm_type``, the index of the dark matter particle. Default is 1.
+ * ``outbase``, This is where the out*list files that Rockstar makes should be
+ placed. Default is 'rockstar_halos'.
+ * ``num_readers``, the number of reader tasks (which are idle most of the
+ time.) Default is 1.
+ * ``num_writers``, the number of writer tasks (which are fed particles and
+ do most of the analysis). Default is MPI_TASKS-num_readers-1.
+ If left undefined, the above options are automatically
+ configured from the number of available MPI tasks.
+ * ``force_res``, the resolution that Rockstar uses for various calculations
+ and smoothing lengths. This is in units of Mpc/h.
+ If no value is provided, this parameter is automatically set to
+ the width of the smallest grid element in the simulation from the
+ last data snapshot (i.e. the one where time has evolved the
+ longest) in the time series:
+ ``pf_last.index.get_smallest_dx() * pf_last['mpch']``.
+ * ``total_particles``, if supplied, this is a pre-calculated
+ total number of dark matter
+ particles present in the simulation. For example, this is useful
+ when analyzing a series of snapshots where the number of dark
+ matter particles should not change and this will save some disk
+ access time. If left unspecified, it will
+ be calculated automatically. Default: ``None``.
+ * ``dm_only``, if set to ``True``, it will be assumed that there are
+ only dark matter particles present in the simulation.
+ This option does not modify the halos found by Rockstar, however
+ this option can save disk access time if there are no star particles
+ (or other non-dark matter particles) in the simulation. Default: ``False``.
+
+
+Rockstar dumps halo information in a series of text (halo*list and
+out*list) and binary (halo*bin) files inside the ``outbase`` directory.
+We use the halo list classes to recover the information.
+
+Inside the ``outbase`` directory there is a text file named ``pfs.txt``
+that records the connection between pf names and the Rockstar file names.
+
+
+Parallel HOP and FOF
+--------------------
+
+Both the HOP and FoF halo finders can run in parallel using simple
+spatial decomposition. In order to run them in parallel it is helpful
+to understand how it works. Below in the first plot (i) is a simplified
+depiction of three haloes labeled 1,2 and 3:
+
+.. image:: _images/ParallelHaloFinder.png
+ :width: 500
+
+Halo 3 is twice reflected around the periodic boundary conditions.
+
+In (ii), the volume has been sub-divided into four equal subregions,
+A,B,C and D, shown with dotted lines. Notice that halo 2 is now in
+two different subregions, C and D, and that halo 3 is now in three,
+A, B and D. If the halo finder is run on these four separate subregions,
+halo 1 is be identified as a single halo, but haloes 2 and 3 are split
+up into multiple haloes, which is incorrect. The solution is to give
+each subregion padding to oversample into neighboring regions.
+
+In (iii), subregion C has oversampled into the other three regions,
+with the periodic boundary conditions taken into account, shown by
+dot-dashed lines. The other subregions oversample in a similar way.
+
+The halo finder is then run on each padded subregion independently
+and simultaneously. By oversampling like this, haloes 2 and 3 will
+both be enclosed fully in at least one subregion and identified
+completely.
+
+Haloes identified with centers of mass inside the padded part of a
+subregion are thrown out, eliminating the problem of halo duplication.
+The centers for the three haloes are shown with stars. Halo 1 will
+belong to subregion A, 2 to C and 3 to B.
+
+To run with parallel halo finding, you must supply a value for
+padding in the finder_kwargs argument. The ``padding`` parameter
+is in simulation units and defaults to 0.02. This parameter is how
+much padding is added to each of the six sides of a subregion.
+This value should be 2x-3x larger than the largest expected halo
+in the simulation. It is unlikely, of course, that the largest
+object in the simulation will be on a subregion boundary, but there
+is no way of knowing before the halo finder is run.
+
+.. code-block:: python
+
+ from yt.mods import *
+ from yt.analysis_modules.halo_finding.api import *
+ pf = load("data0001")
+ halo_list = HaloFinder(pf,padding=0.02)
+ # --or--
+ halo_list = FOFHaloFinder(pf,padding=0.02)
+
+In general, a little bit of padding goes a long way, and too much
+just slows down the analysis and doesn't improve the answer (but
+doesn't change it). It may be worth your time to run the parallel
+halo finder at a few paddings to find the right amount, especially
+if you're analyzing many similar datasets.
+
+Rockstar Installation
+=====================
+
+The Rockstar is slightly patched and modified to run as a library inside of
+yt. By default it will be built with yt using the ``install_script.sh``.
+If it wasn't installed, please make sure that the installation setting
+``INST_ROCKSTAR=1`` is defined in the ``install_script.sh`` and re-run
+the installation script.
+
diff -r 4be1ed732c05c6eae4e9a2e2aac05291e65b1c62 -r 95be98a1282eab1c60b5acd3967b359fd66126f3 doc/source/analyzing/analysis_modules/running_halofinder.rst
--- a/doc/source/analyzing/analysis_modules/running_halofinder.rst
+++ /dev/null
@@ -1,612 +0,0 @@
-.. _halo_finding:
-
-Halo Finding
-============
-.. sectionauthor:: Stephen Skory <sskory at physics.ucsd.edu>
-
-There are four methods of finding particle haloes in yt. The recommended and default method is called HOP, a
-method described in `Eisenstein and Hut (1998) <http://adsabs.harvard.edu/abs/1998ApJ...498..137E>`_.
-A basic friends-of-friends (e.g. `Efstathiou et al. (1985) <http://adsabs.harvard.edu/abs/1985ApJS...57..241E>`_)
-halo finder is also implemented.
-Parallel HOP (`Skory et al. (2010) <http://adsabs.harvard.edu/abs/2010ApJS..191...43S>`_)
-is a true parallelization of the HOP method can analyze massive datasets on
-hundreds of processors.
-Finally Rockstar (`Behroozi et a. (2011) <http://adsabs.harvard.edu/abs/2011arXiv1110.4372B>`_)
-is a 6D-phase space halo finder developed by Peter Behroozi
-that excels in finding subhalos and substrcture,
-but does not allow multiple particle masses.
-
-HOP
----
-
-The version of HOP used in yt is an upgraded version of the `publicly available HOP code
-<http://cmb.as.arizona.edu/~eisenste/hop/hop.html>`_. Support for 64-bit floats and integers has been
-added, as well as parallel analysis through spatial decomposition. HOP builds groups in this fashion:
-
- 1. Estimates the local density at each particle using a smoothing kernel.
- 2. Builds chains of linked particles by 'hopping' from one particle to its densest neighbor.
- A particle which is its own densest neighbor is the end of the chain.
- 3. All chains that share the same densest particle are grouped together.
- 4. Groups are included, linked together, or discarded depending on the user-supplied over density
- threshold parameter. The default is 160.0.
-
-Please see the `HOP method paper <http://adsabs.harvard.edu/abs/1998ApJ...498..137E>`_
-for full details.
-
-Friends-of-Friends
-------------------
-
-The version of FoF in yt is based on the `publicly available FoF code <http://www-hpcc.astro.washington.edu/tools/fof.html>`_ from the University of Washington. Like HOP,
-FoF supports parallel analysis through spatial decomposition. FoF is much simpler than HOP:
-
- 1. From the total number of particles, and the volume of the region, the average
- inter-particle spacing is calculated.
- 2. Pairs of particles closer together than some fraction of the average inter-particle spacing
- (the default is 0.2) are linked together. Particles can be paired with more than one other particle.
- 3. The final groups are formed the networks of particles linked together by friends, hence the name.
-
-.. warning:: The FoF halo finder in yt is not thoroughly tested! It is probably fine to use, but you
- are strongly encouraged to check your results against the data for errors.
-
-Running HaloFinder
-------------------
-
-Running HOP on a dataset is straightforward
-
-.. code-block:: python
-
- from yt.mods import *
- from yt.analysis_modules.halo_finding.api import *
- pf = load("data0001")
- halo_list = HaloFinder(pf)
-
-Running FoF is similar:
-
-.. code-block:: python
-
- from yt.mods import *
- from yt.analysis_modules.halo_finding.api import *
- pf = load("data0001")
- halo_list = FOFHaloFinder(pf)
-
-Halo Data Access
-----------------
-
-``halo_list`` is a list of ``Halo`` class objects ordered by decreasing halo mass. A ``Halo`` object
-has convenient ways to access halo data. This loop will print the location of the center of mass
-for each halo found
-
-.. code-block:: python
-
- for halo in halo_list:
- print halo.center_of_mass()
-
-All the methods are:
-
- * .center_of_mass() - the center of mass for the halo.
- * .maximum_density() - the maximum density in "HOP" units.
- * .maximum_density_location() - the location of the maximum density particle in the HOP halo.
- * .total_mass() - the mass of the halo in Msol (not Msol/h).
- * .bulk_velocity() - the velocity of the center of mass of the halo in simulation units.
- * .maximum_radius() - the distance from the center of mass to the most distant particle in the halo
- in simulation units.
- * .get_size() - the number of particles in the halo.
- * .get_sphere() - returns an an EnzoSphere object using the center of mass and maximum radius.
- * .virial_mass(virial_overdensity=float, bins=int) - Finds the virial
- mass for a halo using just the particles. This is inferior to the full
- Halo Profiler extension (:ref:`halo_profiling`), but useful nonetheless in some cases.
- Returns the mass in Msol, or -1 if the halo is not virialized.
- Defaults: ``virial_overdensity=200.0`` and ``bins=300``.
- * .virial_radius(virial_overdensity=float, bins=int) - Fins the virial
- radius of the halo using just the particles. Returns the radius in code
- units, or -1 if the halo is not virialized.
- Defaults: ``virial_overdensity=200.0`` and ``bins=300``.
-
-.. note:: For FOF the maximum density value is meaningless and is set to -1 by default. For FOF
- the maximum density location will be identical to the center of mass location.
-
-For each halo the data for the particles in the halo can be accessed like this
-
-.. code-block:: python
-
- for halo in halo_list:
- print halo["particle_index"]
- print halo["particle_position_x"] # in simulation units
-
-Halo List Data Access
----------------------
-
-These are methods that operate on the list of halo objects, rather than on the
-haloes themselves (e.g. ``halo_list.write_out()`` instead of ``halo_list[0].center_of_mass()``).
-For example, The command
-
-.. code-block:: python
-
- halo_list.write_out("HaloAnalysis.out")
-
-will output the haloes to a text file named ``HaloAnalysis.out``.
-
- * .write_out(``name``) - Writes out the center of mass, maximum density point,
- number of particles, mass, index, bulk velocity and maximum radius for all the haloes
- to a text file ``name``.
- * .write_particle_lists(``name``) - Writes the data for the particles in haloes
- (position, velocity, mass and particle index) to a HDF5 file with prefix ``name``, or one HDF5
- file per CPU when running in parallel.
- * .write_particle_lists_txt(``name``) - Writes out one text file with prefix ``name`` that gives the
- location of the particle data for haloes in the HDF5 files. This is only
- necessary when running in parallel.
- * .dump(``basename``) - Calls all of the above three functions using
- ``basename`` in each. This function is meant to be used in combination with
- loading halos off disk (:ref:`load_haloes`).
- * .nearest_neighbors_3D(haloID, num_neighbors=int, search_radius=float) -
- For a given halo ``haloID``, this finds the ``num_neighbors`` nearest (periodic)
- neighbors that are within ``search_radius`` distance from it.
- It returns a list of the neighbors distances and ID with format
- [distance,haloID]. Defaults: ``num_neighbors=7``, ``search_radius=0.2``.
- * .nearest_neighbors_2D(haloID, num_neighbors=int, search_radius=float, proj_dim={0,1,2}) -
- Similarly to the 3D search, this finds the nearest (periodic) neighbors to a halo, but
- with the positions of the haloes projected onto a 2D plane. The normal to the
- projection plane is set with ``proj_dim``, which is set to {0,1,2} for the
- {x,y,z}-axis. Defaults: ``num_neighbors=7``, ``search_radius=0.2`` and ``proj_dim=0``.
- Returns a list of neighbors in the same format as the 3D case, but the distances
- are the 2D projected distance.
-
-.. _load_haloes:
-
-Loading Haloes Off Disk
------------------------
-
-It is possible to load haloes off disk and use them as if they had just been
-located by the halo finder. This has at least two advantages. Quite obviously
-this means that if the halos are properly saved (e.g. ``haloes.dump()``, see
-above and below), halo finding does not need to be run again, saving time.
-Another benefit is loaded haloes only use as much memory as needed because the
-particle data for the haloes is loaded off disk on demand. If only a few haloes
-are being examined, a dataset that required parallel analysis for halo finding
-can be analyzed in serial, interactively.
-
-The first step is to save the haloes in a consistent manner, which is made
-simple with the ``.dump()`` function:
-
-.. code-block:: python
-
- from yt.mods import *
- from yt.analysis_modules.halo_finding.api import *
- pf = load("data0001")
- haloes = HaloFinder(pf)
- haloes.dump("basename")
-
-It is easy to load the halos using the ``LoadHaloes`` class:
-
-.. code-block:: python
-
- from yt.mods import *
- from yt.analysis_modules.halo_finding.api import *
- pf = load("data0001")
- haloes = LoadHaloes(pf, "basename")
-
-Everything that can be done with ``haloes`` in the first example should be
-possible with ``haloes`` in the second.
-
-General Parallel Halo Analysis
-------------------------------
-
-Both the HOP and FoF halo finders can run in parallel using simple spatial decomposition.
-In order to run them
-in parallel it is helpful to understand how it works.
-
-Below in the first plot (i) is a simplified depiction of three haloes labeled 1,2 and 3:
-
-.. image:: _images/ParallelHaloFinder.png
- :width: 500
-
-Halo 3 is twice reflected around the periodic boundary conditions.
-
-In (ii), the volume has been
-sub-divided into four equal subregions, A,B,C and D, shown with dotted lines. Notice that halo 2
-is now in two different subregions,
-C and D, and that halo 3 is now in three, A, B and D. If the halo finder is run on these four separate subregions,
-halo 1 is be identified as a single halo, but haloes 2 and 3 are split up into multiple haloes, which is incorrect.
-The solution is to give each subregion padding to oversample into neighboring regions.
-
-In (iii), subregion C has oversampled into the other three regions, with the periodic boundary conditions taken
-into account, shown by dot-dashed lines. The other subregions oversample in a similar way.
-
-The halo finder is then run on each padded subregion independently and simultaneously.
-By oversampling like this, haloes 2 and 3 will both be enclosed fully in at least one subregion and
-identified completely.
-
-Haloes identified with centers of mass inside the padded part of a subregion are thrown out, eliminating
-the problem of halo duplication. The centers for the three haloes are shown with stars. Halo 1 will
-belong to subregion A, 2 to C and 3 to B.
-
-Parallel HaloFinder padding
-^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-To run with parallel halo finding, there is a slight modification to the script
-
-.. code-block:: python
-
- from yt.mods import *
- from yt.analysis_modules.halo_finding.api import *
- pf = load("data0001")
- halo_list = HaloFinder(pf,padding=0.02)
- # --or--
- halo_list = FOFHaloFinder(pf,padding=0.02)
-
-The ``padding`` parameter is in simulation units and defaults to 0.02. This parameter is how much padding
-is added to each of the six sides of a subregion. This value should be 2x-3x larger than the largest
-expected halo in the simulation. It is unlikely, of course, that the largest object in the simulation
-will be on a subregion boundary, but there is no way of knowing before the halo finder is run.
-
-In general, a little bit of padding goes a long way, and too much just slows down the analysis and doesn't
-improve the answer (but doesn't change it).
-It may be worth your time to run the parallel halo finder at a few paddings to
-find the right amount, especially if you're analyzing many similar datasets.
-
-Parallel HOP
-------------
-
-**Parallel HOP** (not to be confused with HOP running in parallel as described
-above) is a wholly-new halo finder based on the HOP method.
-For extensive details and benchmarks of Parallel HOP, please see the
-pre-print version of the `method paper <http://adsabs.harvard.edu/abs/2010ApJS..191...43S>`_ at
-arXiv.org.
-While the method
-of parallelization described above can be quite effective, it has its limits.
-In particular
-for highly unbalanced datasets, where most of the particles are in a single
-part of the simulation's volume, it can become impossible to subdivide the
-volume sufficiently to fit a subvolume into a single node's memory.
-
-Parallel HOP is designed to be parallel at all levels of operation. There is
-a minimal amount of copied data across tasks. Unlike the parallel method above,
-whole haloes do not need to exist entirely in a single subvolume. In fact, a
-halo may have particles in several subvolumes simultaneously without a problem.
-
-Parallel HOP is appropriate for very large datasets where normal HOP, or
-the parallel method described above, won't work. For smaller datasets, it is
-actually faster to use the simpler methods above because the mechanisms employed for
-full parallelism are somewhat expensive.
-Whether to use Parallel HOP or not depends on the number of particles and
-the size of the largest object in the simulation.
-Because the padding of the other parallel method described above depends on
-the relative size to the box of the largest object, for smaller cosmologies
-that method may not work.
-If the largest object is quite large, the minimum padding will be a
-significant fraction of the full volume, and therefore the minimum number of
-particles per task can stay quite high.
-Below and including 256^3 particles, the other parallel methods are likely
-faster.
-However, above this and for smaller cosmologies (100 Mpc/h and smaller),
-Parallel HOP will offer better performance.
-
-The haloes identified by Parallel HOP are slightly different than normal HOP
-when run on the same dataset with the same over-density threshold.
-For a given threshold value, a few haloes have slightly different numbers of particles.
-Overall, it is not a big difference. In fact, changing the threshold value by
-a percent gives a far greater difference than the differences between HOP and
-Parallel HOP.
-
-HOP and Parallel HOP both use `KD Trees <http://en.wikipedia.org/wiki/Kd_tree>`_
-for nearest-neighbor searches.
-Parallel HOP uses the Fortran version of
-`KDTREE 2 <http://arxiv.org/abs/physics/0408067>`_ written by Matthew B. Kennel.
-The KD Tree in normal HOP calculates the distances
-between particles incorrectly by approximately one part in a million.
-KDTREE 2 is far more accurate (up to machine error),
-and this slight difference is sufficient to make perfect agreement between
-normal and Parallel HOP impossible.
-Therefore Parallel HOP is not a direct substitution for
-normal HOP, but is very similar.
-
-Running Parallel HOP
-^^^^^^^^^^^^^^^^^^^^
-
-Note: This is probably broken now that the Fortran kdtree has been removed.
-
-In the simplest form, Parallel HOP is run very similarly to the other halo finders.
-In the example below, Parallel HOP will be run on a dataset with all the default
-values. Parallel HOP can be run in serial, but as mentioned above, it is
-slower than normal HOP.
-
-.. code-block:: python
-
- from yt.mods import *
- from yt.analysis_modules.halo_finding.api import *
- pf = load("data0001")
- halo_list = parallelHF(pf)
-
-Parallel HOP has these user-set options:
-
- * ``threshold``, positive float: This is the same as the option for normal HOP. Default=160.0.
- * ``dm_only``, True/False: Whether or not to include particles other than dark
- matter when building haloes. Default=True.
- * ``resize``, True/False: Parallel HOP can load-balance the particles, such that
- each subvolume has the same number of particles.
- In general, this option is a good idea for simulations' volumes
- smaller than about 300 Mpc/h, and absolutely required for those under
- 100 Mpc/h. For larger volumes the particles are distributed evenly enough
- that this option is unnecessary. Default=True.
- * ``sample``, positive float: In order to load-balance, a random subset of the
- particle positions are read off disk, and the load-balancing routine is
- applied to them. This parameter controls what fraction of the full dataset
- population is used. Larger values result in more accurate load-balancing,
- and smaller values are faster. The value cannot be too large as the data
- for the subset of particles is communicated to one task for
- load-balancing (meaning a value
- of 1.0 will not work on very large datasets).
- Tests show that values as low as 0.0003 keep the min/max variation between
- tasks below 10%. Default = 0.03.
- * ``rearrange``, True/False: The KD Tree used by Parallel HOP can make an
- internal copy of the particle data which increases the speed of nearest
- neighbor searches by approximately 20%. The only reason to turn this option
- off is if memory is a concern. Default=True.
- * ``safety``, positive float: Unlike the simpler parallel method, Parallel
- HOP calculates the padding automatically. The padding is a
- function of the inter-particle spacing inside each subvolume. This parameter
- is multiplied to the padding distance to increase the padding volume to account for
- density variations on the boundaries of the subvolumes. Increasing this
- parameter beyond a certain point will have no effect other than consuming
- more memory and slowing down runtimes.
- Reducing it will speed up the calculation and use less memory, but
- going too far will result in degraded halo finding.
- Default=1.5, but values as low as 1.0 will probably work for many datasets.
- * ``fancy_padding``, True/False: When this is set to True, the amount of padding
- is calculated independently for each of the six faces of each subvolume. When this is
- False, the padding is the same on all six faces. There is generally no
- good reason to set this to False. Default=True.
- * ``premerge``, True/False: This option will pre-merge only the most dense
- haloes in each subvolume, before haloes are merged on the global level. In
- some cases this can speed up the runtime by a factor of two and reduce peak memory
- greatly. At worst it slows down the runtime by a small amount. It has the
- side-effect of changing the haloes slightly as a function of task count. Put in
- another way, two otherwise identical runs of Parallel HOP on a dataset will end
- up with very slightly different haloes when run with two different task counts
- with this option turned on. Not all haloes are changed between runs. This is
- due to the way merging happens in HOP - pre-merging destroys the global
- determinacy of halo merging. Default=True.
- * ``tree``, string: There are two kD-trees that may be used as part of the
- halo-finding process. The Fortran ("F") one is (presently) faster, but requires
- more memory. One based on `scipy.spatial
- <http://docs.scipy.org/doc/scipy/reference/spatial.html>`_ utilizes
- Cython ("C") and is (presently) slower, but is more memory efficient.
- Default = "F".
-
-All the same halo data can be accessed from Parallel HOP haloes as with the other halo finders.
-However, when running in parallel, there are some
-important differences in the output of a couple of these functions.
-
- * .write_particle_lists(``name``) - Because haloes may exist in more than
- one subvolume, particle data for a halo may be saved in more than one HDF5 file.
- * .write_particle_lists_txt(``name``) - If the particles for a halo is saved
- in more than one HDF5 file, there will be more than one HDF5 file listed for
- each halo in the text file.
-
-In this example script below, Parallel HOP is run on a dataset and the results
-saved to files. The summary of the haloes to ``ParallelHopAnalysis.out``, the
-particles to files named ``parts????.h5`` and the list of haloes in HDF5 files
-to ``parts.txt``.
-
-.. code-block:: python
-
- from yt.mods import *
- from yt.analysis_modules.halo_finding.api import *
- pf = load("data0001")
- halo_list = parallelHF(pf, threshold=80.0, dm_only=True, resize=False,
- rearrange=True, safety=1.5, premerge=True)
- halo_list.write_out("ParallelHopAnalysis.out")
- halo_list.write_particle_list("parts")
- halo_list.write_particle_lists_txt("parts")
-
-Halo Finding In A Subvolume
----------------------------
-
-It is possible to run any of the halo finders over a subvolume.
-This may be advantageous when only one object or region of a simulation
-is being analyzed.
-The subvolume must be a ``region`` and cannot be a
-non-rectilinear shape.
-The halo finding can be performed in parallel on a subvolume, but it may
-not be necessary depending on the size of the subvolume.
-Below is a simple example for HOP; the other halo finders use the same
-``subvolume`` keyword identically.
-
-.. code-block:: python
-
- from yt.mods import *
- from yt.analysis_modules.halo_finding.api import *
- pf = load('data0458')
- # Note that the first term below, [0.5]*3, defines the center of
- # the region and is not used. It can be any value.
- sv = pf.region([0.5]*3, [0.21, .21, .72], [.28, .28, .79])
- halos = HaloFinder(pf, subvolume = sv)
- halos.write_out("sv.out")
-
-
-Rockstar Halo Finding
-=====================
-.. sectionauthor:: Matthew Turk <matthewturk at gmail.com>
-.. sectionauthor:: Christopher Erick Moody<cemoody at ucsc.edu>
-.. sectionauthor:: Stephen Skory <s at skory.us>
-
-Rockstar uses an adaptive hierarchical refinement of friends-of-friends
-groups in six phase-space dimensions and one time dimension, which
-allows for robust (grid-independent, shape-independent, and noise-
-resilient) tracking of substructure. The code is prepackaged with yt,
-but also `separately available <http://code.google.com/p/rockstar>`_. The lead
-developer is Peter Behroozi, and the methods are described in `Behroozi
-et al. 2011 <http://rockstar.googlecode.com/files/rockstar_ap101911.pdf>`_.
-
-.. note:: At the moment, Rockstar does not support multiple particle masses,
- instead using a fixed particle mass. This will not affect most dark matter
- simulations, but does make it less useful for finding halos from the stellar
- mass. Also note that halo finding in a subvolume is not supported by
- Rockstar.
-
-To run the Rockstar Halo finding, you must launch python with MPI and
-parallelization enabled. While Rockstar itself does not require MPI to run,
-the MPI libraries allow yt to distribute particle information across multiple
-nodes.
-
-.. warning:: At the moment, running Rockstar inside of yt on multiple compute nodes
- connected by an Infiniband network can be problematic. Therefore, for now
- we recommend forcing the use of the non-Infiniband network (e.g. Ethernet)
- using this flag: ``--mca btl ^openib``.
- For example, here is how Rockstar might be called using 24 cores:
- ``mpirun -n 24 --mca btl ^openib python ./run_rockstar.py --parallel``.
-
-Designing the python script itself is straightforward:
-
-.. code-block:: python
-
- from yt.mods import *
- from yt.analysis_modules.halo_finding.rockstar.api import RockstarHaloFinder
-
- #find all of our simulation files
- files = glob.glob("Enzo_64/DD*/\*index")
- #hopefully the file name order is chronological
- files.sort()
- ts = DatasetSeries.from_filenames(files[:])
- rh = RockstarHaloFinder(ts)
- rh.run()
-
-The script above configures the Halo finder, launches a server process which
-disseminates run information and coordinates writer-reader processes.
-Afterwards, it launches reader and writer tasks, filling the available MPI
-slots, which alternately read particle information and analyze for halo
-content.
-
-The RockstarHaloFinder class has these options:
- * ``dm_type``, the index of the dark matter particle. Default is 1.
- * ``outbase``, This is where the out*list files that Rockstar makes should be
- placed. Default is 'rockstar_halos'.
- * ``num_readers``, the number of reader tasks (which are idle most of the
- time.) Default is 1.
- * ``num_writers``, the number of writer tasks (which are fed particles and
- do most of the analysis). Default is MPI_TASKS-num_readers-1.
- If left undefined, the above options are automatically
- configured from the number of available MPI tasks.
- * ``force_res``, the resolution that Rockstar uses for various calculations
- and smoothing lengths. This is in units of Mpc/h.
- If no value is provided, this parameter is automatically set to
- the width of the smallest grid element in the simulation from the
- last data snapshot (i.e. the one where time has evolved the
- longest) in the time series:
- ``pf_last.index.get_smallest_dx() * pf_last['mpch']``.
- * ``total_particles``, if supplied, this is a pre-calculated
- total number of dark matter
- particles present in the simulation. For example, this is useful
- when analyzing a series of snapshots where the number of dark
- matter particles should not change and this will save some disk
- access time. If left unspecified, it will
- be calculated automatically. Default: ``None``.
- * ``dm_only``, if set to ``True``, it will be assumed that there are
- only dark matter particles present in the simulation.
- This option does not modify the halos found by Rockstar, however
- this option can save disk access time if there are no star particles
- (or other non-dark matter particles) in the simulation. Default: ``False``.
-
-
-Output Analysis
----------------
-
-Rockstar dumps halo information in a series of text (halo*list and
-out*list) and binary (halo*bin) files inside the ``outbase`` directory.
-We use the halo list classes to recover the information.
-
-Inside the ``outbase`` directory there is a text file named ``pfs.txt``
-that records the connection between pf names and the Rockstar file names.
-
-The halo list can be automatically generated from the RockstarHaloFinder
-object by calling ``RockstarHaloFinder.halo_list()``. Alternatively, the halo
-lists can be built from the RockstarHaloList class directly
-``LoadRockstarHalos(pf,'outbase/out_0.list')``.
-
-.. code-block:: python
-
- rh = RockstarHaloFinder(pf)
- #First method of creating the halo lists:
- halo_list = rh.halo_list()
- #Alternate method of creating halo_list:
- halo_list = LoadRockstarHalos(pf, 'rockstar_halos/out_0.list')
-
-The above ``halo_list`` is very similar to any other list of halos loaded off
-disk.
-It is possible to access particle data and use the halos in a manner like any
-other halo object, and the particle data is only loaded on demand.
-Additionally, each halo object has additional information attached that is
-pulled directly from the Rockstar output:
-
-.. code-block:: python
-
- >>> halo_list[0].supp
- Out[3]:
- {'J': array([ -6.15271728e+15, -1.36593609e+17, -7.80776865e+16], dtype=float32),
- 'bulkvel': array([-132.05046082, 11.53190422, 42.16183472], dtype=float32),
- 'child_r': 2.6411054,
- 'corevel': array([-132.05046082, 11.53190422, 42.16183472], dtype=float32),
- 'desc': 0,
- 'energy': -8.106986e+21,
- 'flags': 1,
- 'id': 166,
- 'm': 1.5341227e+15,
- 'mgrav': 1.5341227e+15,
- 'min_bulkvel_err': 1821.8152,
- 'min_pos_err': 0.00049575343,
- 'min_vel_err': 1821.8152,
- 'n_core': 1958,
- 'num_child_particles': 2764,
- 'num_p': 2409,
- 'p_start': 6540,
- 'pos': array([ 0.20197368, 0.54656458, 0.11256824, -104.33285522,
- 29.02485085, 43.5154953 ], dtype=float32),
- 'r': 0.018403014,
- 'rs': 0.0026318002,
- 'rvmax': 1133.2,
- 'spin': 0.035755754,
- 'vmax': 1877.125,
- 'vrms': 1886.2648}
-
-Installation
-------------
-
-The Rockstar is slightly patched and modified to run as a library inside of
-yt. By default it will be built with yt using the ``install_script.sh``.
-If it wasn't installed, please make sure that the installation setting
-``INST_ROCKSTAR=1`` is defined in the ``install_script.sh`` and re-run
-the installation script.
-
-Rockstar Inline with Enzo
--------------------------
-
-It is possible to run Rockstar inline with Enzo. Setting up
-Enzo with inline yt is covered
-`here <http://enzo-project.org/doc/user_guide/EmbeddedPython.html>`_.
-It is not necessary to run Enzo with load balancing off to use Rockstar.
-Here is an example ``user_script.py``:
-
-.. code-block:: python
-
- from yt.mods import *
- from yt.analysis_modules.halo_finding.api import *
- from yt.config import ytcfg
- from yt.analysis_modules.halo_finding.rockstar.api import *
-
- def main():
- import enzo
- pf = EnzoDatasetInMemory()
- mine = ytcfg.getint('yt','__topcomm_parallel_rank')
- size = ytcfg.getint('yt','__topcomm_parallel_size')
-
- # Call rockstar.
- ts = DatasetSeries([pf])
- outbase = "./rockstar_halos_%04d" % pf['NumberOfPythonTopGridCalls']
- rh = RockstarHaloFinder(ts, num_readers = size,
- outbase = outbase)
- rh.run()
-
- # Load the halos off disk.
- fname = outbase + "/out_0.list"
- rhalos = LoadRockstarHalos(pf, fname)
-
https://bitbucket.org/yt_analysis/yt/commits/d62074402985/
Changeset: d62074402985
Branch: yt-3.0
User: hegan
Date: 2014-07-17 21:33:35
Summary: finished rewriting halo finder doc file
Affected #: 1 file
diff -r 95be98a1282eab1c60b5acd3967b359fd66126f3 -r d6207440298572783f6df62cf96c6bd6c341c219 doc/source/analyzing/analysis_modules/halo_finders.rst
--- a/doc/source/analyzing/analysis_modules/halo_finders.rst
+++ b/doc/source/analyzing/analysis_modules/halo_finders.rst
@@ -59,6 +59,7 @@
3. The final groups are formed the networks of particles linked
together by friends, hence the name.
+
.. warning:: The FoF halo finder in yt is not thoroughly tested!
It is probably fine to use, but you are strongly encouraged
to check your results against the data for errors.
@@ -80,7 +81,10 @@
.. note:: At the moment, Rockstar does not support multiple particle masses,
instead using a fixed particle mass. This will not affect most dark matter
simulations, but does make it less useful for finding halos from the stellar
- mass.
+ mass. In simulations where the highest-resolution particles all have the
+ same mass (ie: zoom-in grid based simulations), one can set up a particle
+ filter to select the lowest mass particles and perform the halo finding
+ only on those.
To run the Rockstar Halo finding, you must launch python with MPI and
parallelization enabled. While Rockstar itself does not require MPI to run,
@@ -94,28 +98,14 @@
For example, here is how Rockstar might be called using 24 cores:
``mpirun -n 24 --mca btl ^openib python ./run_rockstar.py --parallel``.
-Designing the python script itself is straightforward:
-
-.. code-block:: python
-
- from yt.mods import *
- from yt.analysis_modules.halo_finding.rockstar.api import RockstarHaloFinder
-
- #find all of our simulation files
- files = glob.glob("Enzo_64/DD*/\*index")
- #hopefully the file name order is chronological
- files.sort()
- ts = DatasetSeries.from_filenames(files[:])
- rh = RockstarHaloFinder(ts)
- rh.run()
-
The script above configures the Halo finder, launches a server process which
disseminates run information and coordinates writer-reader processes.
Afterwards, it launches reader and writer tasks, filling the available MPI
slots, which alternately read particle information and analyze for halo
content.
-The RockstarHaloFinder class has these options:
+The RockstarHaloFinder class has these options that can be supplied to the
+halo catalog through the ``finder_kwargs`` argument:
* ``dm_type``, the index of the dark matter particle. Default is 1.
* ``outbase``, This is where the out*list files that Rockstar makes should be
placed. Default is 'rockstar_halos'.
@@ -201,11 +191,14 @@
.. code-block:: python
from yt.mods import *
- from yt.analysis_modules.halo_finding.api import *
- pf = load("data0001")
- halo_list = HaloFinder(pf,padding=0.02)
+ from yt.analysis_modules.halo_analysis.api import *
+ ds = load("data0001")
+ hc= HaloCatalog(data_pf =ds,finder_method='hop'
+ finder_kwargs={'padding':0.02})
# --or--
- halo_list = FOFHaloFinder(pf,padding=0.02)
+ hc= HaloCatalog(data_pf =ds,finder_method='fof'
+ finder_kwargs={'padding':0.02})
+
In general, a little bit of padding goes a long way, and too much
just slows down the analysis and doesn't improve the answer (but
https://bitbucket.org/yt_analysis/yt/commits/d3b24f37cb49/
Changeset: d3b24f37cb49
Branch: yt-3.0
User: hegan
Date: 2014-07-17 21:40:54
Summary: removed halo profiling doc
Affected #: 1 file
diff -r d6207440298572783f6df62cf96c6bd6c341c219 -r d3b24f37cb494f5d078212e34a45db93c7c55eb6 doc/source/analyzing/analysis_modules/halo_profiling.rst
--- a/doc/source/analyzing/analysis_modules/halo_profiling.rst
+++ /dev/null
@@ -1,451 +0,0 @@
-.. _halo_profiling:
-
-Halo Profiling
-==============
-.. sectionauthor:: Britton Smith <brittonsmith at gmail.com>,
- Stephen Skory <s at skory.us>
-
-The ``HaloProfiler`` provides a means of performing analysis on multiple halos
-in a parallel-safe way.
-
-The halo profiler performs three primary functions: radial profiles,
-projections, and custom analysis. See the cookbook for a recipe demonstrating
-all of these features.
-
-Configuring the Halo Profiler
------------------------------
-
-The only argument required to create a ``HaloProfiler`` object is the path
-to the dataset.
-
-.. code-block:: python
-
- from yt.analysis_modules.halo_profiler.api import *
- hp = HaloProfiler("enzo_tiny_cosmology/DD0046/DD0046")
-
-Most of the halo profiler's options are configured with additional keyword
-arguments:
-
- * **output_dir** (*str*): if specified, all output will be put into this path
- instead of in the dataset directories. Default: None.
-
- * **halos** (*str*): "multiple" for profiling more than one halo. In this mode
- halos are read in from a list or identified with a
- `halo finder <../cookbook/running_halofinder.html>`_. In "single" mode, the
- one and only halo center is identified automatically as the location of the
- peak in the density field. Default: "multiple".
-
- * **halo_list_file** (*str*): name of file containing the list of halos.
- The halo profiler will look for this file in the data directory.
- Default: "HopAnalysis.out".
-
- * **halo_list_format** (*str* or *dict*): the format of the halo list file.
- "yt_hop" for the format given by yt's halo finders. "enzo_hop" for the
- format written by enzo_hop. This keyword can also be given in the form of a
- dictionary specifying the column in which various properties can be found.
- For example, {"id": 0, "center": [1, 2, 3], "mass": 4, "radius": 5}.
- Default: "yt_hop".
-
- * **halo_finder_function** (*function*): If halos is set to multiple and the
- file given by halo_list_file does not exit, the halo finding function
- specified here will be called. Default: HaloFinder (yt_hop).
-
- * **halo_finder_args** (*tuple*): args given with call to halo finder function.
- Default: None.
-
- * **halo_finder_kwargs** (*dict*): kwargs given with call to halo finder
- function. Default: None.
-
- * **recenter** (*string* or function name): The name of a function
- that will be used to move the center of the halo for the purposes of
- analysis. See explanation and examples, below. Default: None, which
- is equivalent to the center of mass of the halo as output by the halo
- finder.
-
- * **halo_radius** (*float*): if no halo radii are provided in the halo list
- file, this parameter is used to specify the radius out to which radial
- profiles will be made. This keyword is also used when halos is set to
- single. Default: 0.1.
-
- * **radius_units** (*str*): the units of **halo_radius**.
- Default: "1" (code units).
-
- * **n_profile_bins** (*int*): the number of bins in the radial profiles.
- Default: 50.
-
- * **profile_output_dir** (*str*): the subdirectory, inside the data directory,
- in which radial profile output files will be created. The directory will be
- created if it does not exist. Default: "radial_profiles".
-
- * **projection_output_dir** (*str*): the subdirectory, inside the data
- directory, in which projection output files will be created. The directory
- will be created if it does not exist. Default: "projections".
-
- * **projection_width** (*float*): the width of halo projections.
- Default: 8.0.
-
- * **projection_width_units** (*str*): the units of projection_width.
- Default: "mpc".
-
- * **project_at_level** (*int* or "max"): the maximum refinement level to be
- included in projections. Default: "max" (maximum level within the dataset).
-
- * **velocity_center** (*list*): the method in which the halo bulk velocity is
- calculated (used for calculation of radial and tangential velocities. Valid
- options are:
- - ["bulk", "halo"] (Default): the velocity provided in the halo list
- - ["bulk", "sphere"]: the bulk velocity of the sphere centered on the halo center.
- - ["max", field]: the velocity of the cell that is the location of the maximum of the field specified.
-
- * **filter_quantities** (*list*): quantities from the original halo list
- file to be written out in the filtered list file. Default: ['id','center'].
-
- * **use_critical_density** (*bool*): if True, the definition of overdensity
- for virial quantities is calculated with respect to the critical
- density. If False, overdensity is with respect to mean matter density,
- which is lower by a factor of Omega_M. Default: False.
-
-Profiles
---------
-
-Once the halo profiler object has been instantiated, fields can be added for
-profiling with the :meth:`add_profile` method:
-
-.. code-block:: python
-
- hp.add_profile('cell_volume', weight_field=None, accumulation=True)
- hp.add_profile('TotalMassMsun', weight_field=None, accumulation=True)
- hp.add_profile('density', weight_field=None, accumulation=False)
- hp.add_profile('temperature', weight_field='cell_mass', accumulation=False)
- hp.make_profiles(njobs=-1, prefilters=["halo['mass'] > 1e13"],
- filename='VirialQuantities.h5')
-
-The :meth:`make_profiles` method will begin the profiling. Use the
-**njobs** keyword to control the number of jobs over which the
-profiling is divided. Setting to -1 results in a single processor per
-halo. Setting to 1 results in all available processors working on the
-same halo. The prefilters keyword tells the profiler to skip all halos with
-masses (as loaded from the halo finder) less than a given amount. See below
-for more information. Additional keyword arguments are:
-
- * **filename** (*str*): If set, a file will be written with all of the
- filtered halos and the quantities returned by the filter functions.
- Default: None.
-
- * **prefilters** (*list*): A single dataset can contain thousands or tens of
- thousands of halos. Significant time can be saved by not profiling halos
- that are certain to not pass any filter functions in place. Simple filters
- based on quantities provided in the initial halo list can be used to filter
- out unwanted halos using this parameter. Default: None.
-
- * **njobs** (*int*): The number of jobs over which to split the profiling.
- Set to -1 so that each halo is done by a single processor. Default: -1.
-
- * **dynamic** (*bool*): If True, distribute halos using a task queue. If
- False, distribute halos evenly over all jobs. Default: False.
-
- * **profile_format** (*str*): The file format for the radial profiles,
- 'ascii' or 'hdf5'. Default: 'ascii'.
-
-.. image:: _images/profiles.png
- :width: 500
-
-Radial profiles of Overdensity (left) and Temperature (right) for five halos.
-
-Projections
------------
-
-The process of making projections is similar to that of profiles:
-
-.. code-block:: python
-
- hp.add_projection('density', weight_field=None)
- hp.add_projection('temperature', weight_field='density')
- hp.add_projection('metallicity', weight_field='density')
- hp.make_projections(axes=[0, 1, 2], save_cube=True, save_images=True,
- halo_list="filtered", njobs=-1)
-
-If **save_cube** is set to True, the projection data
-will be written to a set of hdf5 files
-in the directory given by **projection_output_dir**.
-The keyword, **halo_list**, can be
-used to select between the full list of halos ("all"),
-the filtered list ("filtered"), or
-an entirely new list given in the form of a file name.
-See :ref:`filter_functions` for a
-discussion of filtering halos. Use the **njobs** keyword to control
-the number of jobs over which the profiling is divided. Setting to -1
-results in a single processor per halo. Setting to 1 results in all
-available processors working on the same halo. The keyword arguments are:
-
- * **axes** (*list*): A list of the axes to project along, using the usual
- 0,1,2 convention. Default=[0,1,2].
-
- * **halo_list** (*str*) {'filtered', 'all'}: Which set of halos to make
- profiles of, either ones passed by the halo filters (if enabled/added), or
- all halos. Default='filtered'.
-
- * **save_images** (*bool*): Whether or not to save images of the projections.
- Default=False.
-
- * **save_cube** (*bool*): Whether or not to save the HDF5 files of the halo
- projections. Default=True.
-
- * **njobs** (*int*): The number of jobs over which to split the projections.
- Set to -1 so that each halo is done by a single processor. Default: -1.
-
- * **dynamic** (*bool*): If True, distribute halos using a task queue. If
- False, distribute halos evenly over all jobs. Default: False.
-
-.. image:: _images/projections.png
- :width: 500
-
-Projections of Density (top) and Temperature,
-weighted by Density (bottom), in the x (left),
-y (middle), and z (right) directions for a single halo with a width of 8 Mpc.
-
-Halo Filters
-------------
-
-Filters can be added to create a refined list of
-halos based on their profiles or to avoid
-profiling halos altogether based on information
-given in the halo list file.
-
-.. _filter_functions:
-
-Filter Functions
-^^^^^^^^^^^^^^^^
-
-It is often the case that one is looking to
-identify halos with a specific set of
-properties. This can be accomplished through the creation
-of filter functions. A filter
-function can take as many args and kwargs as you like,
-as long as the first argument is a
-profile object, or at least a dictionary which contains
-the profile arrays for each field.
-Filter functions must return a list of two things.
-The first is a True or False indicating
-whether the halo passed the filter.
-The second is a dictionary containing quantities
-calculated for that halo that will be written to a
-file if the halo passes the filter.
-A sample filter function based on virial quantities can be found in
-``yt/analysis_modules/halo_profiler/halo_filters.py``.
-
-Halo filtering takes place during the call to :meth:`make_profiles`.
-The :meth:`add_halo_filter` method is used to add a filter to be used
-during the profiling:
-
-.. code-block:: python
-
- hp.add_halo_filter(HP.VirialFilter, must_be_virialized=True,
- overdensity_field='ActualOverdensity',
- virial_overdensity=200,
- virial_filters=[['TotalMassMsun','>=','1e14']],
- virial_quantities=['TotalMassMsun','RadiusMpc'],
- use_log=True)
-
-The addition above will calculate and return virial quantities,
-mass and radius, for an
-overdensity of 200. In order to pass the filter, at least one
-point in the profile must be
-above the specified overdensity and the virial mass must be at
-least 1e14 solar masses. The **use_log** keyword indicates that interpolation
-should be done in log space. If
-the VirialFilter function has been added to the filter list,
-the halo profiler will make
-sure that the fields necessary for calculating virial quantities are added.
-As many filters as desired can be added. If filters have been added,
-the next call to :meth:`make_profiles` will filter by all of
-the added filter functions:
-
-.. code-block:: python
-
- hp.make_profiles(filename="FilteredQuantities.out")
-
-If the **filename** keyword is set, a file will be written with all of the
-filtered halos and the quantities returned by the filter functions.
-
-.. note:: If the profiles have already been run, the halo profiler will read
- in the previously created output files instead of re-running the profiles.
- The halo profiler will check to make sure the output file contains all of
- the requested halo fields. If not, the profile will be made again from
- scratch.
-
-.. _halo_profiler_pre_filters:
-
-Pre-filters
-^^^^^^^^^^^
-
-A single dataset can contain thousands or tens of thousands of halos.
-Significant time can
-be saved by not profiling halos that are certain to not pass any filter
-functions in place.
-Simple filters based on quantities provided in the initial halo list
-can be used to filter
-out unwanted halos using the **prefilters** keyword:
-
-.. code-block:: python
-
- hp.make_profiles(filename="FilteredQuantities.out",
- prefilters=["halo['mass'] > 1e13"])
-
-Arguments provided with the **prefilters** keyword should be given
-as a list of strings.
-Each string in the list will be evaluated with an *eval*.
-
-.. note:: If a VirialFilter function has been added with a filter based
- on mass (as in the example above), a prefilter will be automatically
- added to filter out halos with masses greater or less than (depending
- on the conditional of the filter) a factor of ten of the specified
- virial mass.
-
-Recentering the Halo For Analysis
----------------------------------
-
-It is possible to move the center of the halo to a new point using an
-arbitrary function for making profiles.
-By default, the center is provided by the halo finder,
-which outputs the center of mass of the particles. For the purposes of
-analysis, it may be important to recenter onto a gas density maximum,
-or a temperature minimum.
-
-There are a number of built-in functions to do this, listed below.
-Each of the functions uses mass-weighted fields for the calculations
-of new center points.
-To use
-them, supply the HaloProfiler with the ``recenter`` option and
-the name of the function, as in the example below.
-
-.. code-block:: python
-
- hp = HaloProfiler("enzo_tiny_cosmology/DD0046/DD0046",
- recenter="Max_Dark_Matter_Density")
-
-Additional options are:
-
- * *Min_Dark_Matter_Density* - Recenter on the point of minimum dark matter
- density in the halo.
-
- * *Max_Dark_Matter_Density* - Recenter on the point of maximum dark matter
- density in the halo.
-
- * *CoM_Dark_Matter_Density* - Recenter on the center of mass of the dark
- matter density field. This will be very similar to what the halo finder
- provides, but not precisely similar.
-
- * *Min_Gas_Density* - Recenter on the point of minimum gas density in the
- halo.
-
- * *Max_Gas_Density* - Recenter on the point of maximum gas density in the
- halo.
-
- * *CoM_Gas_Density* - Recenter on the center of mass of the gas density field
- in the halo.
-
- * *Min_Total_Density* - Recenter on the point of minimum total (gas + dark
- matter) density in the halo.
-
- * *Max_Total_Density* - Recenter on the point of maximum total density in the
- halo.
-
- * *CoM_Total_Density* - Recenter on the center of mass for the total density
- in the halo.
-
- * *Min_Temperature* - Recenter on the point of minimum temperature in the
- halo.
-
- * *Max_Temperature* - Recenter on the point of maximum temperature in the
- halo.
-
-It is also possible to supply a user-defined function to the HaloProfiler.
-This can be used if the pre-defined functions above are not sufficient.
-The function takes a single argument, a data container for the halo,
-which is a sphere. The function returns a 3-list with the new center.
-
-In this example below, a function is used such that the halos will be
-re-centered on the point of absolute minimum temperature, that is not
-mass weighted.
-
-.. code-block:: python
-
- from yt.mods import *
-
- def find_min_temp(sphere):
- ma, mini, mx, my, mz, mg = sphere.quantities['MinLocation']('temperature')
- return [mx,my,mz]
-
- hp = HaloProfiler("enzo_tiny_cosmology/DD0046/DD0046", recenter=find_min_temp)
-
-It is possible to make more complicated functions. This example below extends
-the example above to include a distance control that prevents the center from
-being moved too far. If the recenter moves too far, ``[-1, -1, -1]`` is
-returned which will prevent the halo from being profiled.
-Any triplet of values less than the ``domain_left_edge`` will suffice.
-There will be a note made in the output (stderr) showing which halos were
-skipped.
-
-.. code-block:: python
-
- from yt.mods import *
- from yt.utilities.math_utils import periodic_dist
-
- def find_min_temp_dist(sphere):
- old = sphere.center
- ma, mini, mx, my, mz, mg = sphere.quantities['MinLocation']('temperature')
- d = sphere.pf['kpc'] * periodic_dist(old, [mx, my, mz],
- sphere.pf.domain_right_edge - sphere.pf.domain_left_edge)
- # If new center farther than 5 kpc away, don't recenter
- if d > 5.: return [-1, -1, -1]
- return [mx,my,mz]
-
- hp = HaloProfiler("enzo_tiny_cosmology/DD0046/DD0046",
- recenter=find_min_temp_dist)
-
-Custom Halo Analysis
---------------------
-
-Besides radial profiles and projections, the halo profiler has the
-ability to run custom analysis functions on each halo. Custom halo
-analysis functions take two arguments: a halo dictionary containing
-the id, center, etc; and a sphere object. The example function shown
-below creates a 2D profile of the total mass in bins of density and
-temperature for a given halo.
-
-.. code-block:: python
-
- from yt.mods import *
- from yt.data_objects.profiles import BinnedProfile2D
-
- def halo_2D_profile(halo, sphere):
- "Make a 2D profile for a halo."
- my_profile = BinnedProfile2D(sphere,
- 128, 'density', 1e-30, 1e-24, True,
- 128, 'temperature', 1e2, 1e7, True,
- end_collect=False)
- my_profile.add_fields('cell_mass', weight=None, fractional=False)
- my_filename = os.path.join(sphere.pf.fullpath, '2D_profiles',
- 'Halo_%04d.h5' % halo['id'])
- my_profile.write_out_h5(my_filename)
-
-Using the :meth:`analyze_halo_spheres` function, the halo profiler
-will create a sphere centered on each halo, and perform the analysis
-from the custom routine.
-
-.. code-block:: python
-
- hp.analyze_halo_sphere(halo_2D_profile, halo_list='filtered',
- analysis_output_dir='2D_profiles',
- njobs=-1, dynamic=False)
-
-Just like with the :meth:`make_projections` function, the keyword,
-**halo_list**, can be used to select between the full list of halos
-("all"), the filtered list ("filtered"), or an entirely new list given
-in the form of a file name. If the **analysis_output_dir** keyword is
-set, the halo profiler will make sure the desired directory exists in
-a parallel-safe manner. Use the **njobs** keyword to control the
-number of jobs over which the profiling is divided. Setting to -1
-results in a single processor per halo. Setting to 1 results in all
-available processors working on the same halo.
https://bitbucket.org/yt_analysis/yt/commits/4bc377ee9256/
Changeset: 4bc377ee9256
Branch: yt-3.0
User: hegan
Date: 2014-07-18 00:33:49
Summary: filled out all of transition documentation
Affected #: 2 files
diff -r d3b24f37cb494f5d078212e34a45db93c7c55eb6 -r 4bc377ee9256063d668187b08554b0c0930f7ecb doc/source/analyzing/analysis_modules/halo_catalogs.rst
--- a/doc/source/analyzing/analysis_modules/halo_catalogs.rst
+++ b/doc/source/analyzing/analysis_modules/halo_catalogs.rst
@@ -8,7 +8,7 @@
different from the limited framework included in yt-2.x and is only
backwards compatible in that output from old halo finders may be loaded.
For a direct translation of various halo analysis tasks using yt-2.x
-to yt-3.0 please see :ref:`halo_translation'.
+to yt-3.0 please see :ref:`halo_transition`.
A catalog of halos can be created from any initial dataset given to halo
catalog through data_pf. These halos can be found using friends-of-friends,
diff -r d3b24f37cb494f5d078212e34a45db93c7c55eb6 -r 4bc377ee9256063d668187b08554b0c0930f7ecb doc/source/analyzing/analysis_modules/halo_transition.rst
--- /dev/null
+++ b/doc/source/analyzing/analysis_modules/halo_transition.rst
@@ -0,0 +1,106 @@
+
+Getting up to Speed with Halo Analysis in yt-3.0
+================================================
+
+If you're used to halo analysis in yt-2.x, heres a guide to
+how to update your analysis pipeline to take advantage of
+the new halo catalog infrastructure.
+
+Finding Halos
+-------------
+
+Previously, halos were found using calls to ``HaloFinder``,
+``FOFHaloFinder`` and ``RockstarHaloFinder``. Now it is
+encouraged that you find the halos upon creation of the halo catalog
+by supplying a value to the ``finder_method`` keyword when calling
+``HaloCatalog``. Currently, only halos found using rockstar or a
+previous instance of a halo catalog are able to be loaded
+using the ``halos_pf`` keyword.
+
+To pass additional arguments to the halo finders
+themselves, supply a dictionary to ``finder_kwargs`` where
+each key in the dictionary is a keyword of the halo finder
+and the corresponding value is the value to be passed for
+that keyword.
+
+Getting Halo Information
+------------------------
+All quantities that used to be present in a ``halo_list`` are
+still able to be found but are not necessarily included by default.
+Every halo will by default have the following properties:
+
+* particle_position_i (where i can be x,y,z)
+* particle_mass
+* virial_radius
+* particle_identifier
+
+If other quantities are desired, they can be included by adding
+the corresponding quantity before the catalog is created. See
+the full halo catalog documentation for further information about
+how to add these quantities and what quantities are available.
+
+You no longer have to iteratre over halos in the ``halo_list``.
+Now a halo dataset can be treated as a regular dataset and
+all quantities are available by accessing ``all_data``.
+Specifically, all quantities can be accessed as shown:
+
+.. code-block:: python
+ from yt.mods import *
+ from yt.analysis_modules.halo_analysis.api import HaloCatalog
+ data_pf = load('Enzo_64/RD0006/RedshiftOutput0006')
+ hc = HaloCatalog(data_pf=data_pf, finder_method='hop')
+ hc.create()
+ ad = hc.all_data()
+ masses = ad['particle_mass'][:]
+
+
+Prefiltering Halos
+------------------
+
+Prefiltering halos before analysis takes place is now done
+by adding a filter before the call to create. An example
+is shown below
+
+.. code-block:: python
+ from yt.mods import *
+ from yt.analysis_modules.halo_analysis.api import HaloCatalog
+ data_pf = load('Enzo_64/RD0006/RedshiftOutput0006')
+ hc = HaloCatalog(data_pf=data_pf, finder_method='hop')
+ hc.add_filter("quantity_value", "particle_mass", ">", 1e13, "Msun")
+ hc.create()
+
+Profiling Halos
+---------------
+
+The halo profiler available in yt-2.x has been removed, and
+profiling functionality is now completely contained within the
+halo catalog. A complete example of how to profile halos by
+radius using the new infrastructure is given in
+:ref:`halo_analysis_example`.
+
+Plotting Halos
+--------------
+
+Annotating halo locations onto a slice or projection works in
+the same way as in yt-2.x, but now a halo catalog must be
+passed to the annotate halo call rather than a halo list.
+
+.. code-block:: python
+ from yt.mods import *
+ from yt.analysis_modules.halo_analysis.api import HaloCatalog
+
+ data_pf = load('Enzo_64/RD0006/RedshiftOutput0006')
+ hc = HaloCatalog(data_pf=data_pf, finder_method='hop')
+ hc.create()
+
+ prj = ProjectionPlot(data_pf, 'z', 'density')
+ prj.annotate_halos(hc)
+ prj.save()
+
+Written Data
+------------
+
+Data is now written out in the form of h5 files rather than
+text files. The directory they are written out to is
+controlled by the keyword ``output_dir``. Each quantity
+is a field in the file.
https://bitbucket.org/yt_analysis/yt/commits/63fda44592b2/
Changeset: 63fda44592b2
Branch: yt-3.0
User: hegan
Date: 2014-07-18 00:50:01
Summary: changed cookbook halo profiling to new profiler
Affected #: 1 file
diff -r 4bc377ee9256063d668187b08554b0c0930f7ecb -r 63fda44592b21ce1593e80d31eaa5418cc792410 doc/source/cookbook/halo_profiler.py
--- a/doc/source/cookbook/halo_profiler.py
+++ b/doc/source/cookbook/halo_profiler.py
@@ -1,51 +1,44 @@
-### THIS RECIPE IS CURRENTLY BROKEN IN YT-3.0
-### DO NOT TRUST THIS RECIPE UNTIL THIS LINE IS REMOVED
+from yt.mods import *
+from yt.analysis_modules.halo_analysis.api import *
-from yt.mods import *
+# Load the data set with the full simulation information
+# and rockstar halos
+data_pf = load('Enzo_64/RD0006/RedshiftOutput0006')
+halos_pf = load('rockstar_halos/halos_0.0.bin')
-from yt.analysis_modules.halo_profiler.api import *
+# Instantiate a catalog using those two paramter files
+hc = HaloCatalog(data_pf=data_pf, halos_pf=halos_pf)
-# Define a custom function to be called on all halos.
-# The first argument is a dictionary containing the
-# halo id, center, etc.
-# The second argument is the sphere centered on the halo.
-def get_density_extrema(halo, sphere):
- my_extrema = sphere.quantities['Extrema']('density')
- mylog.info('Halo %d has density extrema: %s',
- halo['id'], my_extrema)
+# Filter out less massive halos
+hc.add_filter("quantity_value", "particle_mass", ">", 1e14, "Msun")
+# attach a sphere object to each halo whose radius extends
+# to twice the radius of the halo
+hc.add_callback("sphere", factor=2.0)
-# Instantiate HaloProfiler for this dataset.
-hp = HaloProfiler('enzo_tiny_cosmology/DD0046/DD0046',
- output_dir='.')
+# use the sphere to calculate radial profiles of gas density
+# weighted by cell volume in terms of the virial radius
+hc.add_callback("profile", x_field="radius",
+ y_fields=[("gas", "overdensity")],
+ weight_field="cell_volume",
+ accumulation=False,
+ storage="virial_quantities_profiles")
-# Add a filter to remove halos that have no profile points with overdensity
-# above 200, and with virial masses less than 1e10 solar masses.
-# Also, return the virial mass and radius to be written out to a file.
-hp.add_halo_filter(amods.halo_profiler.VirialFilter, must_be_virialized=True,
- overdensity_field='ActualOverdensity',
- virial_overdensity=200,
- virial_filters=[['TotalMassMsun', '>=', '1e10']],
- virial_quantities=['TotalMassMsun', 'RadiusMpc'])
-# Add profile fields.
-hp.add_profile('cell_volume', weight_field=None, accumulation=True)
-hp.add_profile('TotalMassMsun', weight_field=None, accumulation=True)
-hp.add_profile('density', weight_field='cell_mass', accumulation=False)
-hp.add_profile('temperature', weight_field='cell_mass', accumulation=False)
+hc.add_callback("virial_quantities", ["radius"],
+ profile_storage="virial_quantities_profiles")
+hc.add_callback('delete_attribute', 'virial_quantities_profiles')
-# Make profiles and output filtered halo list to FilteredQuantities.h5.
-hp.make_profiles(filename="FilteredQuantities.h5",
- profile_format='hdf5', njobs=-1)
+field_params = dict(virial_radius=('quantity', 'radius_200'))
+hc.add_callback('sphere', radius_field='radius_200', factor=5,
+ field_parameters=field_params)
+hc.add_callback('profile', 'virial_radius', [('gas', 'temperature')],
+ storage='virial_profiles',
+ weight_field='cell_mass',
+ accumulation=False, output_dir='profiles')
-# Add projection fields.
-hp.add_projection('density', weight_field=None)
-hp.add_projection('temperature', weight_field='density')
-hp.add_projection('metallicity', weight_field='density')
+# Save the profiles
+hc.add_callback("save_profiles", storage="virial_profiles",
+ output_dir="profiles")
-# Make projections just along the x axis using the filtered halo list.
-hp.make_projections(save_cube=False, save_images=True,
- halo_list='filtered', axes=[0], njobs=-1)
-
-# Run our custom analysis function on all halos in the filtered list.
-hp.analyze_halo_spheres(get_density_extrema, njobs=-1)
+hc.create()
https://bitbucket.org/yt_analysis/yt/commits/21544e0585a0/
Changeset: 21544e0585a0
Branch: yt-3.0
User: hegan
Date: 2014-07-18 00:50:56
Summary: made a note that merger trees dont work with new halo analysis tools
Affected #: 1 file
diff -r 63fda44592b21ce1593e80d31eaa5418cc792410 -r 21544e0585a0b180a5719b59adc3c5ec37f5dfa3 doc/source/analyzing/analysis_modules/merger_tree.rst
--- a/doc/source/analyzing/analysis_modules/merger_tree.rst
+++ b/doc/source/analyzing/analysis_modules/merger_tree.rst
@@ -5,6 +5,9 @@
.. sectionauthor:: Stephen Skory <sskory at physics.ucsd.edu>
.. versionadded:: 1.7
+.. note:: At the moment the merger tree is not yet implemented using new
+ halo catalog functionality.
+
The Halo Merger Tree extension is capable of building a database of halo mergers
over a set of time-ordered Enzo datasets. The fractional contribution of older
'parent' halos to younger 'child' halos is calculated by comparing the unique
https://bitbucket.org/yt_analysis/yt/commits/2db3f4377e8d/
Changeset: 2db3f4377e8d
Branch: yt-3.0
User: hegan
Date: 2014-07-18 00:52:30
Summary: tweaks to halo transition
Affected #: 2 files
diff -r 21544e0585a0b180a5719b59adc3c5ec37f5dfa3 -r 2db3f4377e8df30ff6325c44cc6fef2af3cb0a62 doc/source/analyzing/analysis_modules/halo_analysis.rst
--- a/doc/source/analyzing/analysis_modules/halo_analysis.rst
+++ b/doc/source/analyzing/analysis_modules/halo_analysis.rst
@@ -8,6 +8,7 @@
:maxdepth: 1
halo_catalogs
+ halo_transition
halo_finding
halo_mass_function
halo_analysis_example
diff -r 21544e0585a0b180a5719b59adc3c5ec37f5dfa3 -r 2db3f4377e8df30ff6325c44cc6fef2af3cb0a62 doc/source/analyzing/analysis_modules/halo_translation.rst
--- a/doc/source/analyzing/analysis_modules/halo_translation.rst
+++ /dev/null
@@ -1,7 +0,0 @@
-
-Getting up to Speed with Halo Analysis in yt-3.0
-================================================
-
-If you're used to halo analysis in yt-2.x, heres a guide to
-how to update your analysis pipeline to take advantage of
-the new halo catalog infrastructure.
https://bitbucket.org/yt_analysis/yt/commits/d0d742f3ad2e/
Changeset: d0d742f3ad2e
Branch: yt-3.0
User: hegan
Date: 2014-07-18 19:40:20
Summary: removed section authors
Affected #: 2 files
diff -r 2db3f4377e8df30ff6325c44cc6fef2af3cb0a62 -r d0d742f3ad2eafa6f8948b00197de265071a7e08 doc/source/analyzing/analysis_modules/halo_finders.rst
--- a/doc/source/analyzing/analysis_modules/halo_finders.rst
+++ b/doc/source/analyzing/analysis_modules/halo_finders.rst
@@ -2,7 +2,6 @@
Halo Finding
============
-.. sectionauthor:: Stephen Skory <sskory at physics.ucsd.edu>
There are four methods of finding particle haloes in yt. The
recommended and default method is called HOP, a method described
diff -r 2db3f4377e8df30ff6325c44cc6fef2af3cb0a62 -r d0d742f3ad2eafa6f8948b00197de265071a7e08 doc/source/analyzing/analysis_modules/merger_tree.rst
--- a/doc/source/analyzing/analysis_modules/merger_tree.rst
+++ b/doc/source/analyzing/analysis_modules/merger_tree.rst
@@ -2,8 +2,6 @@
Halo Merger Tree
================
-.. sectionauthor:: Stephen Skory <sskory at physics.ucsd.edu>
-.. versionadded:: 1.7
.. note:: At the moment the merger tree is not yet implemented using new
halo catalog functionality.
https://bitbucket.org/yt_analysis/yt/commits/b5b5a819adf0/
Changeset: b5b5a819adf0
Branch: yt-3.0
User: hegan
Date: 2014-07-18 21:15:13
Summary: halo_pf -> halo_ds
Affected #: 4 files
diff -r d0d742f3ad2eafa6f8948b00197de265071a7e08 -r b5b5a819adf0354aacda16aae95814420bc96ba7 doc/source/analyzing/analysis_modules/Halo_Analysis.ipynb
--- a/doc/source/analyzing/analysis_modules/Halo_Analysis.ipynb
+++ b/doc/source/analyzing/analysis_modules/Halo_Analysis.ipynb
@@ -44,7 +44,7 @@
"tmpdir = tempfile.mkdtemp()\n",
"\n",
"# Load the data set with the full simulation information\n",
- "data_pf = load('Enzo_64/RD0006/RedshiftOutput0006')"
+ "data_ds = load('Enzo_64/RD0006/RedshiftOutput0006')"
],
"language": "python",
"metadata": {},
@@ -62,7 +62,7 @@
"collapsed": false,
"input": [
"# Load the rockstar data files\n",
- "halos_pf = load('rockstar_halos/halos_0.0.bin')"
+ "halos_ds = load('rockstar_halos/halos_0.0.bin')"
],
"language": "python",
"metadata": {},
@@ -80,7 +80,7 @@
"collapsed": false,
"input": [
"# Instantiate a catalog using those two paramter files\n",
- "hc = HaloCatalog(data_pf=data_pf, halos_pf=halos_pf, \n",
+ "hc = HaloCatalog(data_ds=data_ds, halos_ds=halos_ds, \n",
" output_dir=os.path.join(tmpdir, 'halo_catalog'))"
],
"language": "python",
diff -r d0d742f3ad2eafa6f8948b00197de265071a7e08 -r b5b5a819adf0354aacda16aae95814420bc96ba7 doc/source/analyzing/analysis_modules/halo_catalogs.rst
--- a/doc/source/analyzing/analysis_modules/halo_catalogs.rst
+++ b/doc/source/analyzing/analysis_modules/halo_catalogs.rst
@@ -11,7 +11,7 @@
to yt-3.0 please see :ref:`halo_transition`.
A catalog of halos can be created from any initial dataset given to halo
-catalog through data_pf. These halos can be found using friends-of-friends,
+catalog through data_ds. These halos can be found using friends-of-friends,
HOP, and Rockstar. The finder_method keyword dictates which halo finder to
use. The available arguments are 'fof', 'hop', and'rockstar'. For more
details on the relative differences between these halo finders see
@@ -21,32 +21,32 @@
from yt.mods import *
from yt.analysis_modules.halo_analysis.api import HaloCatalog
- data_pf = load('Enzo_64/RD0006/RedshiftOutput0006')
- hc = HaloCatalog(data_pf=data_pf, finder_method='hop')
+ data_ds = load('Enzo_64/RD0006/RedshiftOutput0006')
+ hc = HaloCatalog(data_ds=data_ds, finder_method='hop')
A halo catalog may also be created from already run rockstar outputs.
This method is not implemented for previously run friends-of-friends or
HOP finders. Even though rockstar creates one file per processor,
specifying any one file allows the full catalog to be loaded. Here we
only specify the file output by the processor with ID 0. Note that the
-argument for supplying a rockstar output is `halos_pf`, not `data_pf`.
+argument for supplying a rockstar output is `halos_ds`, not `data_ds`.
.. code-block:: python
- halos_pf = load(path+'rockstar_halos/halos_0.0.bin')
- hc = HaloCatalog(halos_pf=halos_pf)
+ halos_ds = load(path+'rockstar_halos/halos_0.0.bin')
+ hc = HaloCatalog(halos_ds=halos_ds)
Although supplying only the binary output of the rockstar halo finder
is sufficient for creating a halo catalog, it is not possible to find
any new information about the identified halos. To associate the halos
with the dataset from which they were found, supply arguments to both
-halos_pf and data_pf.
+halos_ds and data_ds.
.. code-block:: python
- halos_pf = load(path+'rockstar_halos/halos_0.0.bin')
- data_pf = load('Enzo_64/RD0006/RedshiftOutput0006')
- hc = HaloCatalog(data_pf=data_pf, halos_pf=halos_pf)
+ halos_ds = load(path+'rockstar_halos/halos_0.0.bin')
+ data_ds = load('Enzo_64/RD0006/RedshiftOutput0006')
+ hc = HaloCatalog(data_ds=data_ds, halos_ds=halos_ds)
A data container can also be supplied via keyword data_source,
associated with either dataset, to control the spatial region in
@@ -218,7 +218,7 @@
.. code-block:: python
hpf = load(path+"halo_catalogs/catalog_0046/catalog_0046.0.h5")
- hc = HaloCatalog(halos_pf=hpf,
+ hc = HaloCatalog(halos_ds=hpf,
output_dir="halo_catalogs/catalog_0046")
hc.add_callback("load_profiles", output_dir="profiles",
filename="virial_profiles")
diff -r d0d742f3ad2eafa6f8948b00197de265071a7e08 -r b5b5a819adf0354aacda16aae95814420bc96ba7 doc/source/analyzing/analysis_modules/halo_finders.rst
--- a/doc/source/analyzing/analysis_modules/halo_finders.rst
+++ b/doc/source/analyzing/analysis_modules/halo_finders.rst
@@ -192,10 +192,10 @@
from yt.mods import *
from yt.analysis_modules.halo_analysis.api import *
ds = load("data0001")
- hc= HaloCatalog(data_pf =ds,finder_method='hop'
+ hc= HaloCatalog(data_ds =ds,finder_method='hop'
finder_kwargs={'padding':0.02})
# --or--
- hc= HaloCatalog(data_pf =ds,finder_method='fof'
+ hc= HaloCatalog(data_ds =ds,finder_method='fof'
finder_kwargs={'padding':0.02})
diff -r d0d742f3ad2eafa6f8948b00197de265071a7e08 -r b5b5a819adf0354aacda16aae95814420bc96ba7 doc/source/analyzing/analysis_modules/halo_transition.rst
--- a/doc/source/analyzing/analysis_modules/halo_transition.rst
+++ b/doc/source/analyzing/analysis_modules/halo_transition.rst
@@ -15,7 +15,7 @@
by supplying a value to the ``finder_method`` keyword when calling
``HaloCatalog``. Currently, only halos found using rockstar or a
previous instance of a halo catalog are able to be loaded
-using the ``halos_pf`` keyword.
+using the ``halos_ds`` keyword.
To pass additional arguments to the halo finders
themselves, supply a dictionary to ``finder_kwargs`` where
@@ -47,8 +47,8 @@
.. code-block:: python
from yt.mods import *
from yt.analysis_modules.halo_analysis.api import HaloCatalog
- data_pf = load('Enzo_64/RD0006/RedshiftOutput0006')
- hc = HaloCatalog(data_pf=data_pf, finder_method='hop')
+ data_ds = load('Enzo_64/RD0006/RedshiftOutput0006')
+ hc = HaloCatalog(data_ds=data_ds, finder_method='hop')
hc.create()
ad = hc.all_data()
masses = ad['particle_mass'][:]
@@ -64,8 +64,8 @@
.. code-block:: python
from yt.mods import *
from yt.analysis_modules.halo_analysis.api import HaloCatalog
- data_pf = load('Enzo_64/RD0006/RedshiftOutput0006')
- hc = HaloCatalog(data_pf=data_pf, finder_method='hop')
+ data_ds = load('Enzo_64/RD0006/RedshiftOutput0006')
+ hc = HaloCatalog(data_ds=data_ds, finder_method='hop')
hc.add_filter("quantity_value", "particle_mass", ">", 1e13, "Msun")
hc.create()
@@ -89,11 +89,11 @@
from yt.mods import *
from yt.analysis_modules.halo_analysis.api import HaloCatalog
- data_pf = load('Enzo_64/RD0006/RedshiftOutput0006')
- hc = HaloCatalog(data_pf=data_pf, finder_method='hop')
+ data_ds = load('Enzo_64/RD0006/RedshiftOutput0006')
+ hc = HaloCatalog(data_ds=data_ds, finder_method='hop')
hc.create()
- prj = ProjectionPlot(data_pf, 'z', 'density')
+ prj = ProjectionPlot(data_ds, 'z', 'density')
prj.annotate_halos(hc)
prj.save()
https://bitbucket.org/yt_analysis/yt/commits/f1871c6efd4d/
Changeset: f1871c6efd4d
Branch: yt-3.0
User: hegan
Date: 2014-07-18 22:36:07
Summary: removing even more section authors
Affected #: 1 file
diff -r b5b5a819adf0354aacda16aae95814420bc96ba7 -r f1871c6efd4d0de3fd8027b2f0bf25ac9ae955e1 doc/source/analyzing/analysis_modules/halo_finders.rst
--- a/doc/source/analyzing/analysis_modules/halo_finders.rst
+++ b/doc/source/analyzing/analysis_modules/halo_finders.rst
@@ -65,9 +65,6 @@
Rockstar Halo Finding
---------------------
-.. sectionauthor:: Matthew Turk <matthewturk at gmail.com>
-.. sectionauthor:: Christopher Erick Moody<cemoody at ucsc.edu>
-.. sectionauthor:: Stephen Skory <s at skory.us>
Rockstar uses an adaptive hierarchical refinement of friends-of-friends
groups in six phase-space dimensions and one time dimension, which
https://bitbucket.org/yt_analysis/yt/commits/5974c24e4185/
Changeset: 5974c24e4185
Branch: yt-3.0
User: hegan
Date: 2014-07-18 22:46:58
Summary: changed fields to not enzo fields, and made one part unitless because there are no units anywhere else in this functionality at the moment
Affected #: 1 file
diff -r b5b5a819adf0354aacda16aae95814420bc96ba7 -r 5974c24e4185655dcb9fad5bccbb44787bdd15d3 yt/analysis_modules/absorption_spectrum/absorption_spectrum.py
--- a/yt/analysis_modules/absorption_spectrum/absorption_spectrum.py
+++ b/yt/analysis_modules/absorption_spectrum/absorption_spectrum.py
@@ -118,7 +118,7 @@
if True, include line of sight velocity for shifting lines.
"""
- input_fields = ['dl', 'redshift', 'Temperature']
+ input_fields = ['dl', 'redshift', 'temperature']
field_data = {}
if use_peculiar_velocity: input_fields.append('los_velocity')
for feature in self.line_list + self.continuum_list:
@@ -201,14 +201,14 @@
delta_lambda += line['wavelength'] * (1 + field_data['redshift']) * \
field_data['los_velocity'] / speed_of_light_cgs
thermal_b = km_per_cm * np.sqrt((2 * boltzmann_constant_cgs *
- field_data['Temperature']) /
+ field_data['temperature']) /
(amu_cgs * line['atomic_mass']))
center_bins = np.digitize((delta_lambda + line['wavelength']),
self.lambda_bins)
# ratio of line width to bin width
- width_ratio = (line['wavelength'] + delta_lambda) * \
- thermal_b / speed_of_light_kms / self.bin_width
+ width_ratio = ((line['wavelength'] + delta_lambda) * \
+ thermal_b / speed_of_light_kms / self.bin_width).value
# do voigt profiles for a subset of the full spectrum
left_index = (center_bins -
https://bitbucket.org/yt_analysis/yt/commits/76d6ddf88ca0/
Changeset: 76d6ddf88ca0
Branch: yt-3.0
User: hegan
Date: 2014-07-18 22:47:33
Summary: changed fields to be properly named, and removed keyword that no longer exists
Affected #: 1 file
diff -r 5974c24e4185655dcb9fad5bccbb44787bdd15d3 -r 76d6ddf88ca0d5802e5b387dd3214684fca2498d doc/source/cookbook/fit_spectrum.py
--- a/doc/source/cookbook/fit_spectrum.py
+++ b/doc/source/cookbook/fit_spectrum.py
@@ -1,6 +1,3 @@
-### THIS RECIPE IS CURRENTLY BROKEN IN YT-3.0
-### DO NOT TRUST THIS RECIPE UNTIL THIS LINE IS REMOVED
-
import yt
from yt.analysis_modules.cosmological_observation.light_ray.api import LightRay
from yt.analysis_modules.absorption_spectrum.api import AbsorptionSpectrum
@@ -10,9 +7,9 @@
# Do *NOT* use this for science, because this is not how OVI actually behaves;
# it is just an example.
- at yt.derived_field(name='OVI_number_density', units='cm**-3')
+ at yt.derived_field(name='O_p5_number_density', units='cm**-3')
def _OVI_number_density(field, data):
- return data['HI_NumberDensity']*2.0
+ return data['H_number_density']*2.0
# Define species and associated parameters to add to continuum
@@ -23,7 +20,7 @@
# of lines, and f,gamma, and wavelength will have multiple values.
HI_parameters = {'name': 'HI',
- 'field': 'HI_NumberDensity',
+ 'field': 'H_number_density',
'f': [.4164],
'Gamma': [6.265E8],
'wavelength': [1215.67],
@@ -36,7 +33,7 @@
'init_N': 1E14}
OVI_parameters = {'name': 'OVI',
- 'field': 'OVI_number_density',
+ 'field': 'O_p5_number_density',
'f': [.1325, .06580],
'Gamma': [4.148E8, 4.076E8],
'wavelength': [1031.9261, 1037.6167],
@@ -69,7 +66,6 @@
solution_filename='lightraysolution.txt',
data_filename='lightray.h5',
fields=fields,
- get_nearest_halo=False,
get_los_velocity=True,
njobs=-1)
https://bitbucket.org/yt_analysis/yt/commits/83f4f730bfe5/
Changeset: 83f4f730bfe5
Branch: yt-3.0
User: hegan
Date: 2014-07-18 22:53:09
Summary: merged
Affected #: 1 file
diff -r 76d6ddf88ca0d5802e5b387dd3214684fca2498d -r 83f4f730bfe587895531595d53af6201399015f9 doc/source/analyzing/analysis_modules/halo_finders.rst
--- a/doc/source/analyzing/analysis_modules/halo_finders.rst
+++ b/doc/source/analyzing/analysis_modules/halo_finders.rst
@@ -65,9 +65,6 @@
Rockstar Halo Finding
---------------------
-.. sectionauthor:: Matthew Turk <matthewturk at gmail.com>
-.. sectionauthor:: Christopher Erick Moody<cemoody at ucsc.edu>
-.. sectionauthor:: Stephen Skory <s at skory.us>
Rockstar uses an adaptive hierarchical refinement of friends-of-friends
groups in six phase-space dimensions and one time dimension, which
https://bitbucket.org/yt_analysis/yt/commits/0c1c4e59fd64/
Changeset: 0c1c4e59fd64
Branch: yt-3.0
User: chummels
Date: 2014-07-18 23:05:29
Summary: Merged in hegan/yt/yt-3.0 (pull request #1031)
Halo Analysis Docs
Affected #: 11 files
diff -r 894d44f837d4d259550bb7b741091424226375b7 -r 0c1c4e59fd64dbcac22216190a615d14e3985e9d doc/source/analyzing/analysis_modules/Halo_Analysis.ipynb
--- a/doc/source/analyzing/analysis_modules/Halo_Analysis.ipynb
+++ b/doc/source/analyzing/analysis_modules/Halo_Analysis.ipynb
@@ -44,7 +44,7 @@
"tmpdir = tempfile.mkdtemp()\n",
"\n",
"# Load the data set with the full simulation information\n",
- "data_pf = load('Enzo_64/RD0006/RedshiftOutput0006')"
+ "data_ds = load('Enzo_64/RD0006/RedshiftOutput0006')"
],
"language": "python",
"metadata": {},
@@ -62,7 +62,7 @@
"collapsed": false,
"input": [
"# Load the rockstar data files\n",
- "halos_pf = load('rockstar_halos/halos_0.0.bin')"
+ "halos_ds = load('rockstar_halos/halos_0.0.bin')"
],
"language": "python",
"metadata": {},
@@ -80,7 +80,7 @@
"collapsed": false,
"input": [
"# Instantiate a catalog using those two paramter files\n",
- "hc = HaloCatalog(data_pf=data_pf, halos_pf=halos_pf, \n",
+ "hc = HaloCatalog(data_ds=data_ds, halos_ds=halos_ds, \n",
" output_dir=os.path.join(tmpdir, 'halo_catalog'))"
],
"language": "python",
diff -r 894d44f837d4d259550bb7b741091424226375b7 -r 0c1c4e59fd64dbcac22216190a615d14e3985e9d doc/source/analyzing/analysis_modules/halo_analysis.rst
--- a/doc/source/analyzing/analysis_modules/halo_analysis.rst
+++ b/doc/source/analyzing/analysis_modules/halo_analysis.rst
@@ -8,6 +8,7 @@
:maxdepth: 1
halo_catalogs
+ halo_transition
halo_finding
halo_mass_function
halo_analysis_example
diff -r 894d44f837d4d259550bb7b741091424226375b7 -r 0c1c4e59fd64dbcac22216190a615d14e3985e9d doc/source/analyzing/analysis_modules/halo_catalogs.rst
--- a/doc/source/analyzing/analysis_modules/halo_catalogs.rst
+++ b/doc/source/analyzing/analysis_modules/halo_catalogs.rst
@@ -7,9 +7,11 @@
together into a single framework. This framework is substantially
different from the limited framework included in yt-2.x and is only
backwards compatible in that output from old halo finders may be loaded.
+For a direct translation of various halo analysis tasks using yt-2.x
+to yt-3.0 please see :ref:`halo_transition`.
A catalog of halos can be created from any initial dataset given to halo
-catalog through data_pf. These halos can be found using friends-of-friends,
+catalog through data_ds. These halos can be found using friends-of-friends,
HOP, and Rockstar. The finder_method keyword dictates which halo finder to
use. The available arguments are 'fof', 'hop', and'rockstar'. For more
details on the relative differences between these halo finders see
@@ -19,32 +21,32 @@
from yt.mods import *
from yt.analysis_modules.halo_analysis.api import HaloCatalog
- data_pf = load('Enzo_64/RD0006/RedshiftOutput0006')
- hc = HaloCatalog(data_pf=data_pf, finder_method='hop')
+ data_ds = load('Enzo_64/RD0006/RedshiftOutput0006')
+ hc = HaloCatalog(data_ds=data_ds, finder_method='hop')
A halo catalog may also be created from already run rockstar outputs.
This method is not implemented for previously run friends-of-friends or
HOP finders. Even though rockstar creates one file per processor,
specifying any one file allows the full catalog to be loaded. Here we
only specify the file output by the processor with ID 0. Note that the
-argument for supplying a rockstar output is `halos_pf`, not `data_pf`.
+argument for supplying a rockstar output is `halos_ds`, not `data_ds`.
.. code-block:: python
- halos_pf = load(path+'rockstar_halos/halos_0.0.bin')
- hc = HaloCatalog(halos_pf=halos_pf)
+ halos_ds = load(path+'rockstar_halos/halos_0.0.bin')
+ hc = HaloCatalog(halos_ds=halos_ds)
Although supplying only the binary output of the rockstar halo finder
is sufficient for creating a halo catalog, it is not possible to find
any new information about the identified halos. To associate the halos
with the dataset from which they were found, supply arguments to both
-halos_pf and data_pf.
+halos_ds and data_ds.
.. code-block:: python
- halos_pf = load(path+'rockstar_halos/halos_0.0.bin')
- data_pf = load('Enzo_64/RD0006/RedshiftOutput0006')
- hc = HaloCatalog(data_pf=data_pf, halos_pf=halos_pf)
+ halos_ds = load(path+'rockstar_halos/halos_0.0.bin')
+ data_ds = load('Enzo_64/RD0006/RedshiftOutput0006')
+ hc = HaloCatalog(data_ds=data_ds, halos_ds=halos_ds)
A data container can also be supplied via keyword data_source,
associated with either dataset, to control the spatial region in
@@ -216,7 +218,7 @@
.. code-block:: python
hpf = load(path+"halo_catalogs/catalog_0046/catalog_0046.0.h5")
- hc = HaloCatalog(halos_pf=hpf,
+ hc = HaloCatalog(halos_ds=hpf,
output_dir="halo_catalogs/catalog_0046")
hc.add_callback("load_profiles", output_dir="profiles",
filename="virial_profiles")
diff -r 894d44f837d4d259550bb7b741091424226375b7 -r 0c1c4e59fd64dbcac22216190a615d14e3985e9d doc/source/analyzing/analysis_modules/halo_finders.rst
--- /dev/null
+++ b/doc/source/analyzing/analysis_modules/halo_finders.rst
@@ -0,0 +1,213 @@
+.. _halo_finding:
+
+Halo Finding
+============
+
+There are four methods of finding particle haloes in yt. The
+recommended and default method is called HOP, a method described
+in `Eisenstein and Hut (1998)
+<http://adsabs.harvard.edu/abs/1998ApJ...498..137E>`_. A basic
+friends-of-friends (e.g. `Efstathiou et al. (1985)
+<http://adsabs.harvard.edu/abs/1985ApJS...57..241E>`_) halo
+finder is also implemented. Finally Rockstar (`Behroozi et a.
+(2011) <http://adsabs.harvard.edu/abs/2011arXiv1110.4372B>`_) is
+a 6D-phase space halo finder developed by Peter Behroozi that
+excels in finding subhalos and substrcture, but does not allow
+multiple particle masses.
+
+HOP
+---
+
+The version of HOP used in yt is an upgraded version of the
+`publicly available HOP code
+<http://cmb.as.arizona.edu/~eisenste/hop/hop.html>`_. Support
+for 64-bit floats and integers has been added, as well as
+parallel analysis through spatial decomposition. HOP builds
+groups in this fashion:
+
+ 1. Estimates the local density at each particle using a
+ smoothing kernel.
+ 2. Builds chains of linked particles by 'hopping' from one
+ particle to its densest neighbor. A particle which is
+ its own densest neighbor is the end of the chain.
+ 3. All chains that share the same densest particle are
+ grouped together.
+ 4. Groups are included, linked together, or discarded
+ depending on the user-supplied over density
+ threshold parameter. The default is 160.0.
+
+Please see the `HOP method paper
+<http://adsabs.harvard.edu/abs/1998ApJ...498..137E>`_ for
+full details.
+
+Friends-of-Friends
+------------------
+
+The version of FoF in yt is based on the `publicly available
+FoF code <http://www-hpcc.astro.washington.edu/tools/fof.html>`_
+from the University of Washington. Like HOP, FoF supports
+parallel analysis through spatial decomposition. FoF is much
+simpler than HOP:
+
+ 1. From the total number of particles, and the volume of the
+ region, the average inter-particle spacing is calculated.
+ 2. Pairs of particles closer together than some fraction of
+ the average inter-particle spacing (the default is 0.2)
+ are linked together. Particles can be paired with more
+ than one other particle.
+ 3. The final groups are formed the networks of particles linked
+ together by friends, hence the name.
+
+
+.. warning:: The FoF halo finder in yt is not thoroughly tested!
+ It is probably fine to use, but you are strongly encouraged
+ to check your results against the data for errors.
+
+Rockstar Halo Finding
+---------------------
+
+Rockstar uses an adaptive hierarchical refinement of friends-of-friends
+groups in six phase-space dimensions and one time dimension, which
+allows for robust (grid-independent, shape-independent, and noise-
+resilient) tracking of substructure. The code is prepackaged with yt,
+but also `separately available <http://code.google.com/p/rockstar>`_. The lead
+developer is Peter Behroozi, and the methods are described in `Behroozi
+et al. 2011 <http://rockstar.googlecode.com/files/rockstar_ap101911.pdf>`_.
+
+.. note:: At the moment, Rockstar does not support multiple particle masses,
+ instead using a fixed particle mass. This will not affect most dark matter
+ simulations, but does make it less useful for finding halos from the stellar
+ mass. In simulations where the highest-resolution particles all have the
+ same mass (ie: zoom-in grid based simulations), one can set up a particle
+ filter to select the lowest mass particles and perform the halo finding
+ only on those.
+
+To run the Rockstar Halo finding, you must launch python with MPI and
+parallelization enabled. While Rockstar itself does not require MPI to run,
+the MPI libraries allow yt to distribute particle information across multiple
+nodes.
+
+.. warning:: At the moment, running Rockstar inside of yt on multiple compute nodes
+ connected by an Infiniband network can be problematic. Therefore, for now
+ we recommend forcing the use of the non-Infiniband network (e.g. Ethernet)
+ using this flag: ``--mca btl ^openib``.
+ For example, here is how Rockstar might be called using 24 cores:
+ ``mpirun -n 24 --mca btl ^openib python ./run_rockstar.py --parallel``.
+
+The script above configures the Halo finder, launches a server process which
+disseminates run information and coordinates writer-reader processes.
+Afterwards, it launches reader and writer tasks, filling the available MPI
+slots, which alternately read particle information and analyze for halo
+content.
+
+The RockstarHaloFinder class has these options that can be supplied to the
+halo catalog through the ``finder_kwargs`` argument:
+ * ``dm_type``, the index of the dark matter particle. Default is 1.
+ * ``outbase``, This is where the out*list files that Rockstar makes should be
+ placed. Default is 'rockstar_halos'.
+ * ``num_readers``, the number of reader tasks (which are idle most of the
+ time.) Default is 1.
+ * ``num_writers``, the number of writer tasks (which are fed particles and
+ do most of the analysis). Default is MPI_TASKS-num_readers-1.
+ If left undefined, the above options are automatically
+ configured from the number of available MPI tasks.
+ * ``force_res``, the resolution that Rockstar uses for various calculations
+ and smoothing lengths. This is in units of Mpc/h.
+ If no value is provided, this parameter is automatically set to
+ the width of the smallest grid element in the simulation from the
+ last data snapshot (i.e. the one where time has evolved the
+ longest) in the time series:
+ ``pf_last.index.get_smallest_dx() * pf_last['mpch']``.
+ * ``total_particles``, if supplied, this is a pre-calculated
+ total number of dark matter
+ particles present in the simulation. For example, this is useful
+ when analyzing a series of snapshots where the number of dark
+ matter particles should not change and this will save some disk
+ access time. If left unspecified, it will
+ be calculated automatically. Default: ``None``.
+ * ``dm_only``, if set to ``True``, it will be assumed that there are
+ only dark matter particles present in the simulation.
+ This option does not modify the halos found by Rockstar, however
+ this option can save disk access time if there are no star particles
+ (or other non-dark matter particles) in the simulation. Default: ``False``.
+
+
+Rockstar dumps halo information in a series of text (halo*list and
+out*list) and binary (halo*bin) files inside the ``outbase`` directory.
+We use the halo list classes to recover the information.
+
+Inside the ``outbase`` directory there is a text file named ``pfs.txt``
+that records the connection between pf names and the Rockstar file names.
+
+
+Parallel HOP and FOF
+--------------------
+
+Both the HOP and FoF halo finders can run in parallel using simple
+spatial decomposition. In order to run them in parallel it is helpful
+to understand how it works. Below in the first plot (i) is a simplified
+depiction of three haloes labeled 1,2 and 3:
+
+.. image:: _images/ParallelHaloFinder.png
+ :width: 500
+
+Halo 3 is twice reflected around the periodic boundary conditions.
+
+In (ii), the volume has been sub-divided into four equal subregions,
+A,B,C and D, shown with dotted lines. Notice that halo 2 is now in
+two different subregions, C and D, and that halo 3 is now in three,
+A, B and D. If the halo finder is run on these four separate subregions,
+halo 1 is be identified as a single halo, but haloes 2 and 3 are split
+up into multiple haloes, which is incorrect. The solution is to give
+each subregion padding to oversample into neighboring regions.
+
+In (iii), subregion C has oversampled into the other three regions,
+with the periodic boundary conditions taken into account, shown by
+dot-dashed lines. The other subregions oversample in a similar way.
+
+The halo finder is then run on each padded subregion independently
+and simultaneously. By oversampling like this, haloes 2 and 3 will
+both be enclosed fully in at least one subregion and identified
+completely.
+
+Haloes identified with centers of mass inside the padded part of a
+subregion are thrown out, eliminating the problem of halo duplication.
+The centers for the three haloes are shown with stars. Halo 1 will
+belong to subregion A, 2 to C and 3 to B.
+
+To run with parallel halo finding, you must supply a value for
+padding in the finder_kwargs argument. The ``padding`` parameter
+is in simulation units and defaults to 0.02. This parameter is how
+much padding is added to each of the six sides of a subregion.
+This value should be 2x-3x larger than the largest expected halo
+in the simulation. It is unlikely, of course, that the largest
+object in the simulation will be on a subregion boundary, but there
+is no way of knowing before the halo finder is run.
+
+.. code-block:: python
+
+ from yt.mods import *
+ from yt.analysis_modules.halo_analysis.api import *
+ ds = load("data0001")
+ hc= HaloCatalog(data_ds =ds,finder_method='hop'
+ finder_kwargs={'padding':0.02})
+ # --or--
+ hc= HaloCatalog(data_ds =ds,finder_method='fof'
+ finder_kwargs={'padding':0.02})
+
+
+In general, a little bit of padding goes a long way, and too much
+just slows down the analysis and doesn't improve the answer (but
+doesn't change it). It may be worth your time to run the parallel
+halo finder at a few paddings to find the right amount, especially
+if you're analyzing many similar datasets.
+
+Rockstar Installation
+=====================
+
+The Rockstar is slightly patched and modified to run as a library inside of
+yt. By default it will be built with yt using the ``install_script.sh``.
+If it wasn't installed, please make sure that the installation setting
+``INST_ROCKSTAR=1`` is defined in the ``install_script.sh`` and re-run
+the installation script.
+
diff -r 894d44f837d4d259550bb7b741091424226375b7 -r 0c1c4e59fd64dbcac22216190a615d14e3985e9d doc/source/analyzing/analysis_modules/halo_profiling.rst
--- a/doc/source/analyzing/analysis_modules/halo_profiling.rst
+++ /dev/null
@@ -1,451 +0,0 @@
-.. _halo_profiling:
-
-Halo Profiling
-==============
-.. sectionauthor:: Britton Smith <brittonsmith at gmail.com>,
- Stephen Skory <s at skory.us>
-
-The ``HaloProfiler`` provides a means of performing analysis on multiple halos
-in a parallel-safe way.
-
-The halo profiler performs three primary functions: radial profiles,
-projections, and custom analysis. See the cookbook for a recipe demonstrating
-all of these features.
-
-Configuring the Halo Profiler
------------------------------
-
-The only argument required to create a ``HaloProfiler`` object is the path
-to the dataset.
-
-.. code-block:: python
-
- from yt.analysis_modules.halo_profiler.api import *
- hp = HaloProfiler("enzo_tiny_cosmology/DD0046/DD0046")
-
-Most of the halo profiler's options are configured with additional keyword
-arguments:
-
- * **output_dir** (*str*): if specified, all output will be put into this path
- instead of in the dataset directories. Default: None.
-
- * **halos** (*str*): "multiple" for profiling more than one halo. In this mode
- halos are read in from a list or identified with a
- `halo finder <../cookbook/running_halofinder.html>`_. In "single" mode, the
- one and only halo center is identified automatically as the location of the
- peak in the density field. Default: "multiple".
-
- * **halo_list_file** (*str*): name of file containing the list of halos.
- The halo profiler will look for this file in the data directory.
- Default: "HopAnalysis.out".
-
- * **halo_list_format** (*str* or *dict*): the format of the halo list file.
- "yt_hop" for the format given by yt's halo finders. "enzo_hop" for the
- format written by enzo_hop. This keyword can also be given in the form of a
- dictionary specifying the column in which various properties can be found.
- For example, {"id": 0, "center": [1, 2, 3], "mass": 4, "radius": 5}.
- Default: "yt_hop".
-
- * **halo_finder_function** (*function*): If halos is set to multiple and the
- file given by halo_list_file does not exit, the halo finding function
- specified here will be called. Default: HaloFinder (yt_hop).
-
- * **halo_finder_args** (*tuple*): args given with call to halo finder function.
- Default: None.
-
- * **halo_finder_kwargs** (*dict*): kwargs given with call to halo finder
- function. Default: None.
-
- * **recenter** (*string* or function name): The name of a function
- that will be used to move the center of the halo for the purposes of
- analysis. See explanation and examples, below. Default: None, which
- is equivalent to the center of mass of the halo as output by the halo
- finder.
-
- * **halo_radius** (*float*): if no halo radii are provided in the halo list
- file, this parameter is used to specify the radius out to which radial
- profiles will be made. This keyword is also used when halos is set to
- single. Default: 0.1.
-
- * **radius_units** (*str*): the units of **halo_radius**.
- Default: "1" (code units).
-
- * **n_profile_bins** (*int*): the number of bins in the radial profiles.
- Default: 50.
-
- * **profile_output_dir** (*str*): the subdirectory, inside the data directory,
- in which radial profile output files will be created. The directory will be
- created if it does not exist. Default: "radial_profiles".
-
- * **projection_output_dir** (*str*): the subdirectory, inside the data
- directory, in which projection output files will be created. The directory
- will be created if it does not exist. Default: "projections".
-
- * **projection_width** (*float*): the width of halo projections.
- Default: 8.0.
-
- * **projection_width_units** (*str*): the units of projection_width.
- Default: "mpc".
-
- * **project_at_level** (*int* or "max"): the maximum refinement level to be
- included in projections. Default: "max" (maximum level within the dataset).
-
- * **velocity_center** (*list*): the method in which the halo bulk velocity is
- calculated (used for calculation of radial and tangential velocities. Valid
- options are:
- - ["bulk", "halo"] (Default): the velocity provided in the halo list
- - ["bulk", "sphere"]: the bulk velocity of the sphere centered on the halo center.
- - ["max", field]: the velocity of the cell that is the location of the maximum of the field specified.
-
- * **filter_quantities** (*list*): quantities from the original halo list
- file to be written out in the filtered list file. Default: ['id','center'].
-
- * **use_critical_density** (*bool*): if True, the definition of overdensity
- for virial quantities is calculated with respect to the critical
- density. If False, overdensity is with respect to mean matter density,
- which is lower by a factor of Omega_M. Default: False.
-
-Profiles
---------
-
-Once the halo profiler object has been instantiated, fields can be added for
-profiling with the :meth:`add_profile` method:
-
-.. code-block:: python
-
- hp.add_profile('cell_volume', weight_field=None, accumulation=True)
- hp.add_profile('TotalMassMsun', weight_field=None, accumulation=True)
- hp.add_profile('density', weight_field=None, accumulation=False)
- hp.add_profile('temperature', weight_field='cell_mass', accumulation=False)
- hp.make_profiles(njobs=-1, prefilters=["halo['mass'] > 1e13"],
- filename='VirialQuantities.h5')
-
-The :meth:`make_profiles` method will begin the profiling. Use the
-**njobs** keyword to control the number of jobs over which the
-profiling is divided. Setting to -1 results in a single processor per
-halo. Setting to 1 results in all available processors working on the
-same halo. The prefilters keyword tells the profiler to skip all halos with
-masses (as loaded from the halo finder) less than a given amount. See below
-for more information. Additional keyword arguments are:
-
- * **filename** (*str*): If set, a file will be written with all of the
- filtered halos and the quantities returned by the filter functions.
- Default: None.
-
- * **prefilters** (*list*): A single dataset can contain thousands or tens of
- thousands of halos. Significant time can be saved by not profiling halos
- that are certain to not pass any filter functions in place. Simple filters
- based on quantities provided in the initial halo list can be used to filter
- out unwanted halos using this parameter. Default: None.
-
- * **njobs** (*int*): The number of jobs over which to split the profiling.
- Set to -1 so that each halo is done by a single processor. Default: -1.
-
- * **dynamic** (*bool*): If True, distribute halos using a task queue. If
- False, distribute halos evenly over all jobs. Default: False.
-
- * **profile_format** (*str*): The file format for the radial profiles,
- 'ascii' or 'hdf5'. Default: 'ascii'.
-
-.. image:: _images/profiles.png
- :width: 500
-
-Radial profiles of Overdensity (left) and Temperature (right) for five halos.
-
-Projections
------------
-
-The process of making projections is similar to that of profiles:
-
-.. code-block:: python
-
- hp.add_projection('density', weight_field=None)
- hp.add_projection('temperature', weight_field='density')
- hp.add_projection('metallicity', weight_field='density')
- hp.make_projections(axes=[0, 1, 2], save_cube=True, save_images=True,
- halo_list="filtered", njobs=-1)
-
-If **save_cube** is set to True, the projection data
-will be written to a set of hdf5 files
-in the directory given by **projection_output_dir**.
-The keyword, **halo_list**, can be
-used to select between the full list of halos ("all"),
-the filtered list ("filtered"), or
-an entirely new list given in the form of a file name.
-See :ref:`filter_functions` for a
-discussion of filtering halos. Use the **njobs** keyword to control
-the number of jobs over which the profiling is divided. Setting to -1
-results in a single processor per halo. Setting to 1 results in all
-available processors working on the same halo. The keyword arguments are:
-
- * **axes** (*list*): A list of the axes to project along, using the usual
- 0,1,2 convention. Default=[0,1,2].
-
- * **halo_list** (*str*) {'filtered', 'all'}: Which set of halos to make
- profiles of, either ones passed by the halo filters (if enabled/added), or
- all halos. Default='filtered'.
-
- * **save_images** (*bool*): Whether or not to save images of the projections.
- Default=False.
-
- * **save_cube** (*bool*): Whether or not to save the HDF5 files of the halo
- projections. Default=True.
-
- * **njobs** (*int*): The number of jobs over which to split the projections.
- Set to -1 so that each halo is done by a single processor. Default: -1.
-
- * **dynamic** (*bool*): If True, distribute halos using a task queue. If
- False, distribute halos evenly over all jobs. Default: False.
-
-.. image:: _images/projections.png
- :width: 500
-
-Projections of Density (top) and Temperature,
-weighted by Density (bottom), in the x (left),
-y (middle), and z (right) directions for a single halo with a width of 8 Mpc.
-
-Halo Filters
-------------
-
-Filters can be added to create a refined list of
-halos based on their profiles or to avoid
-profiling halos altogether based on information
-given in the halo list file.
-
-.. _filter_functions:
-
-Filter Functions
-^^^^^^^^^^^^^^^^
-
-It is often the case that one is looking to
-identify halos with a specific set of
-properties. This can be accomplished through the creation
-of filter functions. A filter
-function can take as many args and kwargs as you like,
-as long as the first argument is a
-profile object, or at least a dictionary which contains
-the profile arrays for each field.
-Filter functions must return a list of two things.
-The first is a True or False indicating
-whether the halo passed the filter.
-The second is a dictionary containing quantities
-calculated for that halo that will be written to a
-file if the halo passes the filter.
-A sample filter function based on virial quantities can be found in
-``yt/analysis_modules/halo_profiler/halo_filters.py``.
-
-Halo filtering takes place during the call to :meth:`make_profiles`.
-The :meth:`add_halo_filter` method is used to add a filter to be used
-during the profiling:
-
-.. code-block:: python
-
- hp.add_halo_filter(HP.VirialFilter, must_be_virialized=True,
- overdensity_field='ActualOverdensity',
- virial_overdensity=200,
- virial_filters=[['TotalMassMsun','>=','1e14']],
- virial_quantities=['TotalMassMsun','RadiusMpc'],
- use_log=True)
-
-The addition above will calculate and return virial quantities,
-mass and radius, for an
-overdensity of 200. In order to pass the filter, at least one
-point in the profile must be
-above the specified overdensity and the virial mass must be at
-least 1e14 solar masses. The **use_log** keyword indicates that interpolation
-should be done in log space. If
-the VirialFilter function has been added to the filter list,
-the halo profiler will make
-sure that the fields necessary for calculating virial quantities are added.
-As many filters as desired can be added. If filters have been added,
-the next call to :meth:`make_profiles` will filter by all of
-the added filter functions:
-
-.. code-block:: python
-
- hp.make_profiles(filename="FilteredQuantities.out")
-
-If the **filename** keyword is set, a file will be written with all of the
-filtered halos and the quantities returned by the filter functions.
-
-.. note:: If the profiles have already been run, the halo profiler will read
- in the previously created output files instead of re-running the profiles.
- The halo profiler will check to make sure the output file contains all of
- the requested halo fields. If not, the profile will be made again from
- scratch.
-
-.. _halo_profiler_pre_filters:
-
-Pre-filters
-^^^^^^^^^^^
-
-A single dataset can contain thousands or tens of thousands of halos.
-Significant time can
-be saved by not profiling halos that are certain to not pass any filter
-functions in place.
-Simple filters based on quantities provided in the initial halo list
-can be used to filter
-out unwanted halos using the **prefilters** keyword:
-
-.. code-block:: python
-
- hp.make_profiles(filename="FilteredQuantities.out",
- prefilters=["halo['mass'] > 1e13"])
-
-Arguments provided with the **prefilters** keyword should be given
-as a list of strings.
-Each string in the list will be evaluated with an *eval*.
-
-.. note:: If a VirialFilter function has been added with a filter based
- on mass (as in the example above), a prefilter will be automatically
- added to filter out halos with masses greater or less than (depending
- on the conditional of the filter) a factor of ten of the specified
- virial mass.
-
-Recentering the Halo For Analysis
----------------------------------
-
-It is possible to move the center of the halo to a new point using an
-arbitrary function for making profiles.
-By default, the center is provided by the halo finder,
-which outputs the center of mass of the particles. For the purposes of
-analysis, it may be important to recenter onto a gas density maximum,
-or a temperature minimum.
-
-There are a number of built-in functions to do this, listed below.
-Each of the functions uses mass-weighted fields for the calculations
-of new center points.
-To use
-them, supply the HaloProfiler with the ``recenter`` option and
-the name of the function, as in the example below.
-
-.. code-block:: python
-
- hp = HaloProfiler("enzo_tiny_cosmology/DD0046/DD0046",
- recenter="Max_Dark_Matter_Density")
-
-Additional options are:
-
- * *Min_Dark_Matter_Density* - Recenter on the point of minimum dark matter
- density in the halo.
-
- * *Max_Dark_Matter_Density* - Recenter on the point of maximum dark matter
- density in the halo.
-
- * *CoM_Dark_Matter_Density* - Recenter on the center of mass of the dark
- matter density field. This will be very similar to what the halo finder
- provides, but not precisely similar.
-
- * *Min_Gas_Density* - Recenter on the point of minimum gas density in the
- halo.
-
- * *Max_Gas_Density* - Recenter on the point of maximum gas density in the
- halo.
-
- * *CoM_Gas_Density* - Recenter on the center of mass of the gas density field
- in the halo.
-
- * *Min_Total_Density* - Recenter on the point of minimum total (gas + dark
- matter) density in the halo.
-
- * *Max_Total_Density* - Recenter on the point of maximum total density in the
- halo.
-
- * *CoM_Total_Density* - Recenter on the center of mass for the total density
- in the halo.
-
- * *Min_Temperature* - Recenter on the point of minimum temperature in the
- halo.
-
- * *Max_Temperature* - Recenter on the point of maximum temperature in the
- halo.
-
-It is also possible to supply a user-defined function to the HaloProfiler.
-This can be used if the pre-defined functions above are not sufficient.
-The function takes a single argument, a data container for the halo,
-which is a sphere. The function returns a 3-list with the new center.
-
-In this example below, a function is used such that the halos will be
-re-centered on the point of absolute minimum temperature, that is not
-mass weighted.
-
-.. code-block:: python
-
- from yt.mods import *
-
- def find_min_temp(sphere):
- ma, mini, mx, my, mz, mg = sphere.quantities['MinLocation']('temperature')
- return [mx,my,mz]
-
- hp = HaloProfiler("enzo_tiny_cosmology/DD0046/DD0046", recenter=find_min_temp)
-
-It is possible to make more complicated functions. This example below extends
-the example above to include a distance control that prevents the center from
-being moved too far. If the recenter moves too far, ``[-1, -1, -1]`` is
-returned which will prevent the halo from being profiled.
-Any triplet of values less than the ``domain_left_edge`` will suffice.
-There will be a note made in the output (stderr) showing which halos were
-skipped.
-
-.. code-block:: python
-
- from yt.mods import *
- from yt.utilities.math_utils import periodic_dist
-
- def find_min_temp_dist(sphere):
- old = sphere.center
- ma, mini, mx, my, mz, mg = sphere.quantities['MinLocation']('temperature')
- d = sphere.pf['kpc'] * periodic_dist(old, [mx, my, mz],
- sphere.pf.domain_right_edge - sphere.pf.domain_left_edge)
- # If new center farther than 5 kpc away, don't recenter
- if d > 5.: return [-1, -1, -1]
- return [mx,my,mz]
-
- hp = HaloProfiler("enzo_tiny_cosmology/DD0046/DD0046",
- recenter=find_min_temp_dist)
-
-Custom Halo Analysis
---------------------
-
-Besides radial profiles and projections, the halo profiler has the
-ability to run custom analysis functions on each halo. Custom halo
-analysis functions take two arguments: a halo dictionary containing
-the id, center, etc; and a sphere object. The example function shown
-below creates a 2D profile of the total mass in bins of density and
-temperature for a given halo.
-
-.. code-block:: python
-
- from yt.mods import *
- from yt.data_objects.profiles import BinnedProfile2D
-
- def halo_2D_profile(halo, sphere):
- "Make a 2D profile for a halo."
- my_profile = BinnedProfile2D(sphere,
- 128, 'density', 1e-30, 1e-24, True,
- 128, 'temperature', 1e2, 1e7, True,
- end_collect=False)
- my_profile.add_fields('cell_mass', weight=None, fractional=False)
- my_filename = os.path.join(sphere.pf.fullpath, '2D_profiles',
- 'Halo_%04d.h5' % halo['id'])
- my_profile.write_out_h5(my_filename)
-
-Using the :meth:`analyze_halo_spheres` function, the halo profiler
-will create a sphere centered on each halo, and perform the analysis
-from the custom routine.
-
-.. code-block:: python
-
- hp.analyze_halo_sphere(halo_2D_profile, halo_list='filtered',
- analysis_output_dir='2D_profiles',
- njobs=-1, dynamic=False)
-
-Just like with the :meth:`make_projections` function, the keyword,
-**halo_list**, can be used to select between the full list of halos
-("all"), the filtered list ("filtered"), or an entirely new list given
-in the form of a file name. If the **analysis_output_dir** keyword is
-set, the halo profiler will make sure the desired directory exists in
-a parallel-safe manner. Use the **njobs** keyword to control the
-number of jobs over which the profiling is divided. Setting to -1
-results in a single processor per halo. Setting to 1 results in all
-available processors working on the same halo.
diff -r 894d44f837d4d259550bb7b741091424226375b7 -r 0c1c4e59fd64dbcac22216190a615d14e3985e9d doc/source/analyzing/analysis_modules/halo_transition.rst
--- /dev/null
+++ b/doc/source/analyzing/analysis_modules/halo_transition.rst
@@ -0,0 +1,106 @@
+
+Getting up to Speed with Halo Analysis in yt-3.0
+================================================
+
+If you're used to halo analysis in yt-2.x, heres a guide to
+how to update your analysis pipeline to take advantage of
+the new halo catalog infrastructure.
+
+Finding Halos
+-------------
+
+Previously, halos were found using calls to ``HaloFinder``,
+``FOFHaloFinder`` and ``RockstarHaloFinder``. Now it is
+encouraged that you find the halos upon creation of the halo catalog
+by supplying a value to the ``finder_method`` keyword when calling
+``HaloCatalog``. Currently, only halos found using rockstar or a
+previous instance of a halo catalog are able to be loaded
+using the ``halos_ds`` keyword.
+
+To pass additional arguments to the halo finders
+themselves, supply a dictionary to ``finder_kwargs`` where
+each key in the dictionary is a keyword of the halo finder
+and the corresponding value is the value to be passed for
+that keyword.
+
+Getting Halo Information
+------------------------
+All quantities that used to be present in a ``halo_list`` are
+still able to be found but are not necessarily included by default.
+Every halo will by default have the following properties:
+
+* particle_position_i (where i can be x,y,z)
+* particle_mass
+* virial_radius
+* particle_identifier
+
+If other quantities are desired, they can be included by adding
+the corresponding quantity before the catalog is created. See
+the full halo catalog documentation for further information about
+how to add these quantities and what quantities are available.
+
+You no longer have to iteratre over halos in the ``halo_list``.
+Now a halo dataset can be treated as a regular dataset and
+all quantities are available by accessing ``all_data``.
+Specifically, all quantities can be accessed as shown:
+
+.. code-block:: python
+ from yt.mods import *
+ from yt.analysis_modules.halo_analysis.api import HaloCatalog
+ data_ds = load('Enzo_64/RD0006/RedshiftOutput0006')
+ hc = HaloCatalog(data_ds=data_ds, finder_method='hop')
+ hc.create()
+ ad = hc.all_data()
+ masses = ad['particle_mass'][:]
+
+
+Prefiltering Halos
+------------------
+
+Prefiltering halos before analysis takes place is now done
+by adding a filter before the call to create. An example
+is shown below
+
+.. code-block:: python
+ from yt.mods import *
+ from yt.analysis_modules.halo_analysis.api import HaloCatalog
+ data_ds = load('Enzo_64/RD0006/RedshiftOutput0006')
+ hc = HaloCatalog(data_ds=data_ds, finder_method='hop')
+ hc.add_filter("quantity_value", "particle_mass", ">", 1e13, "Msun")
+ hc.create()
+
+Profiling Halos
+---------------
+
+The halo profiler available in yt-2.x has been removed, and
+profiling functionality is now completely contained within the
+halo catalog. A complete example of how to profile halos by
+radius using the new infrastructure is given in
+:ref:`halo_analysis_example`.
+
+Plotting Halos
+--------------
+
+Annotating halo locations onto a slice or projection works in
+the same way as in yt-2.x, but now a halo catalog must be
+passed to the annotate halo call rather than a halo list.
+
+.. code-block:: python
+ from yt.mods import *
+ from yt.analysis_modules.halo_analysis.api import HaloCatalog
+
+ data_ds = load('Enzo_64/RD0006/RedshiftOutput0006')
+ hc = HaloCatalog(data_ds=data_ds, finder_method='hop')
+ hc.create()
+
+ prj = ProjectionPlot(data_ds, 'z', 'density')
+ prj.annotate_halos(hc)
+ prj.save()
+
+Written Data
+------------
+
+Data is now written out in the form of h5 files rather than
+text files. The directory they are written out to is
+controlled by the keyword ``output_dir``. Each quantity
+is a field in the file.
diff -r 894d44f837d4d259550bb7b741091424226375b7 -r 0c1c4e59fd64dbcac22216190a615d14e3985e9d doc/source/analyzing/analysis_modules/merger_tree.rst
--- a/doc/source/analyzing/analysis_modules/merger_tree.rst
+++ b/doc/source/analyzing/analysis_modules/merger_tree.rst
@@ -2,8 +2,9 @@
Halo Merger Tree
================
-.. sectionauthor:: Stephen Skory <sskory at physics.ucsd.edu>
-.. versionadded:: 1.7
+
+.. note:: At the moment the merger tree is not yet implemented using new
+ halo catalog functionality.
The Halo Merger Tree extension is capable of building a database of halo mergers
over a set of time-ordered Enzo datasets. The fractional contribution of older
diff -r 894d44f837d4d259550bb7b741091424226375b7 -r 0c1c4e59fd64dbcac22216190a615d14e3985e9d doc/source/analyzing/analysis_modules/running_halofinder.rst
--- a/doc/source/analyzing/analysis_modules/running_halofinder.rst
+++ /dev/null
@@ -1,612 +0,0 @@
-.. _halo_finding:
-
-Halo Finding
-============
-.. sectionauthor:: Stephen Skory <sskory at physics.ucsd.edu>
-
-There are four methods of finding particle haloes in yt. The recommended and default method is called HOP, a
-method described in `Eisenstein and Hut (1998) <http://adsabs.harvard.edu/abs/1998ApJ...498..137E>`_.
-A basic friends-of-friends (e.g. `Efstathiou et al. (1985) <http://adsabs.harvard.edu/abs/1985ApJS...57..241E>`_)
-halo finder is also implemented.
-Parallel HOP (`Skory et al. (2010) <http://adsabs.harvard.edu/abs/2010ApJS..191...43S>`_)
-is a true parallelization of the HOP method can analyze massive datasets on
-hundreds of processors.
-Finally Rockstar (`Behroozi et a. (2011) <http://adsabs.harvard.edu/abs/2011arXiv1110.4372B>`_)
-is a 6D-phase space halo finder developed by Peter Behroozi
-that excels in finding subhalos and substrcture,
-but does not allow multiple particle masses.
-
-HOP
----
-
-The version of HOP used in yt is an upgraded version of the `publicly available HOP code
-<http://cmb.as.arizona.edu/~eisenste/hop/hop.html>`_. Support for 64-bit floats and integers has been
-added, as well as parallel analysis through spatial decomposition. HOP builds groups in this fashion:
-
- 1. Estimates the local density at each particle using a smoothing kernel.
- 2. Builds chains of linked particles by 'hopping' from one particle to its densest neighbor.
- A particle which is its own densest neighbor is the end of the chain.
- 3. All chains that share the same densest particle are grouped together.
- 4. Groups are included, linked together, or discarded depending on the user-supplied over density
- threshold parameter. The default is 160.0.
-
-Please see the `HOP method paper <http://adsabs.harvard.edu/abs/1998ApJ...498..137E>`_
-for full details.
-
-Friends-of-Friends
-------------------
-
-The version of FoF in yt is based on the `publicly available FoF code <http://www-hpcc.astro.washington.edu/tools/fof.html>`_ from the University of Washington. Like HOP,
-FoF supports parallel analysis through spatial decomposition. FoF is much simpler than HOP:
-
- 1. From the total number of particles, and the volume of the region, the average
- inter-particle spacing is calculated.
- 2. Pairs of particles closer together than some fraction of the average inter-particle spacing
- (the default is 0.2) are linked together. Particles can be paired with more than one other particle.
- 3. The final groups are formed the networks of particles linked together by friends, hence the name.
-
-.. warning:: The FoF halo finder in yt is not thoroughly tested! It is probably fine to use, but you
- are strongly encouraged to check your results against the data for errors.
-
-Running HaloFinder
-------------------
-
-Running HOP on a dataset is straightforward
-
-.. code-block:: python
-
- from yt.mods import *
- from yt.analysis_modules.halo_finding.api import *
- pf = load("data0001")
- halo_list = HaloFinder(pf)
-
-Running FoF is similar:
-
-.. code-block:: python
-
- from yt.mods import *
- from yt.analysis_modules.halo_finding.api import *
- pf = load("data0001")
- halo_list = FOFHaloFinder(pf)
-
-Halo Data Access
-----------------
-
-``halo_list`` is a list of ``Halo`` class objects ordered by decreasing halo mass. A ``Halo`` object
-has convenient ways to access halo data. This loop will print the location of the center of mass
-for each halo found
-
-.. code-block:: python
-
- for halo in halo_list:
- print halo.center_of_mass()
-
-All the methods are:
-
- * .center_of_mass() - the center of mass for the halo.
- * .maximum_density() - the maximum density in "HOP" units.
- * .maximum_density_location() - the location of the maximum density particle in the HOP halo.
- * .total_mass() - the mass of the halo in Msol (not Msol/h).
- * .bulk_velocity() - the velocity of the center of mass of the halo in simulation units.
- * .maximum_radius() - the distance from the center of mass to the most distant particle in the halo
- in simulation units.
- * .get_size() - the number of particles in the halo.
- * .get_sphere() - returns an an EnzoSphere object using the center of mass and maximum radius.
- * .virial_mass(virial_overdensity=float, bins=int) - Finds the virial
- mass for a halo using just the particles. This is inferior to the full
- Halo Profiler extension (:ref:`halo_profiling`), but useful nonetheless in some cases.
- Returns the mass in Msol, or -1 if the halo is not virialized.
- Defaults: ``virial_overdensity=200.0`` and ``bins=300``.
- * .virial_radius(virial_overdensity=float, bins=int) - Fins the virial
- radius of the halo using just the particles. Returns the radius in code
- units, or -1 if the halo is not virialized.
- Defaults: ``virial_overdensity=200.0`` and ``bins=300``.
-
-.. note:: For FOF the maximum density value is meaningless and is set to -1 by default. For FOF
- the maximum density location will be identical to the center of mass location.
-
-For each halo the data for the particles in the halo can be accessed like this
-
-.. code-block:: python
-
- for halo in halo_list:
- print halo["particle_index"]
- print halo["particle_position_x"] # in simulation units
-
-Halo List Data Access
----------------------
-
-These are methods that operate on the list of halo objects, rather than on the
-haloes themselves (e.g. ``halo_list.write_out()`` instead of ``halo_list[0].center_of_mass()``).
-For example, The command
-
-.. code-block:: python
-
- halo_list.write_out("HaloAnalysis.out")
-
-will output the haloes to a text file named ``HaloAnalysis.out``.
-
- * .write_out(``name``) - Writes out the center of mass, maximum density point,
- number of particles, mass, index, bulk velocity and maximum radius for all the haloes
- to a text file ``name``.
- * .write_particle_lists(``name``) - Writes the data for the particles in haloes
- (position, velocity, mass and particle index) to a HDF5 file with prefix ``name``, or one HDF5
- file per CPU when running in parallel.
- * .write_particle_lists_txt(``name``) - Writes out one text file with prefix ``name`` that gives the
- location of the particle data for haloes in the HDF5 files. This is only
- necessary when running in parallel.
- * .dump(``basename``) - Calls all of the above three functions using
- ``basename`` in each. This function is meant to be used in combination with
- loading halos off disk (:ref:`load_haloes`).
- * .nearest_neighbors_3D(haloID, num_neighbors=int, search_radius=float) -
- For a given halo ``haloID``, this finds the ``num_neighbors`` nearest (periodic)
- neighbors that are within ``search_radius`` distance from it.
- It returns a list of the neighbors distances and ID with format
- [distance,haloID]. Defaults: ``num_neighbors=7``, ``search_radius=0.2``.
- * .nearest_neighbors_2D(haloID, num_neighbors=int, search_radius=float, proj_dim={0,1,2}) -
- Similarly to the 3D search, this finds the nearest (periodic) neighbors to a halo, but
- with the positions of the haloes projected onto a 2D plane. The normal to the
- projection plane is set with ``proj_dim``, which is set to {0,1,2} for the
- {x,y,z}-axis. Defaults: ``num_neighbors=7``, ``search_radius=0.2`` and ``proj_dim=0``.
- Returns a list of neighbors in the same format as the 3D case, but the distances
- are the 2D projected distance.
-
-.. _load_haloes:
-
-Loading Haloes Off Disk
------------------------
-
-It is possible to load haloes off disk and use them as if they had just been
-located by the halo finder. This has at least two advantages. Quite obviously
-this means that if the halos are properly saved (e.g. ``haloes.dump()``, see
-above and below), halo finding does not need to be run again, saving time.
-Another benefit is loaded haloes only use as much memory as needed because the
-particle data for the haloes is loaded off disk on demand. If only a few haloes
-are being examined, a dataset that required parallel analysis for halo finding
-can be analyzed in serial, interactively.
-
-The first step is to save the haloes in a consistent manner, which is made
-simple with the ``.dump()`` function:
-
-.. code-block:: python
-
- from yt.mods import *
- from yt.analysis_modules.halo_finding.api import *
- pf = load("data0001")
- haloes = HaloFinder(pf)
- haloes.dump("basename")
-
-It is easy to load the halos using the ``LoadHaloes`` class:
-
-.. code-block:: python
-
- from yt.mods import *
- from yt.analysis_modules.halo_finding.api import *
- pf = load("data0001")
- haloes = LoadHaloes(pf, "basename")
-
-Everything that can be done with ``haloes`` in the first example should be
-possible with ``haloes`` in the second.
-
-General Parallel Halo Analysis
-------------------------------
-
-Both the HOP and FoF halo finders can run in parallel using simple spatial decomposition.
-In order to run them
-in parallel it is helpful to understand how it works.
-
-Below in the first plot (i) is a simplified depiction of three haloes labeled 1,2 and 3:
-
-.. image:: _images/ParallelHaloFinder.png
- :width: 500
-
-Halo 3 is twice reflected around the periodic boundary conditions.
-
-In (ii), the volume has been
-sub-divided into four equal subregions, A,B,C and D, shown with dotted lines. Notice that halo 2
-is now in two different subregions,
-C and D, and that halo 3 is now in three, A, B and D. If the halo finder is run on these four separate subregions,
-halo 1 is be identified as a single halo, but haloes 2 and 3 are split up into multiple haloes, which is incorrect.
-The solution is to give each subregion padding to oversample into neighboring regions.
-
-In (iii), subregion C has oversampled into the other three regions, with the periodic boundary conditions taken
-into account, shown by dot-dashed lines. The other subregions oversample in a similar way.
-
-The halo finder is then run on each padded subregion independently and simultaneously.
-By oversampling like this, haloes 2 and 3 will both be enclosed fully in at least one subregion and
-identified completely.
-
-Haloes identified with centers of mass inside the padded part of a subregion are thrown out, eliminating
-the problem of halo duplication. The centers for the three haloes are shown with stars. Halo 1 will
-belong to subregion A, 2 to C and 3 to B.
-
-Parallel HaloFinder padding
-^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-To run with parallel halo finding, there is a slight modification to the script
-
-.. code-block:: python
-
- from yt.mods import *
- from yt.analysis_modules.halo_finding.api import *
- pf = load("data0001")
- halo_list = HaloFinder(pf,padding=0.02)
- # --or--
- halo_list = FOFHaloFinder(pf,padding=0.02)
-
-The ``padding`` parameter is in simulation units and defaults to 0.02. This parameter is how much padding
-is added to each of the six sides of a subregion. This value should be 2x-3x larger than the largest
-expected halo in the simulation. It is unlikely, of course, that the largest object in the simulation
-will be on a subregion boundary, but there is no way of knowing before the halo finder is run.
-
-In general, a little bit of padding goes a long way, and too much just slows down the analysis and doesn't
-improve the answer (but doesn't change it).
-It may be worth your time to run the parallel halo finder at a few paddings to
-find the right amount, especially if you're analyzing many similar datasets.
-
-Parallel HOP
-------------
-
-**Parallel HOP** (not to be confused with HOP running in parallel as described
-above) is a wholly-new halo finder based on the HOP method.
-For extensive details and benchmarks of Parallel HOP, please see the
-pre-print version of the `method paper <http://adsabs.harvard.edu/abs/2010ApJS..191...43S>`_ at
-arXiv.org.
-While the method
-of parallelization described above can be quite effective, it has its limits.
-In particular
-for highly unbalanced datasets, where most of the particles are in a single
-part of the simulation's volume, it can become impossible to subdivide the
-volume sufficiently to fit a subvolume into a single node's memory.
-
-Parallel HOP is designed to be parallel at all levels of operation. There is
-a minimal amount of copied data across tasks. Unlike the parallel method above,
-whole haloes do not need to exist entirely in a single subvolume. In fact, a
-halo may have particles in several subvolumes simultaneously without a problem.
-
-Parallel HOP is appropriate for very large datasets where normal HOP, or
-the parallel method described above, won't work. For smaller datasets, it is
-actually faster to use the simpler methods above because the mechanisms employed for
-full parallelism are somewhat expensive.
-Whether to use Parallel HOP or not depends on the number of particles and
-the size of the largest object in the simulation.
-Because the padding of the other parallel method described above depends on
-the relative size to the box of the largest object, for smaller cosmologies
-that method may not work.
-If the largest object is quite large, the minimum padding will be a
-significant fraction of the full volume, and therefore the minimum number of
-particles per task can stay quite high.
-Below and including 256^3 particles, the other parallel methods are likely
-faster.
-However, above this and for smaller cosmologies (100 Mpc/h and smaller),
-Parallel HOP will offer better performance.
-
-The haloes identified by Parallel HOP are slightly different than normal HOP
-when run on the same dataset with the same over-density threshold.
-For a given threshold value, a few haloes have slightly different numbers of particles.
-Overall, it is not a big difference. In fact, changing the threshold value by
-a percent gives a far greater difference than the differences between HOP and
-Parallel HOP.
-
-HOP and Parallel HOP both use `KD Trees <http://en.wikipedia.org/wiki/Kd_tree>`_
-for nearest-neighbor searches.
-Parallel HOP uses the Fortran version of
-`KDTREE 2 <http://arxiv.org/abs/physics/0408067>`_ written by Matthew B. Kennel.
-The KD Tree in normal HOP calculates the distances
-between particles incorrectly by approximately one part in a million.
-KDTREE 2 is far more accurate (up to machine error),
-and this slight difference is sufficient to make perfect agreement between
-normal and Parallel HOP impossible.
-Therefore Parallel HOP is not a direct substitution for
-normal HOP, but is very similar.
-
-Running Parallel HOP
-^^^^^^^^^^^^^^^^^^^^
-
-Note: This is probably broken now that the Fortran kdtree has been removed.
-
-In the simplest form, Parallel HOP is run very similarly to the other halo finders.
-In the example below, Parallel HOP will be run on a dataset with all the default
-values. Parallel HOP can be run in serial, but as mentioned above, it is
-slower than normal HOP.
-
-.. code-block:: python
-
- from yt.mods import *
- from yt.analysis_modules.halo_finding.api import *
- pf = load("data0001")
- halo_list = parallelHF(pf)
-
-Parallel HOP has these user-set options:
-
- * ``threshold``, positive float: This is the same as the option for normal HOP. Default=160.0.
- * ``dm_only``, True/False: Whether or not to include particles other than dark
- matter when building haloes. Default=True.
- * ``resize``, True/False: Parallel HOP can load-balance the particles, such that
- each subvolume has the same number of particles.
- In general, this option is a good idea for simulations' volumes
- smaller than about 300 Mpc/h, and absolutely required for those under
- 100 Mpc/h. For larger volumes the particles are distributed evenly enough
- that this option is unnecessary. Default=True.
- * ``sample``, positive float: In order to load-balance, a random subset of the
- particle positions are read off disk, and the load-balancing routine is
- applied to them. This parameter controls what fraction of the full dataset
- population is used. Larger values result in more accurate load-balancing,
- and smaller values are faster. The value cannot be too large as the data
- for the subset of particles is communicated to one task for
- load-balancing (meaning a value
- of 1.0 will not work on very large datasets).
- Tests show that values as low as 0.0003 keep the min/max variation between
- tasks below 10%. Default = 0.03.
- * ``rearrange``, True/False: The KD Tree used by Parallel HOP can make an
- internal copy of the particle data which increases the speed of nearest
- neighbor searches by approximately 20%. The only reason to turn this option
- off is if memory is a concern. Default=True.
- * ``safety``, positive float: Unlike the simpler parallel method, Parallel
- HOP calculates the padding automatically. The padding is a
- function of the inter-particle spacing inside each subvolume. This parameter
- is multiplied to the padding distance to increase the padding volume to account for
- density variations on the boundaries of the subvolumes. Increasing this
- parameter beyond a certain point will have no effect other than consuming
- more memory and slowing down runtimes.
- Reducing it will speed up the calculation and use less memory, but
- going too far will result in degraded halo finding.
- Default=1.5, but values as low as 1.0 will probably work for many datasets.
- * ``fancy_padding``, True/False: When this is set to True, the amount of padding
- is calculated independently for each of the six faces of each subvolume. When this is
- False, the padding is the same on all six faces. There is generally no
- good reason to set this to False. Default=True.
- * ``premerge``, True/False: This option will pre-merge only the most dense
- haloes in each subvolume, before haloes are merged on the global level. In
- some cases this can speed up the runtime by a factor of two and reduce peak memory
- greatly. At worst it slows down the runtime by a small amount. It has the
- side-effect of changing the haloes slightly as a function of task count. Put in
- another way, two otherwise identical runs of Parallel HOP on a dataset will end
- up with very slightly different haloes when run with two different task counts
- with this option turned on. Not all haloes are changed between runs. This is
- due to the way merging happens in HOP - pre-merging destroys the global
- determinacy of halo merging. Default=True.
- * ``tree``, string: There are two kD-trees that may be used as part of the
- halo-finding process. The Fortran ("F") one is (presently) faster, but requires
- more memory. One based on `scipy.spatial
- <http://docs.scipy.org/doc/scipy/reference/spatial.html>`_ utilizes
- Cython ("C") and is (presently) slower, but is more memory efficient.
- Default = "F".
-
-All the same halo data can be accessed from Parallel HOP haloes as with the other halo finders.
-However, when running in parallel, there are some
-important differences in the output of a couple of these functions.
-
- * .write_particle_lists(``name``) - Because haloes may exist in more than
- one subvolume, particle data for a halo may be saved in more than one HDF5 file.
- * .write_particle_lists_txt(``name``) - If the particles for a halo is saved
- in more than one HDF5 file, there will be more than one HDF5 file listed for
- each halo in the text file.
-
-In this example script below, Parallel HOP is run on a dataset and the results
-saved to files. The summary of the haloes to ``ParallelHopAnalysis.out``, the
-particles to files named ``parts????.h5`` and the list of haloes in HDF5 files
-to ``parts.txt``.
-
-.. code-block:: python
-
- from yt.mods import *
- from yt.analysis_modules.halo_finding.api import *
- pf = load("data0001")
- halo_list = parallelHF(pf, threshold=80.0, dm_only=True, resize=False,
- rearrange=True, safety=1.5, premerge=True)
- halo_list.write_out("ParallelHopAnalysis.out")
- halo_list.write_particle_list("parts")
- halo_list.write_particle_lists_txt("parts")
-
-Halo Finding In A Subvolume
----------------------------
-
-It is possible to run any of the halo finders over a subvolume.
-This may be advantageous when only one object or region of a simulation
-is being analyzed.
-The subvolume must be a ``region`` and cannot be a
-non-rectilinear shape.
-The halo finding can be performed in parallel on a subvolume, but it may
-not be necessary depending on the size of the subvolume.
-Below is a simple example for HOP; the other halo finders use the same
-``subvolume`` keyword identically.
-
-.. code-block:: python
-
- from yt.mods import *
- from yt.analysis_modules.halo_finding.api import *
- pf = load('data0458')
- # Note that the first term below, [0.5]*3, defines the center of
- # the region and is not used. It can be any value.
- sv = pf.region([0.5]*3, [0.21, .21, .72], [.28, .28, .79])
- halos = HaloFinder(pf, subvolume = sv)
- halos.write_out("sv.out")
-
-
-Rockstar Halo Finding
-=====================
-.. sectionauthor:: Matthew Turk <matthewturk at gmail.com>
-.. sectionauthor:: Christopher Erick Moody<cemoody at ucsc.edu>
-.. sectionauthor:: Stephen Skory <s at skory.us>
-
-Rockstar uses an adaptive hierarchical refinement of friends-of-friends
-groups in six phase-space dimensions and one time dimension, which
-allows for robust (grid-independent, shape-independent, and noise-
-resilient) tracking of substructure. The code is prepackaged with yt,
-but also `separately available <http://code.google.com/p/rockstar>`_. The lead
-developer is Peter Behroozi, and the methods are described in `Behroozi
-et al. 2011 <http://rockstar.googlecode.com/files/rockstar_ap101911.pdf>`_.
-
-.. note:: At the moment, Rockstar does not support multiple particle masses,
- instead using a fixed particle mass. This will not affect most dark matter
- simulations, but does make it less useful for finding halos from the stellar
- mass. Also note that halo finding in a subvolume is not supported by
- Rockstar.
-
-To run the Rockstar Halo finding, you must launch python with MPI and
-parallelization enabled. While Rockstar itself does not require MPI to run,
-the MPI libraries allow yt to distribute particle information across multiple
-nodes.
-
-.. warning:: At the moment, running Rockstar inside of yt on multiple compute nodes
- connected by an Infiniband network can be problematic. Therefore, for now
- we recommend forcing the use of the non-Infiniband network (e.g. Ethernet)
- using this flag: ``--mca btl ^openib``.
- For example, here is how Rockstar might be called using 24 cores:
- ``mpirun -n 24 --mca btl ^openib python ./run_rockstar.py --parallel``.
-
-Designing the python script itself is straightforward:
-
-.. code-block:: python
-
- from yt.mods import *
- from yt.analysis_modules.halo_finding.rockstar.api import RockstarHaloFinder
-
- #find all of our simulation files
- files = glob.glob("Enzo_64/DD*/\*index")
- #hopefully the file name order is chronological
- files.sort()
- ts = DatasetSeries.from_filenames(files[:])
- rh = RockstarHaloFinder(ts)
- rh.run()
-
-The script above configures the Halo finder, launches a server process which
-disseminates run information and coordinates writer-reader processes.
-Afterwards, it launches reader and writer tasks, filling the available MPI
-slots, which alternately read particle information and analyze for halo
-content.
-
-The RockstarHaloFinder class has these options:
- * ``dm_type``, the index of the dark matter particle. Default is 1.
- * ``outbase``, This is where the out*list files that Rockstar makes should be
- placed. Default is 'rockstar_halos'.
- * ``num_readers``, the number of reader tasks (which are idle most of the
- time.) Default is 1.
- * ``num_writers``, the number of writer tasks (which are fed particles and
- do most of the analysis). Default is MPI_TASKS-num_readers-1.
- If left undefined, the above options are automatically
- configured from the number of available MPI tasks.
- * ``force_res``, the resolution that Rockstar uses for various calculations
- and smoothing lengths. This is in units of Mpc/h.
- If no value is provided, this parameter is automatically set to
- the width of the smallest grid element in the simulation from the
- last data snapshot (i.e. the one where time has evolved the
- longest) in the time series:
- ``pf_last.index.get_smallest_dx() * pf_last['mpch']``.
- * ``total_particles``, if supplied, this is a pre-calculated
- total number of dark matter
- particles present in the simulation. For example, this is useful
- when analyzing a series of snapshots where the number of dark
- matter particles should not change and this will save some disk
- access time. If left unspecified, it will
- be calculated automatically. Default: ``None``.
- * ``dm_only``, if set to ``True``, it will be assumed that there are
- only dark matter particles present in the simulation.
- This option does not modify the halos found by Rockstar, however
- this option can save disk access time if there are no star particles
- (or other non-dark matter particles) in the simulation. Default: ``False``.
-
-
-Output Analysis
----------------
-
-Rockstar dumps halo information in a series of text (halo*list and
-out*list) and binary (halo*bin) files inside the ``outbase`` directory.
-We use the halo list classes to recover the information.
-
-Inside the ``outbase`` directory there is a text file named ``pfs.txt``
-that records the connection between pf names and the Rockstar file names.
-
-The halo list can be automatically generated from the RockstarHaloFinder
-object by calling ``RockstarHaloFinder.halo_list()``. Alternatively, the halo
-lists can be built from the RockstarHaloList class directly
-``LoadRockstarHalos(pf,'outbase/out_0.list')``.
-
-.. code-block:: python
-
- rh = RockstarHaloFinder(pf)
- #First method of creating the halo lists:
- halo_list = rh.halo_list()
- #Alternate method of creating halo_list:
- halo_list = LoadRockstarHalos(pf, 'rockstar_halos/out_0.list')
-
-The above ``halo_list`` is very similar to any other list of halos loaded off
-disk.
-It is possible to access particle data and use the halos in a manner like any
-other halo object, and the particle data is only loaded on demand.
-Additionally, each halo object has additional information attached that is
-pulled directly from the Rockstar output:
-
-.. code-block:: python
-
- >>> halo_list[0].supp
- Out[3]:
- {'J': array([ -6.15271728e+15, -1.36593609e+17, -7.80776865e+16], dtype=float32),
- 'bulkvel': array([-132.05046082, 11.53190422, 42.16183472], dtype=float32),
- 'child_r': 2.6411054,
- 'corevel': array([-132.05046082, 11.53190422, 42.16183472], dtype=float32),
- 'desc': 0,
- 'energy': -8.106986e+21,
- 'flags': 1,
- 'id': 166,
- 'm': 1.5341227e+15,
- 'mgrav': 1.5341227e+15,
- 'min_bulkvel_err': 1821.8152,
- 'min_pos_err': 0.00049575343,
- 'min_vel_err': 1821.8152,
- 'n_core': 1958,
- 'num_child_particles': 2764,
- 'num_p': 2409,
- 'p_start': 6540,
- 'pos': array([ 0.20197368, 0.54656458, 0.11256824, -104.33285522,
- 29.02485085, 43.5154953 ], dtype=float32),
- 'r': 0.018403014,
- 'rs': 0.0026318002,
- 'rvmax': 1133.2,
- 'spin': 0.035755754,
- 'vmax': 1877.125,
- 'vrms': 1886.2648}
-
-Installation
-------------
-
-The Rockstar is slightly patched and modified to run as a library inside of
-yt. By default it will be built with yt using the ``install_script.sh``.
-If it wasn't installed, please make sure that the installation setting
-``INST_ROCKSTAR=1`` is defined in the ``install_script.sh`` and re-run
-the installation script.
-
-Rockstar Inline with Enzo
--------------------------
-
-It is possible to run Rockstar inline with Enzo. Setting up
-Enzo with inline yt is covered
-`here <http://enzo-project.org/doc/user_guide/EmbeddedPython.html>`_.
-It is not necessary to run Enzo with load balancing off to use Rockstar.
-Here is an example ``user_script.py``:
-
-.. code-block:: python
-
- from yt.mods import *
- from yt.analysis_modules.halo_finding.api import *
- from yt.config import ytcfg
- from yt.analysis_modules.halo_finding.rockstar.api import *
-
- def main():
- import enzo
- pf = EnzoDatasetInMemory()
- mine = ytcfg.getint('yt','__topcomm_parallel_rank')
- size = ytcfg.getint('yt','__topcomm_parallel_size')
-
- # Call rockstar.
- ts = DatasetSeries([pf])
- outbase = "./rockstar_halos_%04d" % pf['NumberOfPythonTopGridCalls']
- rh = RockstarHaloFinder(ts, num_readers = size,
- outbase = outbase)
- rh.run()
-
- # Load the halos off disk.
- fname = outbase + "/out_0.list"
- rhalos = LoadRockstarHalos(pf, fname)
-
diff -r 894d44f837d4d259550bb7b741091424226375b7 -r 0c1c4e59fd64dbcac22216190a615d14e3985e9d doc/source/cookbook/fit_spectrum.py
--- a/doc/source/cookbook/fit_spectrum.py
+++ b/doc/source/cookbook/fit_spectrum.py
@@ -1,6 +1,3 @@
-### THIS RECIPE IS CURRENTLY BROKEN IN YT-3.0
-### DO NOT TRUST THIS RECIPE UNTIL THIS LINE IS REMOVED
-
import yt
from yt.analysis_modules.cosmological_observation.light_ray.api import LightRay
from yt.analysis_modules.absorption_spectrum.api import AbsorptionSpectrum
@@ -10,9 +7,9 @@
# Do *NOT* use this for science, because this is not how OVI actually behaves;
# it is just an example.
- at yt.derived_field(name='OVI_number_density', units='cm**-3')
+ at yt.derived_field(name='O_p5_number_density', units='cm**-3')
def _OVI_number_density(field, data):
- return data['HI_NumberDensity']*2.0
+ return data['H_number_density']*2.0
# Define species and associated parameters to add to continuum
@@ -23,7 +20,7 @@
# of lines, and f,gamma, and wavelength will have multiple values.
HI_parameters = {'name': 'HI',
- 'field': 'HI_NumberDensity',
+ 'field': 'H_number_density',
'f': [.4164],
'Gamma': [6.265E8],
'wavelength': [1215.67],
@@ -36,7 +33,7 @@
'init_N': 1E14}
OVI_parameters = {'name': 'OVI',
- 'field': 'OVI_number_density',
+ 'field': 'O_p5_number_density',
'f': [.1325, .06580],
'Gamma': [4.148E8, 4.076E8],
'wavelength': [1031.9261, 1037.6167],
@@ -69,7 +66,6 @@
solution_filename='lightraysolution.txt',
data_filename='lightray.h5',
fields=fields,
- get_nearest_halo=False,
get_los_velocity=True,
njobs=-1)
diff -r 894d44f837d4d259550bb7b741091424226375b7 -r 0c1c4e59fd64dbcac22216190a615d14e3985e9d doc/source/cookbook/halo_profiler.py
--- a/doc/source/cookbook/halo_profiler.py
+++ b/doc/source/cookbook/halo_profiler.py
@@ -1,51 +1,44 @@
-### THIS RECIPE IS CURRENTLY BROKEN IN YT-3.0
-### DO NOT TRUST THIS RECIPE UNTIL THIS LINE IS REMOVED
+from yt.mods import *
+from yt.analysis_modules.halo_analysis.api import *
-from yt.mods import *
+# Load the data set with the full simulation information
+# and rockstar halos
+data_pf = load('Enzo_64/RD0006/RedshiftOutput0006')
+halos_pf = load('rockstar_halos/halos_0.0.bin')
-from yt.analysis_modules.halo_profiler.api import *
+# Instantiate a catalog using those two paramter files
+hc = HaloCatalog(data_pf=data_pf, halos_pf=halos_pf)
-# Define a custom function to be called on all halos.
-# The first argument is a dictionary containing the
-# halo id, center, etc.
-# The second argument is the sphere centered on the halo.
-def get_density_extrema(halo, sphere):
- my_extrema = sphere.quantities['Extrema']('density')
- mylog.info('Halo %d has density extrema: %s',
- halo['id'], my_extrema)
+# Filter out less massive halos
+hc.add_filter("quantity_value", "particle_mass", ">", 1e14, "Msun")
+# attach a sphere object to each halo whose radius extends
+# to twice the radius of the halo
+hc.add_callback("sphere", factor=2.0)
-# Instantiate HaloProfiler for this dataset.
-hp = HaloProfiler('enzo_tiny_cosmology/DD0046/DD0046',
- output_dir='.')
+# use the sphere to calculate radial profiles of gas density
+# weighted by cell volume in terms of the virial radius
+hc.add_callback("profile", x_field="radius",
+ y_fields=[("gas", "overdensity")],
+ weight_field="cell_volume",
+ accumulation=False,
+ storage="virial_quantities_profiles")
-# Add a filter to remove halos that have no profile points with overdensity
-# above 200, and with virial masses less than 1e10 solar masses.
-# Also, return the virial mass and radius to be written out to a file.
-hp.add_halo_filter(amods.halo_profiler.VirialFilter, must_be_virialized=True,
- overdensity_field='ActualOverdensity',
- virial_overdensity=200,
- virial_filters=[['TotalMassMsun', '>=', '1e10']],
- virial_quantities=['TotalMassMsun', 'RadiusMpc'])
-# Add profile fields.
-hp.add_profile('cell_volume', weight_field=None, accumulation=True)
-hp.add_profile('TotalMassMsun', weight_field=None, accumulation=True)
-hp.add_profile('density', weight_field='cell_mass', accumulation=False)
-hp.add_profile('temperature', weight_field='cell_mass', accumulation=False)
+hc.add_callback("virial_quantities", ["radius"],
+ profile_storage="virial_quantities_profiles")
+hc.add_callback('delete_attribute', 'virial_quantities_profiles')
-# Make profiles and output filtered halo list to FilteredQuantities.h5.
-hp.make_profiles(filename="FilteredQuantities.h5",
- profile_format='hdf5', njobs=-1)
+field_params = dict(virial_radius=('quantity', 'radius_200'))
+hc.add_callback('sphere', radius_field='radius_200', factor=5,
+ field_parameters=field_params)
+hc.add_callback('profile', 'virial_radius', [('gas', 'temperature')],
+ storage='virial_profiles',
+ weight_field='cell_mass',
+ accumulation=False, output_dir='profiles')
-# Add projection fields.
-hp.add_projection('density', weight_field=None)
-hp.add_projection('temperature', weight_field='density')
-hp.add_projection('metallicity', weight_field='density')
+# Save the profiles
+hc.add_callback("save_profiles", storage="virial_profiles",
+ output_dir="profiles")
-# Make projections just along the x axis using the filtered halo list.
-hp.make_projections(save_cube=False, save_images=True,
- halo_list='filtered', axes=[0], njobs=-1)
-
-# Run our custom analysis function on all halos in the filtered list.
-hp.analyze_halo_spheres(get_density_extrema, njobs=-1)
+hc.create()
diff -r 894d44f837d4d259550bb7b741091424226375b7 -r 0c1c4e59fd64dbcac22216190a615d14e3985e9d yt/analysis_modules/absorption_spectrum/absorption_spectrum.py
--- a/yt/analysis_modules/absorption_spectrum/absorption_spectrum.py
+++ b/yt/analysis_modules/absorption_spectrum/absorption_spectrum.py
@@ -118,7 +118,7 @@
if True, include line of sight velocity for shifting lines.
"""
- input_fields = ['dl', 'redshift', 'Temperature']
+ input_fields = ['dl', 'redshift', 'temperature']
field_data = {}
if use_peculiar_velocity: input_fields.append('los_velocity')
for feature in self.line_list + self.continuum_list:
@@ -201,14 +201,14 @@
delta_lambda += line['wavelength'] * (1 + field_data['redshift']) * \
field_data['los_velocity'] / speed_of_light_cgs
thermal_b = km_per_cm * np.sqrt((2 * boltzmann_constant_cgs *
- field_data['Temperature']) /
+ field_data['temperature']) /
(amu_cgs * line['atomic_mass']))
center_bins = np.digitize((delta_lambda + line['wavelength']),
self.lambda_bins)
# ratio of line width to bin width
- width_ratio = (line['wavelength'] + delta_lambda) * \
- thermal_b / speed_of_light_kms / self.bin_width
+ width_ratio = ((line['wavelength'] + delta_lambda) * \
+ thermal_b / speed_of_light_kms / self.bin_width).value
# do voigt profiles for a subset of the full spectrum
left_index = (center_bins -
Repository URL: https://bitbucket.org/yt_analysis/yt/
--
This is a commit notification from bitbucket.org. You are receiving
this because you have the service enabled, addressing the recipient of
this email.
More information about the yt-svn
mailing list