[yt-svn] commit/yt: 4 new changesets
commits-noreply at bitbucket.org
commits-noreply at bitbucket.org
Mon Sep 14 11:35:37 PDT 2015
4 new commits in yt:
https://bitbucket.org/yt_analysis/yt/commits/f9c750791fae/
Changeset: f9c750791fae
Branch: yt
User: ngoldbaum
Date: 2015-09-09 22:00:10+00:00
Summary: Updating the documentation on answer testing
Affected #: 1 file
diff -r 415b19e3578b9ad05adab2b94edded8ec57d9ede -r f9c750791faeb0c31c035065311e7f05928d48f8 doc/source/developing/testing.rst
--- a/doc/source/developing/testing.rst
+++ b/doc/source/developing/testing.rst
@@ -84,6 +84,9 @@
* :func:`~yt.testing.assert_equal` can operate on arrays.
* :func:`~yt.testing.assert_almost_equal` can operate on arrays and accepts a
relative allowable difference.
+* :func:`~yt.testing.assert_allclose_units` raises an error if two arrays are
+ not equal up to a desired absolute or relative tolerance. This wraps numpy's
+ assert_allclose to correctly verify unit consistency as well.
* :func:`~yt.testing.amrspace` provides the ability to create AMR grid
structures.
* :func:`~yt.testing.expand_keywords` provides the ability to iterate over
@@ -99,9 +102,10 @@
#. Inside that directory, create a new python file prefixed with ``test_`` and
including the name of the functionality.
#. Inside that file, create one or more routines prefixed with ``test_`` that
- accept no arguments. These should ``yield`` a set of values of the form
- ``function``, ``arguments``. For example ``yield assert_equal, 1.0, 1.0``
- would evaluate that 1.0 equaled 1.0.
+ accept no arguments. These should ``yield`` a tuple of the form
+ ``function``, ``argument_one``, ``argument_two``, etc. For example
+ ``yield assert_equal, 1.0, 1.0`` would be captured by nose as a test that
+ asserts that 1.0 is equal to 1.0.
#. Use ``fake_random_ds`` to test on datasets, and be sure to test for
several combinations of ``nproc``, so that domain decomposition can be
tested as well.
@@ -113,6 +117,47 @@
``yt/data_objects/tests/test_covering_grid.py``, which covers a great deal of
functionality.
+Debugging failing tests
+^^^^^^^^^^^^^^^^^^^^^^^
+
+When writing new tests, often one exposes bugs or simply write a test
+incorrectly, causing a test failure. The easiest way to debug issues like this
+is to drop into a debugger when nose detect an error or when a test fails. This
+can be accomplished by passing ``--pdb`` and ``--pdb-failures`` to the
+``nosetests`` executable. These options will drop into the pdb debugger whenever
+an error is raised or a failure happens, respectively. Inside the debugger you
+can interactively print out variables and go up and down the call stack to determine the context for your failure or error.
+
+In addition, one can debug more crudely using print-style debugging. To do this,
+you can add print statements to the code as normal. However, the test runner
+will capture all print output by default. To ensure that output gets printed
+to your terminal while the tests are running, pass ``-s`` to the ``nosetests``
+executable.
+
+Lastly, to quickly debug a specific failing test, it is best to only run that
+one test during your testing session. This can be accomplished by explicitly
+passing the name of the test function or class to ``nosetests``, as in the
+following example:
+
+.. code-block:: bash
+
+ $ nosetests yt.visualization.tests.test_plotwindow:TestSetWidth
+
+This nosetests invocation will onyl run the tests defined by the
+``TestSetWidth`` class.
+
+Finally, to determine which test is failing while the tests are running, it helps
+to run the tests in "verbose" mode. This can be done by passing the ``-v`` option
+to the ``nosetests`` executable.
+
+All of the above ``nosetests`` options can be combined. So, for example to run
+the ``TestSetWidth`` tests with verbose output, letting the output of print
+statements come out on the terminal prompt, and enabling pdb debugging on errors or test failures, one would do:
+
+.. code-block:: bash
+
+ $ nosetests --pdb --pdb-failures -v -s yt.visualization.tests.test_plotwindow:TestSetWidth
+
.. _answer_testing:
Answer Testing
@@ -122,8 +167,8 @@
^^^^^^^^^^^^^^^^^^^^^^^
Answer tests test **actual data**, and many operations on that data, to make
-sure that answers don't drift over time. This is how we will be testing
-frontends, as opposed to operations, in yt.
+sure that answers don't drift over time. This is how we test frontends, as
+opposed to operations, in yt.
.. _run_answer_testing:
@@ -133,20 +178,104 @@
The very first step is to make a directory and copy over the data against which
you want to test. Currently, we test:
+NMSU ART
+~~~~~~~~
+
+* ``D9p_500/10MpcBox_HartGal_csf_a0.500.d``
+
+ARTIO
+~~~~~
+
+* ``sizmbhloz-clref04SNth-rs9_a0.9011/sizmbhloz-clref04SNth-rs9_a0.9011.art``
+
+Athena
+~~~~~~
+
+* ``ShockCloud/id0/Cloud.0050.vtk``
+* ``MHDBlast/id0/Blast.0100.vtk``
+* ``RamPressureStripping/id0/rps.0062.vtk``
+* ``MHDSloshing/virgo_low_res.0054.vtk``
+
+Boxlib
+~~~~~~
+
+* ``RadAdvect/plt00000``
+* ``RadTube/plt00500``
+* ``StarParticles/plrd01000``
+
+Chombo
+~~~~~~
+
+* ``TurbBoxLowRes/data.0005.3d.hdf5``
+* ``GaussianCloud/data.0077.3d.hdf5``
+* ``IsothermalSphere/data.0000.3d.hdf5``
+* ``ZeldovichPancake/plt32.2d.hdf5``
+* ``KelvinHelmholtz/data.0004.hdf5``
+
+Enzo
+~~~~
+
* ``DD0010/moving7_0010`` (available in ``tests/`` in the yt distribution)
* ``IsolatedGalaxy/galaxy0030/galaxy0030``
+* ``enzo_tiny_cosmology/DD0046/DD0046``
+* ``enzo_cosmology_pluts/DD0046/DD0046``
+
+FITS
+~~~~
+
+* ``radio_fits/grs-50-cube.fits``
+* ``UnigridData/velocity_field_20.fits``
+
+FLASH
+~~~~~
+
* ``WindTunnel/windtunnel_4lev_hdf5_plt_cnt_0030``
* ``GasSloshingLowRes/sloshing_low_res_hdf5_plt_cnt_0300``
-* ``TurbBoxLowRes/data.0005.3d.hdf5``
-* ``GaussianCloud/data.0077.3d.hdf5``
-* ``RadAdvect/plt00000``
-* ``RadTube/plt00500``
+
+Gadget
+~~~~~~
+
+* ``IsothermalCollapse/snap_505``
+* ``IsothermalCollapse/snap_505.hdf5``
+* ``GadgetDiskGalaxy/snapshot_200.hdf5``
+
+Halo Catalog
+~~~~~~~~~~~~
+
+* ``owls_fof_halos/groups_001/group_001.0.hdf5``
+* ``owls_fof_halos/groups_008/group_008.0.hdf5``
+* ``gadget_fof_halos/groups_005/fof_subhalo_tab_005.0.hdf5``
+* ``gadget_fof_halos/groups_042/fof_subhalo_tab_042.0.hdf5``
+* ``rockstar_halos/halos_0.0.bin``
+
+MOAB
+~~~~
+
+* ``c5/c5.h5m``
+
+
+RAMSES
+~~~~~~
+
+* ``output_00080/info_00080.txt``
+
+Tipsy
+~~~~~
+
+* ``halo1e11_run1.00400/halo1e11_run1.00400``
+* ``agora_1e11.00400/agora_1e11.00400``
+* ``TipsyGalaxy/galaxy.00300``
+
+OWLS
+~~~~
+
+* ``snapshot_033/snap_033.0.hdf5``
These datasets are available at http://yt-project.org/data/.
Next, modify the file ``~/.yt/config`` to include a section ``[yt]``
with the parameter ``test_data_dir``. Set this to point to the
-directory with the test data you want to compare. Here is an example
+directory with the test data you want to test with. Here is an example
config file:
.. code-block:: none
@@ -154,47 +283,44 @@
[yt]
test_data_dir = /Users/tomservo/src/yt-data
-More data will be added over time. To run the tests, you can import the yt
-module and invoke ``yt.run_nose()`` with a new keyword argument:
+More data will be added over time. To run the answer tests, must first generate
+a set of test answers locally on a "known good" revision, then update to the
+revision you want to test, and run the tests again using the locally stored
+answers using the ``nosetests`` executable.
-.. code-block:: python
-
- import yt
- yt.run_nose(run_answer_tests=True)
-
-If you have installed yt using ``python setup.py develop`` you can also
-optionally invoke nose using the ``nosetests`` command line interface:
+Let's focus on running the answer tests for a single frontend. It's possible to
+run the answer tests for **all** the frontends, but due to the large number of
+test datasets we currently use this is not normally done.
.. code-block:: bash
$ cd $YT_HG
- $ nosetests --with-answer-testing
+ $ nosetests --with-answer-testing --local --local-dir $HOME/Documents/test --answer-store frontends.tipsy
-In either case, the current gold standard results will be downloaded from the
-rackspace cloud and compared to what is generated locally. The results from a
-nose testing session are pretty straightforward to understand, the results for
-each test are printed directly to STDOUT. If a test passes, nose prints a
-period, F if a test fails, and E if the test encounters an exception or errors
-out for some reason. If you want to also run tests for the 'big' datasets,
-then you can use the ``answer_big_data`` keyword argument:
-
-.. code-block:: python
-
- import yt
- yt.run_nose(run_answer_tests=True, answer_big_data=True)
-
-or, in the base directory of the yt mercurial repository:
+This command will create a set of local answers from the tipsy frontend tests
+and store them in ``$HOME/Documents/test`` (this does not have to be, but can be
+the same directory as your ``test_data_dir`` configuration variable). To run the
+tipsy frontend's answer tests using a different yt changeset, update to that
+changeset, recompile if necessary, and run the tests using the following
+command:
.. code-block:: bash
- $ nosetests --with-answer-testing --answer-big-data
+ $ nosetests --with-answer-testing --local --local-dir $HOME/Documents/test frontends.tipsy
-It's also possible to only run the answer tests for one frontend. For example,
-to run only the enzo answers tests, one can do,
+The results from a nose testing session are pretty straightforward to
+understand, the results for each test are printed directly to STDOUT. If a test
+passes, nose prints a period, F if a test fails, and E if the test encounters an
+exception or errors out for some reason. Explicit descriptions for each test
+are also printed if you pass ``-v`` to the ``nosetests`` executable. If you
+want to also run tests for the 'big' datasets, then you will need to pass
+``--answer-big-data`` to ``nosetests``. For example, to run the tests for the
+OWLS frontend, do the following:
.. code-block:: bash
- $ nosetests --with-answer-testing yt.frontends.enzo
+ $ nosetests --with-answer-testing --local --local-dir $HOME/Documents/test --answer-store --answer-big-data frontends.owls
+
How to Write Answer Tests
^^^^^^^^^^^^^^^^^^^^^^^^^
@@ -268,30 +394,13 @@
``@requires_ds(file_name)`` This decorate can accept the argument
``big_data`` for if this data is too big to run all the time.
- * There are ``small_patch_amr`` and ``big_patch_amr`` routines that
- you can yield from to execute a bunch of standard tests. This is where
- you should start, and then yield additional tests that stress the
- outputs in whatever ways are necessary to ensure functionality.
+ * There are ``small_patch_amr`` and ``big_patch_amr`` routines that you can
+ yield from to execute a bunch of standard tests. This is where you should
+ start, and then yield additional tests that stress the outputs in whatever
+ ways are necessary to ensure functionality. In addition we have created
+ ``sph_answer`` which is more suited for particle SPH datasets.
* **All tests should be yielded!**
If you are adding to a frontend that has a few tests already, skip the first
two steps.
-
-How to Upload Answers
-^^^^^^^^^^^^^^^^^^^^^
-
-To upload answers you can execute this command:
-
-.. code-block:: bash
-
- $ nosetests --with-answer-testing frontends/enzo/ --answer-store --answer-name=whatever
-
-The current version of the gold standard can be found in the variable
-``_latest`` inside ``yt/utilities/answer_testing/framework.py`` As of
-the time of this writing, it is ``gold007`` Note that the name of the
-suite of results is now disconnected from the dataset's name, so you
-can upload multiple outputs with the same name and not collide.
-
-To upload answers, you **must** have the package boto installed, and you
-**must** have an Amazon key provided by Matt. Contact Matt for these keys.
https://bitbucket.org/yt_analysis/yt/commits/58c558c850d3/
Changeset: 58c558c850d3
Branch: yt
User: ngoldbaum
Date: 2015-09-09 22:01:48+00:00
Summary: line wrapping two long lines
Affected #: 1 file
diff -r f9c750791faeb0c31c035065311e7f05928d48f8 -r 58c558c850d395881bb1ea2bc2d186f0939cbec0 doc/source/developing/testing.rst
--- a/doc/source/developing/testing.rst
+++ b/doc/source/developing/testing.rst
@@ -126,7 +126,8 @@
can be accomplished by passing ``--pdb`` and ``--pdb-failures`` to the
``nosetests`` executable. These options will drop into the pdb debugger whenever
an error is raised or a failure happens, respectively. Inside the debugger you
-can interactively print out variables and go up and down the call stack to determine the context for your failure or error.
+can interactively print out variables and go up and down the call stack to
+determine the context for your failure or error.
In addition, one can debug more crudely using print-style debugging. To do this,
you can add print statements to the code as normal. However, the test runner
@@ -152,7 +153,8 @@
All of the above ``nosetests`` options can be combined. So, for example to run
the ``TestSetWidth`` tests with verbose output, letting the output of print
-statements come out on the terminal prompt, and enabling pdb debugging on errors or test failures, one would do:
+statements come out on the terminal prompt, and enabling pdb debugging on errors
+or test failures, one would do:
.. code-block:: bash
https://bitbucket.org/yt_analysis/yt/commits/39d143f5cff0/
Changeset: 39d143f5cff0
Branch: yt
User: ngoldbaum
Date: 2015-09-11 02:15:27+00:00
Summary: Fixing typos and clarifying
Affected #: 1 file
diff -r 58c558c850d395881bb1ea2bc2d186f0939cbec0 -r 39d143f5cff089e77b003128cf32537b634e015b doc/source/developing/testing.rst
--- a/doc/source/developing/testing.rst
+++ b/doc/source/developing/testing.rst
@@ -120,16 +120,20 @@
Debugging failing tests
^^^^^^^^^^^^^^^^^^^^^^^
-When writing new tests, often one exposes bugs or simply write a test
-incorrectly, causing a test failure. The easiest way to debug issues like this
-is to drop into a debugger when nose detect an error or when a test fails. This
-can be accomplished by passing ``--pdb`` and ``--pdb-failures`` to the
-``nosetests`` executable. These options will drop into the pdb debugger whenever
-an error is raised or a failure happens, respectively. Inside the debugger you
-can interactively print out variables and go up and down the call stack to
-determine the context for your failure or error.
+When writing new tests, often one exposes bugs or writes a test incorrectly,
+causing an exception to be raised or a failed test. To help debug issues like
+this, ``nose`` can drop into a debugger whenever a test fails or raises an
+exception. This can be accomplished by passing ``--pdb`` and ``--pdb-failures``
+to the ``nosetests`` executable. These options will drop into the pdb debugger
+whenever an error is raised or a failure happens, respectively. Inside the
+debugger you can interactively print out variables and go up and down the call
+stack to determine the context for your failure or error.
-In addition, one can debug more crudely using print-style debugging. To do this,
+.. code-block:: bash
+
+ nosetests --pdb --pdb-failures
+
+In addition, one can debug more crudely using print statements. To do this,
you can add print statements to the code as normal. However, the test runner
will capture all print output by default. To ensure that output gets printed
to your terminal while the tests are running, pass ``-s`` to the ``nosetests``
@@ -142,9 +146,9 @@
.. code-block:: bash
- $ nosetests yt.visualization.tests.test_plotwindow:TestSetWidth
+ $ nosetests yt.visualization.tests.test_plotwindow:TestSetWidth
-This nosetests invocation will onyl run the tests defined by the
+This nosetests invocation will only run the tests defined by the
``TestSetWidth`` class.
Finally, to determine which test is failing while the tests are running, it helps
@@ -158,7 +162,7 @@
.. code-block:: bash
- $ nosetests --pdb --pdb-failures -v -s yt.visualization.tests.test_plotwindow:TestSetWidth
+ $ nosetests --pdb --pdb-failures -v -s yt.visualization.tests.test_plotwindow:TestSetWidth
.. _answer_testing:
@@ -285,14 +289,15 @@
[yt]
test_data_dir = /Users/tomservo/src/yt-data
-More data will be added over time. To run the answer tests, must first generate
-a set of test answers locally on a "known good" revision, then update to the
-revision you want to test, and run the tests again using the locally stored
-answers using the ``nosetests`` executable.
+More data will be added over time. To run the answer tests, you must first
+generate a set of test answers locally on a "known good" revision, then update
+to the revision you want to test, and run the tests again using the locally
+stored answers.
Let's focus on running the answer tests for a single frontend. It's possible to
run the answer tests for **all** the frontends, but due to the large number of
-test datasets we currently use this is not normally done.
+test datasets we currently use this is not normally done except on the yt
+project's contiguous integration server.
.. code-block:: bash
@@ -300,11 +305,11 @@
$ nosetests --with-answer-testing --local --local-dir $HOME/Documents/test --answer-store frontends.tipsy
This command will create a set of local answers from the tipsy frontend tests
-and store them in ``$HOME/Documents/test`` (this does not have to be, but can be
-the same directory as your ``test_data_dir`` configuration variable). To run the
-tipsy frontend's answer tests using a different yt changeset, update to that
-changeset, recompile if necessary, and run the tests using the following
-command:
+and store them in ``$HOME/Documents/test`` (this can but does not have to be the
+same directory as the ``test_data_dir`` configuration variable defined in your
+``.yt/config`` file). To run the tipsy frontend's answer tests using a different
+yt changeset, update to that changeset, recompile if necessary, and run the
+tests using the following command:
.. code-block:: bash
@@ -388,19 +393,19 @@
directory.
* Create a new routine that operates similarly to the routines you can see
- in Enzo's outputs.
+ in Enzo's output tests.
* This routine should test a number of different fields and data objects.
* The test routine itself should be decorated with
- ``@requires_ds(file_name)`` This decorate can accept the argument
- ``big_data`` for if this data is too big to run all the time.
+ ``@requires_ds(path_to_test_dataset)``. This decorator can accept the
+ argument ``big_data=True`` if the test is expensive.
* There are ``small_patch_amr`` and ``big_patch_amr`` routines that you can
- yield from to execute a bunch of standard tests. This is where you should
- start, and then yield additional tests that stress the outputs in whatever
- ways are necessary to ensure functionality. In addition we have created
- ``sph_answer`` which is more suited for particle SPH datasets.
+ yield from to execute a bunch of standard tests. In addition we have created
+ ``sph_answer`` which is more suited for particle SPH datasets. This is where
+ you should start, and then yield additional tests that stress the outputs in
+ whatever ways are necessary to ensure functionality.
* **All tests should be yielded!**
https://bitbucket.org/yt_analysis/yt/commits/43e11e9fc943/
Changeset: 43e11e9fc943
Branch: yt
User: xarthisius
Date: 2015-09-14 18:35:26+00:00
Summary: Merged in ngoldbaum/yt (pull request #1742)
Updating the documentation on answer testing. Closes #1083
Affected #: 1 file
diff -r 4c33cffa7ba96675a2cacaecf95a20eb15067413 -r 43e11e9fc943815fd62611e5a0626a27982e24db doc/source/developing/testing.rst
--- a/doc/source/developing/testing.rst
+++ b/doc/source/developing/testing.rst
@@ -84,6 +84,9 @@
* :func:`~yt.testing.assert_equal` can operate on arrays.
* :func:`~yt.testing.assert_almost_equal` can operate on arrays and accepts a
relative allowable difference.
+* :func:`~yt.testing.assert_allclose_units` raises an error if two arrays are
+ not equal up to a desired absolute or relative tolerance. This wraps numpy's
+ assert_allclose to correctly verify unit consistency as well.
* :func:`~yt.testing.amrspace` provides the ability to create AMR grid
structures.
* :func:`~yt.testing.expand_keywords` provides the ability to iterate over
@@ -99,9 +102,10 @@
#. Inside that directory, create a new python file prefixed with ``test_`` and
including the name of the functionality.
#. Inside that file, create one or more routines prefixed with ``test_`` that
- accept no arguments. These should ``yield`` a set of values of the form
- ``function``, ``arguments``. For example ``yield assert_equal, 1.0, 1.0``
- would evaluate that 1.0 equaled 1.0.
+ accept no arguments. These should ``yield`` a tuple of the form
+ ``function``, ``argument_one``, ``argument_two``, etc. For example
+ ``yield assert_equal, 1.0, 1.0`` would be captured by nose as a test that
+ asserts that 1.0 is equal to 1.0.
#. Use ``fake_random_ds`` to test on datasets, and be sure to test for
several combinations of ``nproc``, so that domain decomposition can be
tested as well.
@@ -113,6 +117,53 @@
``yt/data_objects/tests/test_covering_grid.py``, which covers a great deal of
functionality.
+Debugging failing tests
+^^^^^^^^^^^^^^^^^^^^^^^
+
+When writing new tests, often one exposes bugs or writes a test incorrectly,
+causing an exception to be raised or a failed test. To help debug issues like
+this, ``nose`` can drop into a debugger whenever a test fails or raises an
+exception. This can be accomplished by passing ``--pdb`` and ``--pdb-failures``
+to the ``nosetests`` executable. These options will drop into the pdb debugger
+whenever an error is raised or a failure happens, respectively. Inside the
+debugger you can interactively print out variables and go up and down the call
+stack to determine the context for your failure or error.
+
+.. code-block:: bash
+
+ nosetests --pdb --pdb-failures
+
+In addition, one can debug more crudely using print statements. To do this,
+you can add print statements to the code as normal. However, the test runner
+will capture all print output by default. To ensure that output gets printed
+to your terminal while the tests are running, pass ``-s`` to the ``nosetests``
+executable.
+
+Lastly, to quickly debug a specific failing test, it is best to only run that
+one test during your testing session. This can be accomplished by explicitly
+passing the name of the test function or class to ``nosetests``, as in the
+following example:
+
+.. code-block:: bash
+
+ $ nosetests yt.visualization.tests.test_plotwindow:TestSetWidth
+
+This nosetests invocation will only run the tests defined by the
+``TestSetWidth`` class.
+
+Finally, to determine which test is failing while the tests are running, it helps
+to run the tests in "verbose" mode. This can be done by passing the ``-v`` option
+to the ``nosetests`` executable.
+
+All of the above ``nosetests`` options can be combined. So, for example to run
+the ``TestSetWidth`` tests with verbose output, letting the output of print
+statements come out on the terminal prompt, and enabling pdb debugging on errors
+or test failures, one would do:
+
+.. code-block:: bash
+
+ $ nosetests --pdb --pdb-failures -v -s yt.visualization.tests.test_plotwindow:TestSetWidth
+
.. _answer_testing:
Answer Testing
@@ -122,8 +173,8 @@
^^^^^^^^^^^^^^^^^^^^^^^
Answer tests test **actual data**, and many operations on that data, to make
-sure that answers don't drift over time. This is how we will be testing
-frontends, as opposed to operations, in yt.
+sure that answers don't drift over time. This is how we test frontends, as
+opposed to operations, in yt.
.. _run_answer_testing:
@@ -133,20 +184,104 @@
The very first step is to make a directory and copy over the data against which
you want to test. Currently, we test:
+NMSU ART
+~~~~~~~~
+
+* ``D9p_500/10MpcBox_HartGal_csf_a0.500.d``
+
+ARTIO
+~~~~~
+
+* ``sizmbhloz-clref04SNth-rs9_a0.9011/sizmbhloz-clref04SNth-rs9_a0.9011.art``
+
+Athena
+~~~~~~
+
+* ``ShockCloud/id0/Cloud.0050.vtk``
+* ``MHDBlast/id0/Blast.0100.vtk``
+* ``RamPressureStripping/id0/rps.0062.vtk``
+* ``MHDSloshing/virgo_low_res.0054.vtk``
+
+Boxlib
+~~~~~~
+
+* ``RadAdvect/plt00000``
+* ``RadTube/plt00500``
+* ``StarParticles/plrd01000``
+
+Chombo
+~~~~~~
+
+* ``TurbBoxLowRes/data.0005.3d.hdf5``
+* ``GaussianCloud/data.0077.3d.hdf5``
+* ``IsothermalSphere/data.0000.3d.hdf5``
+* ``ZeldovichPancake/plt32.2d.hdf5``
+* ``KelvinHelmholtz/data.0004.hdf5``
+
+Enzo
+~~~~
+
* ``DD0010/moving7_0010`` (available in ``tests/`` in the yt distribution)
* ``IsolatedGalaxy/galaxy0030/galaxy0030``
+* ``enzo_tiny_cosmology/DD0046/DD0046``
+* ``enzo_cosmology_pluts/DD0046/DD0046``
+
+FITS
+~~~~
+
+* ``radio_fits/grs-50-cube.fits``
+* ``UnigridData/velocity_field_20.fits``
+
+FLASH
+~~~~~
+
* ``WindTunnel/windtunnel_4lev_hdf5_plt_cnt_0030``
* ``GasSloshingLowRes/sloshing_low_res_hdf5_plt_cnt_0300``
-* ``TurbBoxLowRes/data.0005.3d.hdf5``
-* ``GaussianCloud/data.0077.3d.hdf5``
-* ``RadAdvect/plt00000``
-* ``RadTube/plt00500``
+
+Gadget
+~~~~~~
+
+* ``IsothermalCollapse/snap_505``
+* ``IsothermalCollapse/snap_505.hdf5``
+* ``GadgetDiskGalaxy/snapshot_200.hdf5``
+
+Halo Catalog
+~~~~~~~~~~~~
+
+* ``owls_fof_halos/groups_001/group_001.0.hdf5``
+* ``owls_fof_halos/groups_008/group_008.0.hdf5``
+* ``gadget_fof_halos/groups_005/fof_subhalo_tab_005.0.hdf5``
+* ``gadget_fof_halos/groups_042/fof_subhalo_tab_042.0.hdf5``
+* ``rockstar_halos/halos_0.0.bin``
+
+MOAB
+~~~~
+
+* ``c5/c5.h5m``
+
+
+RAMSES
+~~~~~~
+
+* ``output_00080/info_00080.txt``
+
+Tipsy
+~~~~~
+
+* ``halo1e11_run1.00400/halo1e11_run1.00400``
+* ``agora_1e11.00400/agora_1e11.00400``
+* ``TipsyGalaxy/galaxy.00300``
+
+OWLS
+~~~~
+
+* ``snapshot_033/snap_033.0.hdf5``
These datasets are available at http://yt-project.org/data/.
Next, modify the file ``~/.yt/config`` to include a section ``[yt]``
with the parameter ``test_data_dir``. Set this to point to the
-directory with the test data you want to compare. Here is an example
+directory with the test data you want to test with. Here is an example
config file:
.. code-block:: none
@@ -154,47 +289,45 @@
[yt]
test_data_dir = /Users/tomservo/src/yt-data
-More data will be added over time. To run the tests, you can import the yt
-module and invoke ``yt.run_nose()`` with a new keyword argument:
+More data will be added over time. To run the answer tests, you must first
+generate a set of test answers locally on a "known good" revision, then update
+to the revision you want to test, and run the tests again using the locally
+stored answers.
-.. code-block:: python
-
- import yt
- yt.run_nose(run_answer_tests=True)
-
-If you have installed yt using ``python setup.py develop`` you can also
-optionally invoke nose using the ``nosetests`` command line interface:
+Let's focus on running the answer tests for a single frontend. It's possible to
+run the answer tests for **all** the frontends, but due to the large number of
+test datasets we currently use this is not normally done except on the yt
+project's contiguous integration server.
.. code-block:: bash
$ cd $YT_HG
- $ nosetests --with-answer-testing
+ $ nosetests --with-answer-testing --local --local-dir $HOME/Documents/test --answer-store frontends.tipsy
-In either case, the current gold standard results will be downloaded from the
-rackspace cloud and compared to what is generated locally. The results from a
-nose testing session are pretty straightforward to understand, the results for
-each test are printed directly to STDOUT. If a test passes, nose prints a
-period, F if a test fails, and E if the test encounters an exception or errors
-out for some reason. If you want to also run tests for the 'big' datasets,
-then you can use the ``answer_big_data`` keyword argument:
-
-.. code-block:: python
-
- import yt
- yt.run_nose(run_answer_tests=True, answer_big_data=True)
-
-or, in the base directory of the yt mercurial repository:
+This command will create a set of local answers from the tipsy frontend tests
+and store them in ``$HOME/Documents/test`` (this can but does not have to be the
+same directory as the ``test_data_dir`` configuration variable defined in your
+``.yt/config`` file). To run the tipsy frontend's answer tests using a different
+yt changeset, update to that changeset, recompile if necessary, and run the
+tests using the following command:
.. code-block:: bash
- $ nosetests --with-answer-testing --answer-big-data
+ $ nosetests --with-answer-testing --local --local-dir $HOME/Documents/test frontends.tipsy
-It's also possible to only run the answer tests for one frontend. For example,
-to run only the enzo answers tests, one can do,
+The results from a nose testing session are pretty straightforward to
+understand, the results for each test are printed directly to STDOUT. If a test
+passes, nose prints a period, F if a test fails, and E if the test encounters an
+exception or errors out for some reason. Explicit descriptions for each test
+are also printed if you pass ``-v`` to the ``nosetests`` executable. If you
+want to also run tests for the 'big' datasets, then you will need to pass
+``--answer-big-data`` to ``nosetests``. For example, to run the tests for the
+OWLS frontend, do the following:
.. code-block:: bash
- $ nosetests --with-answer-testing yt.frontends.enzo
+ $ nosetests --with-answer-testing --local --local-dir $HOME/Documents/test --answer-store --answer-big-data frontends.owls
+
How to Write Answer Tests
^^^^^^^^^^^^^^^^^^^^^^^^^
@@ -260,38 +393,21 @@
directory.
* Create a new routine that operates similarly to the routines you can see
- in Enzo's outputs.
+ in Enzo's output tests.
* This routine should test a number of different fields and data objects.
* The test routine itself should be decorated with
- ``@requires_ds(file_name)`` This decorate can accept the argument
- ``big_data`` for if this data is too big to run all the time.
+ ``@requires_ds(path_to_test_dataset)``. This decorator can accept the
+ argument ``big_data=True`` if the test is expensive.
- * There are ``small_patch_amr`` and ``big_patch_amr`` routines that
- you can yield from to execute a bunch of standard tests. This is where
- you should start, and then yield additional tests that stress the
- outputs in whatever ways are necessary to ensure functionality.
+ * There are ``small_patch_amr`` and ``big_patch_amr`` routines that you can
+ yield from to execute a bunch of standard tests. In addition we have created
+ ``sph_answer`` which is more suited for particle SPH datasets. This is where
+ you should start, and then yield additional tests that stress the outputs in
+ whatever ways are necessary to ensure functionality.
* **All tests should be yielded!**
If you are adding to a frontend that has a few tests already, skip the first
two steps.
-
-How to Upload Answers
-^^^^^^^^^^^^^^^^^^^^^
-
-To upload answers you can execute this command:
-
-.. code-block:: bash
-
- $ nosetests --with-answer-testing frontends/enzo/ --answer-store --answer-name=whatever
-
-The current version of the gold standard can be found in the variable
-``_latest`` inside ``yt/utilities/answer_testing/framework.py`` As of
-the time of this writing, it is ``gold007`` Note that the name of the
-suite of results is now disconnected from the dataset's name, so you
-can upload multiple outputs with the same name and not collide.
-
-To upload answers, you **must** have the package boto installed, and you
-**must** have an Amazon key provided by Matt. Contact Matt for these keys.
Repository URL: https://bitbucket.org/yt_analysis/yt/
--
This is a commit notification from bitbucket.org. You are receiving
this because you have the service enabled, addressing the recipient of
this email.
More information about the yt-svn
mailing list