[yt-dev] Testing Intervention

[Casey W. Stark]

Hey Sam.

I was suggesting...

yt-hg/
  test/
  yt/

So that it is separate from the yt package.

- Casey


On Tue, Sep 25, 2012 at 4:05 PM, Sam Skillman <samskillman at gmail.com> wrote:

> Would root in our case be yt-hg/yt, in which case the tests would live at
> yt-hg/yt/tests/ ?
>
> Sorry if I misunderstood.
> Sam
>
>
> On Tue, Sep 25, 2012 at 5:00 PM, Anthony Scopatz <scopatz at gmail.com>wrote:
>
>> Hi Casey,
>>
>> The reason I suggested that we do it this way originally was because this
>> is the way that most other packages in the scientific python ecosystem are
>> set up.  If you look at numpy, scipy, cython, pytables, ipython, etc. they
>> all have the following file system structure:
>>
>> root/
>>   module1.py
>>   module2.py
>>   tests/
>>     test_module2.py
>>     test_module1.py
>>   subpack1/
>>     module3.py
>>     tests/
>>       test_module3.py
>>
>> I think that it is done this way to make it easy/possible for UnitTest
>> and nose to autodetect the tests.  I am not saying that we have to follow
>> this de facto standard.  However, I don't really see a reason why we
>> shouldn't either.  If you do, please let me know ;)
>>
>> Notably, matplotlib does not follow this standard.  But then again, since
>> they are testing GUI and rendering code their tests are a level of
>> complexity above what we need.
>>
>> Be Well
>> Anthony
>>
>> On Tue, Sep 25, 2012 at 5:50 PM, Casey W. Stark <caseywstark at gmail.com>wrote:
>>
>>> Hey Matt, Anthony.
>>>
>>> Looks like a good start to nosetests. One small thing -- would anyone be
>>> opposed to moving these out of the yt directory and into a root "test"
>>> directory?
>>>
>>> I can take on CIC deposition and interpolation wherever those are.
>>>
>>> - Casey
>>>
>>>
>>> On Tue, Sep 25, 2012 at 11:20 AM, Matthew Turk <matthewturk at gmail.com>wrote:
>>>
>>>> Hi all,
>>>>
>>>> As a note, Anthony just added this to the 3.0 base.  Here's a set of
>>>> tests:
>>>>
>>>>
>>>> https://bitbucket.org/yt_analysis/yt-3.0/src/52a1b2ecea94/yt/utilities/lib/tests/test_alt_ray_tracers.py
>>>>
>>>> and here's a utilities section he added:
>>>>
>>>> https://bitbucket.org/yt_analysis/yt-3.0/src/52a1b2ecea94/yt/testing.py
>>>>
>>>> -Matt
>>>>
>>>> On Mon, Sep 24, 2012 at 10:01 PM, Matthew Turk <matthewturk at gmail.com>
>>>> wrote:
>>>> > Hey Casey,
>>>> >
>>>> > On Mon, Sep 24, 2012 at 5:39 PM, Casey W. Stark <
>>>> caseywstark at gmail.com> wrote:
>>>> >> Hi Matt.
>>>> >>
>>>> >> Glad my example was useful in some way. I guess knowing exactly
>>>> which Cython
>>>> >> routines to test for what is what I meant about where to start.
>>>> >
>>>> > I completely understand -- and that's a failure of my part in the
>>>> > code.  So I think what might help is if we start looking at the
>>>> > routines that could be tested, and then I can jump in when it looks
>>>> > like they're buried.  I will also volunteer to try to refactor this
>>>> > code, once it's tested, to make it clearer what is where and why.
>>>> >
>>>> >>
>>>> >> Thanks for the tip about the stream frontend.
>>>> >
>>>> > No prob, it should be documented better.
>>>> >
>>>> > In terms of generating data, here's an example of what I was meaning:
>>>> >
>>>> > https://hub.yt-project.org/nb/ompxtg
>>>> >
>>>> > This sets up a random (but small-valued) background density and then
>>>> > applies a couple top hat spheres on top of it.  I think this could be
>>>> > a good starting point for unit testing.  I looked at the code that
>>>> > exists for flagging cells and refining them, and it's currently a bit
>>>> > too specific for this, as it was designed to take RAMSES data and
>>>> > partition it, which operates slightly differently.  I'll take a pass
>>>> > at extracting it and making it work in this case.
>>>> >
>>>> > The way I see it working is that one would set up the operators, as is
>>>> > done here, and then a progressive routine would be applied, something
>>>> > like this:
>>>> >
>>>> > while grids.flag() > 0:
>>>> >     grids.refine()
>>>> >     for operator in operators: operator.apply(grids)
>>>> > pf = grids.convert_to_dataset()
>>>> >
>>>> > The missing step in the code base is mostly just the refinement, as
>>>> > for the smoothed covering grids we have the necessary machinery to
>>>> > interpolate from one level to the next.  Adding more types of
>>>> > operators to this library would be beneficial as well.  The
>>>> > finalization step at the end would then convert the collection of
>>>> > grids into a Stream dataset.  So with only a couple lines of code we
>>>> > could in all likelihood be able to generate in-memory datasets.
>>>> >
>>>> > (This underscores in my mind why it would be awesome to have GDF in
>>>> > more places, as with this kind of machinery we've just written initial
>>>> > conditions generation, especially since we have a GDF writer
>>>> > already...)
>>>> >
>>>> > Anyway, what might be really helpful is if interested people would
>>>> > volunteer to address a few of the testable areas?  Then we can start
>>>> > pushing and identifying problem areas.  I'll volunteer to handle at
>>>> > least the data container stuff, and whatever else slips through the
>>>> > cracks; although my time will be somewhat limited in the very near
>>>> > future, I will try to make this a priority.
>>>> >
>>>> > -Matt
>>>> >
>>>> >>
>>>> >> - Casey
>>>> >>
>>>> >>
>>>> >> On Mon, Sep 24, 2012 at 1:52 PM, Matthew Turk <matthewturk at gmail.com>
>>>> wrote:
>>>> >>>
>>>> >>> Hey Casey and Anthony,
>>>> >>>
>>>> >>> On Mon, Sep 24, 2012 at 4:20 PM, Casey W. Stark <
>>>> caseywstark at gmail.com>
>>>> >>> wrote:
>>>> >>> > Hi Anthony.
>>>> >>> >
>>>> >>> > I completely agree that we should target the level of functions
>>>> actually
>>>> >>> > performing the projection rather than yt's organization. The mock
>>>> >>> > frontend
>>>> >>> > suggestion was just a hack to get there. I don't know if there's
>>>> a way
>>>> >>> > around it though...
>>>> >>> >
>>>> >>> > Here's an example of what I sorted through to get to projections:
>>>> >>> > - Load a test plotfile, check pf.h.proj to find it's source.
>>>> >>> > - Read through data_objects/hierarchy.py and
>>>> >>> > utilities/parallel_tools/parallel_analysis_interface.py to find
>>>> where
>>>> >>> > proj
>>>> >>> > is attached, can't find it.
>>>> >>> > - The proj docstring says it is a reference to AMRQuadProj. Can't
>>>> find a
>>>> >>> > class by that name.
>>>> >>> > - Search data_objects sources for "proj", find AMRProjBase.
>>>> >>> >
>>>> >>> > So it looks like the functionality is wrapped up in the
>>>> __project_level
>>>> >>> > and
>>>> >>> > _project_grid methods. I can't think of a way to test those
>>>> without
>>>> >>> > creating
>>>> >>> > an AMRProjBase, and that requires a staticoutput object.
>>>> >>>
>>>> >>> You're right, the projection stuff as *projections* is not easy to
>>>> >>> test.  But in terms of testing the underlying code, which is wrapped
>>>> >>> up in a Cython class called QuadTree, I think it could be done.  The
>>>> >>> steps you're describing are actually all part of the existing answer
>>>> >>> testing machinery, which performs a couple things and verifies that
>>>> >>> they don't change over time:
>>>> >>>
>>>> >>> 1) Project some fields from the disk
>>>> >>> 2) Project a couple derived fields
>>>> >>> 3) Project a derived field that requires spatial derivatives
>>>> >>> 4) Project the "Ones" field, which should be 1.0 everywhere.
>>>> >>>
>>>> >>> So these things are done, but it is also possible that the specific
>>>> >>> quadtree functionality could be tested, in isolation from the
>>>> >>> projection.  I think this may be oneo f the things Anthony is
>>>> talking
>>>> >>> about -- answer testing can handle the big, complex items, and by
>>>> >>> breaking down to the fundamentals we can address isolated items
>>>> from a
>>>> >>> unit testing perspective.
>>>> >>>
>>>> >>> >
>>>> >>> > So unfortunately, I think it would still come down to having a
>>>> fake
>>>> >>> > frontend. It's not ideal, but it seems like any more isolation
>>>> would
>>>> >>> > require
>>>> >>> > big rewrites to yt.
>>>> >>>
>>>> >>> One fun thing that is not usually known is that we have a fake
>>>> >>> frontend already, it just doesn't get used much.  It's called the
>>>> >>> "Stream" frontend and it was designed originally to be used in
>>>> >>> ParaView, but now gets used by the (new,
>>>> not-yet-documented/released)
>>>> >>> load_uniform_grid function as well as by Hyperion, the RT code by
>>>> Tom
>>>> >>> R.  It can set up AMR as well as static mesh.  It's not terribly
>>>> well
>>>> >>> documented, but there are examples on the wiki.
>>>> >>>
>>>> >>> One thing I've been thinking about is actually creating a couple
>>>> fake
>>>> >>> outputs, which could be defined analytically with spheres of
>>>> >>> overdensity inside them.  In principle, if we added refinement
>>>> >>> criteria, we could make this relatively complex data that was
>>>> defined
>>>> >>> with only a few lines of code, but spun up a big in-memory dataset.
>>>> >>>
>>>> >>> (This exact thing is on my list of things to do and then to output
>>>> in
>>>> >>> GDF, by the way...)
>>>> >>>
>>>> >>> That I think could come, down the road a bit.  The refinement
>>>> criteria
>>>> >>> wouldn't be too bad to implement, especially since we already have
>>>> the
>>>> >>> grid splitting routines.  I just don't think we should focus on it
>>>> at
>>>> >>> the moment.  But the uniform grid creation and loading works already
>>>> >>> -- I used it this morning.  You can do it with:
>>>> >>>
>>>> >>> from yt.frontends.stream.api import load_uniform_grid
>>>> >>> ug = load_uniform_grid({"VelocityNorm":data1, "Density":data2},
>>>> [359,
>>>> >>> 359, 359], 1.0)
>>>> >>>
>>>> >>> the list is the dimensions of the data and the value is the to-cm
>>>> >>> conversion.
>>>> >>>
>>>> >>> >
>>>> >>> > Of course, I could be missing something. Matt, can you think of a
>>>> better
>>>> >>> > way?
>>>> >>>
>>>> >>> I think for this specific example (and your damningly complex
>>>> tracing
>>>> >>> of things through the source ...) the easiest thing to do is isolate
>>>> >>> the Cython routine, which it seems I was able to do only because I
>>>> >>> wrote it and which seems quite buried in the code, and to also
>>>> provide
>>>> >>> high-level machinery for faking a frontend.
>>>> >>>
>>>> >>> -Matt
>>>> >>>
>>>> >>> >
>>>> >>> > - Casey
>>>> >>> >
>>>> >>> >
>>>> >>> > On Mon, Sep 24, 2012 at 11:02 AM, Anthony Scopatz <
>>>> scopatz at gmail.com>
>>>> >>> > wrote:
>>>> >>> >>
>>>> >>> >> Helo Casey,
>>>> >>> >>
>>>> >>> >> Sorry for taking the whole weekend to respond.
>>>> >>> >>
>>>> >>> >>>> I would like to help with this, but it's difficult to figure
>>>> out
>>>> >>> >>>> where
>>>> >>> >>>> to start.
>>>> >>> >>
>>>> >>> >>
>>>> >>> >> Not to worry. I think that any of the items listed at the bottom
>>>> of
>>>> >>> >> Matt's
>>>> >>> >> original email
>>>> >>> >> would be a great place to start.
>>>> >>> >>
>>>> >>> >>>>
>>>> >>> >>>>
>>>> >>> >>>> Say I want to test projections. I make a fake 3D density
>>>> field, maybe
>>>> >>> >>>> something as simple as np.arange(4**3).reshape((4, 4, 4)). I
>>>> write
>>>> >>> >>>> down the
>>>> >>> >>>> answer to the x-projection. Now all I need to do is call
>>>> >>> >>>> assert_allclose(yt_result, answer, rtol=1e-15), but I don't
>>>> know what
>>>> >>> >>>> pieces
>>>> >>> >>>> of low-level yt stuff to call to get to `yt_result`. Hopefully
>>>> that's
>>>> >>> >>>> clear...
>>>> >>> >>>>
>>>> >>> >>>> Maybe this comes down to creating a fake frontend we can attach
>>>> >>> >>>> fields
>>>> >>> >>>> to?
>>>> >>> >>
>>>> >>> >>
>>>> >>> >> Actually, I disagree with this strategy, as I told Matt when we
>>>> spoke
>>>> >>> >> last
>>>> >>> >> week.
>>>> >>> >> What is important is that we test the science and math parts of
>>>> the
>>>> >>> >> code
>>>> >>> >> before, if ever, dealing with the software architecture that
>>>> surrounds
>>>> >>> >> them.
>>>> >>> >>
>>>> >>> >> Let's taking your example of projections.  What we need to test
>>>> is the
>>>> >>> >> actual function
>>>> >>> >> or method which actually slogs through the projection
>>>> calculation.  In
>>>> >>> >> many cases in
>>>> >>> >> yt these functions are not directly attached to the front end
>>>> but live
>>>> >>> >> in
>>>> >>> >> analysis, visualization
>>>> >>> >> or utilities subpackages.   It is these such packages that we
>>>> should
>>>> >>> >> worry
>>>> >>> >> about testing.
>>>> >>> >> We can easily create routines to feed them sample data.
>>>> >>> >>
>>>> >>> >> On the other hand, testing or mocking things like frontends
>>>> should be a
>>>> >>> >> very low priority.
>>>> >>> >> At the end of the day what you are testing here is pulling in
>>>> data from
>>>> >>> >> disk or other
>>>> >>> >> sources.  Effectively, this is just re-testing functionality
>>>> present in
>>>> >>> >> h5py, etc.  That is not
>>>> >>> >> really our job.  Yes, in a perfect world, front ends would be
>>>> tested
>>>> >>> >> too.
>>>> >>> >> But I think that the
>>>> >>> >> priority should be placed on things like the KDTree.
>>>> >>> >>
>>>> >>> >> Be Well
>>>> >>> >> Anthony
>>>> >>> >>
>>>> >>> >>>>
>>>> >>> >>>>
>>>> >>> >>>> - Casey
>>>> >>> >>>>
>>>> >>> >>>>
>>>> >>> >>>> On Fri, Sep 21, 2012 at 2:42 PM, Matthew Turk <
>>>> matthewturk at gmail.com>
>>>> >>> >>>> wrote:
>>>> >>> >>>>>
>>>> >>> >>>>> Hi all,
>>>> >>> >>>>>
>>>> >>> >>>>> As some of you have seen (at least Stephen), I filed a ticket
>>>> this
>>>> >>> >>>>> morning about increasing testing coverage.  The other night
>>>> Anthony
>>>> >>> >>>>> and I met up in NYC and he had something of an "intervention"
>>>> about
>>>> >>> >>>>> the sufficiency of answer testing for yt; it didn't take too
>>>> much
>>>> >>> >>>>> work
>>>> >>> >>>>> on his part to convince me that we should be testing not just
>>>> >>> >>>>> against
>>>> >>> >>>>> a gold standard, but also performing unit tests.  In the past
>>>> I had
>>>> >>> >>>>> eschewed unit testing simply because the task of mocking data
>>>> was
>>>> >>> >>>>> quite tricky, and by adding tests that use smaller bits we
>>>> could
>>>> >>> >>>>> cover
>>>> >>> >>>>> unit testable areas with answer testing.
>>>> >>> >>>>>
>>>> >>> >>>>> But, this isn't really a good strategy.  Let's move to having
>>>> both.
>>>> >>> >>>>> The testing infrastructure he recommends is the
>>>> nearly-omnipresent
>>>> >>> >>>>> nose:
>>>> >>> >>>>>
>>>> >>> >>>>> http://nose.readthedocs.org/en/latest/
>>>> >>> >>>>>
>>>> >>> >>>>> The ticket to track this is here:
>>>> >>> >>>>>
>>>> >>> >>>>>
>>>> >>> >>>>>
>>>> >>> >>>>>
>>>> https://bitbucket.org/yt_analysis/yt/issue/426/increase-unit-test-coverage
>>>> >>> >>>>>
>>>> >>> >>>>> There are a couple sub-items here:
>>>> >>> >>>>>
>>>> >>> >>>>> 1) NumPy's nose test plugins provide a lot of necessary
>>>> >>> >>>>> functionality
>>>> >>> >>>>> that we have reimplemented in the answer testing utilities.
>>>>  I'd
>>>> >>> >>>>> like
>>>> >>> >>>>> to start using the numpy plugins, which include things like
>>>> >>> >>>>> conditional test execution, array comparisons, "slow" tests,
>>>> etc
>>>> >>> >>>>> etc.
>>>> >>> >>>>> 2) We can evaluate, using conditional test execution, moving
>>>> to nose
>>>> >>> >>>>> for answer testing.  But that's not on the agenda now.
>>>> >>> >>>>> 3) Writing tests for nose is super easy, and running them is
>>>> too.
>>>> >>> >>>>> Just
>>>> >>> >>>>> do:
>>>> >>> >>>>>
>>>> >>> >>>>> nosetest -w yt/
>>>> >>> >>>>>
>>>> >>> >>>>> when in your source directory.
>>>> >>> >>>>>
>>>> >>> >>>>> 4) I've written a simple sample here:
>>>> >>> >>>>>
>>>> >>> >>>>>
>>>> >>> >>>>>
>>>> >>> >>>>>
>>>> https://bitbucket.org/yt_analysis/yt-3.0/src/da10ffc17f6d/yt/utilities/tests/test_interpolators.py
>>>> >>> >>>>>
>>>> >>> >>>>> 5) I'll handle writing up some mock data that doesn't require
>>>> >>> >>>>> shipping
>>>> >>> >>>>> lots of binary files, which can then be used for checking
>>>> things
>>>> >>> >>>>> that
>>>> >>> >>>>> absolutely require hierarchies.
>>>> >>> >>>>>
>>>> >>> >>>>> --
>>>> >>> >>>>>
>>>> >>> >>>>> The way to organize tests is easy.  Inside each directory with
>>>> >>> >>>>> testable items create a new directory called "tests", and in
>>>> here
>>>> >>> >>>>> toss
>>>> >>> >>>>> some scripts.  You can stick a bunch of functions in those
>>>> scripts.
>>>> >>> >>>>>
>>>> >>> >>>>> Anyway, I'm going to start writing more of these (in the main
>>>> yt
>>>> >>> >>>>> repo,
>>>> >>> >>>>> and this change will be grafted there as well) and I'll write
>>>> back
>>>> >>> >>>>> once the data mocking is ready.  I'd like it if we started
>>>> >>> >>>>> encouraging
>>>> >>> >>>>> or even mandating simple tests (and/or answer tests) for
>>>> >>> >>>>> functionality
>>>> >>> >>>>> that gets added, but that's a discussion that should be held
>>>> >>> >>>>> separately.
>>>> >>> >>>>>
>>>> >>> >>>>> The items on the ticket:
>>>> >>> >>>>>
>>>> >>> >>>>>  * kD-tree for nearest neighbor
>>>> >>> >>>>>  * Geometric selection routines
>>>> >>> >>>>>  * Profiles
>>>> >>> >>>>>  * Projections -- underlying quadtree
>>>> >>> >>>>>  * Data object selection of data containers
>>>> >>> >>>>>  * Data object selection of points
>>>> >>> >>>>>  * Orientation class
>>>> >>> >>>>>  * Pixelization
>>>> >>> >>>>>  * Color maps
>>>> >>> >>>>>  * PNG writing
>>>> >>> >>>>>
>>>> >>> >>>>> Is anyone willing to claim any additional items that they
>>>> will help
>>>> >>> >>>>> write unit tests for?
>>>> >>> >>>>>
>>>> >>> >>>>> -Matt
>>>> >>> >>>>> _______________________________________________
>>>> >>> >>>>> yt-dev mailing list
>>>> >>> >>>>> yt-dev at lists.spacepope.org
>>>> >>> >>>>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>>>> >>> >>>>
>>>> >>> >>>>
>>>> >>> >>>>
>>>> >>> >>>> _______________________________________________
>>>> >>> >>>> yt-dev mailing list
>>>> >>> >>>> yt-dev at lists.spacepope.org
>>>> >>> >>>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>>>> >>> >>>>
>>>> >>> >>
>>>> >>> >>
>>>> >>> >> _______________________________________________
>>>> >>> >> yt-dev mailing list
>>>> >>> >> yt-dev at lists.spacepope.org
>>>> >>> >> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>>>> >>> >>
>>>> >>> >
>>>> >>> >
>>>> >>> > _______________________________________________
>>>> >>> > yt-dev mailing list
>>>> >>> > yt-dev at lists.spacepope.org
>>>> >>> > http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>>>> >>> >
>>>> >>> _______________________________________________
>>>> >>> yt-dev mailing list
>>>> >>> yt-dev at lists.spacepope.org
>>>> >>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>>>> >>
>>>> >>
>>>> >>
>>>> >> _______________________________________________
>>>> >> yt-dev mailing list
>>>> >> yt-dev at lists.spacepope.org
>>>> >> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>>>> >>
>>>> _______________________________________________
>>>> yt-dev mailing list
>>>> yt-dev at lists.spacepope.org
>>>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>>>>
>>>
>>>
>>> _______________________________________________
>>> yt-dev mailing list
>>> yt-dev at lists.spacepope.org
>>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>>>
>>>
>>
>> _______________________________________________
>> yt-dev mailing list
>> yt-dev at lists.spacepope.org
>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>>
>>
>
> _______________________________________________
> yt-dev mailing list
> yt-dev at lists.spacepope.org
> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.spacepope.org/pipermail/yt-dev-spacepope.org/attachments/20120925/aef705ac/attachment.html>