Hey Matt, Anthony.<div><br></div><div>Looks like a good start to nosetests. One small thing -- would anyone be opposed to moving these out of the yt directory and into a root "test" directory?</div><div><br></div>
<div>I can take on CIC deposition and interpolation wherever those are.</div><div><br></div><div>- Casey</div><div><br><br><div class="gmail_quote">On Tue, Sep 25, 2012 at 11:20 AM, Matthew Turk <span dir="ltr"><<a href="mailto:matthewturk@gmail.com" target="_blank">matthewturk@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi all,<br>
<br>
As a note, Anthony just added this to the 3.0 base. Here's a set of tests:<br>
<br>
<a href="https://bitbucket.org/yt_analysis/yt-3.0/src/52a1b2ecea94/yt/utilities/lib/tests/test_alt_ray_tracers.py" target="_blank">https://bitbucket.org/yt_analysis/yt-3.0/src/52a1b2ecea94/yt/utilities/lib/tests/test_alt_ray_tracers.py</a><br>
<br>
and here's a utilities section he added:<br>
<br>
<a href="https://bitbucket.org/yt_analysis/yt-3.0/src/52a1b2ecea94/yt/testing.py" target="_blank">https://bitbucket.org/yt_analysis/yt-3.0/src/52a1b2ecea94/yt/testing.py</a><br>
<br>
-Matt<br>
<div class="HOEnZb"><div class="h5"><br>
On Mon, Sep 24, 2012 at 10:01 PM, Matthew Turk <<a href="mailto:matthewturk@gmail.com">matthewturk@gmail.com</a>> wrote:<br>
> Hey Casey,<br>
><br>
> On Mon, Sep 24, 2012 at 5:39 PM, Casey W. Stark <<a href="mailto:caseywstark@gmail.com">caseywstark@gmail.com</a>> wrote:<br>
>> Hi Matt.<br>
>><br>
>> Glad my example was useful in some way. I guess knowing exactly which Cython<br>
>> routines to test for what is what I meant about where to start.<br>
><br>
> I completely understand -- and that's a failure of my part in the<br>
> code. So I think what might help is if we start looking at the<br>
> routines that could be tested, and then I can jump in when it looks<br>
> like they're buried. I will also volunteer to try to refactor this<br>
> code, once it's tested, to make it clearer what is where and why.<br>
><br>
>><br>
>> Thanks for the tip about the stream frontend.<br>
><br>
> No prob, it should be documented better.<br>
><br>
> In terms of generating data, here's an example of what I was meaning:<br>
><br>
> <a href="https://hub.yt-project.org/nb/ompxtg" target="_blank">https://hub.yt-project.org/nb/ompxtg</a><br>
><br>
> This sets up a random (but small-valued) background density and then<br>
> applies a couple top hat spheres on top of it. I think this could be<br>
> a good starting point for unit testing. I looked at the code that<br>
> exists for flagging cells and refining them, and it's currently a bit<br>
> too specific for this, as it was designed to take RAMSES data and<br>
> partition it, which operates slightly differently. I'll take a pass<br>
> at extracting it and making it work in this case.<br>
><br>
> The way I see it working is that one would set up the operators, as is<br>
> done here, and then a progressive routine would be applied, something<br>
> like this:<br>
><br>
> while grids.flag() > 0:<br>
> grids.refine()<br>
> for operator in operators: operator.apply(grids)<br>
> pf = grids.convert_to_dataset()<br>
><br>
> The missing step in the code base is mostly just the refinement, as<br>
> for the smoothed covering grids we have the necessary machinery to<br>
> interpolate from one level to the next. Adding more types of<br>
> operators to this library would be beneficial as well. The<br>
> finalization step at the end would then convert the collection of<br>
> grids into a Stream dataset. So with only a couple lines of code we<br>
> could in all likelihood be able to generate in-memory datasets.<br>
><br>
> (This underscores in my mind why it would be awesome to have GDF in<br>
> more places, as with this kind of machinery we've just written initial<br>
> conditions generation, especially since we have a GDF writer<br>
> already...)<br>
><br>
> Anyway, what might be really helpful is if interested people would<br>
> volunteer to address a few of the testable areas? Then we can start<br>
> pushing and identifying problem areas. I'll volunteer to handle at<br>
> least the data container stuff, and whatever else slips through the<br>
> cracks; although my time will be somewhat limited in the very near<br>
> future, I will try to make this a priority.<br>
><br>
> -Matt<br>
><br>
>><br>
>> - Casey<br>
>><br>
>><br>
>> On Mon, Sep 24, 2012 at 1:52 PM, Matthew Turk <<a href="mailto:matthewturk@gmail.com">matthewturk@gmail.com</a>> wrote:<br>
>>><br>
>>> Hey Casey and Anthony,<br>
>>><br>
>>> On Mon, Sep 24, 2012 at 4:20 PM, Casey W. Stark <<a href="mailto:caseywstark@gmail.com">caseywstark@gmail.com</a>><br>
>>> wrote:<br>
>>> > Hi Anthony.<br>
>>> ><br>
>>> > I completely agree that we should target the level of functions actually<br>
>>> > performing the projection rather than yt's organization. The mock<br>
>>> > frontend<br>
>>> > suggestion was just a hack to get there. I don't know if there's a way<br>
>>> > around it though...<br>
>>> ><br>
>>> > Here's an example of what I sorted through to get to projections:<br>
>>> > - Load a test plotfile, check pf.h.proj to find it's source.<br>
>>> > - Read through data_objects/hierarchy.py and<br>
>>> > utilities/parallel_tools/parallel_analysis_interface.py to find where<br>
>>> > proj<br>
>>> > is attached, can't find it.<br>
>>> > - The proj docstring says it is a reference to AMRQuadProj. Can't find a<br>
>>> > class by that name.<br>
>>> > - Search data_objects sources for "proj", find AMRProjBase.<br>
>>> ><br>
>>> > So it looks like the functionality is wrapped up in the __project_level<br>
>>> > and<br>
>>> > _project_grid methods. I can't think of a way to test those without<br>
>>> > creating<br>
>>> > an AMRProjBase, and that requires a staticoutput object.<br>
>>><br>
>>> You're right, the projection stuff as *projections* is not easy to<br>
>>> test. But in terms of testing the underlying code, which is wrapped<br>
>>> up in a Cython class called QuadTree, I think it could be done. The<br>
>>> steps you're describing are actually all part of the existing answer<br>
>>> testing machinery, which performs a couple things and verifies that<br>
>>> they don't change over time:<br>
>>><br>
>>> 1) Project some fields from the disk<br>
>>> 2) Project a couple derived fields<br>
>>> 3) Project a derived field that requires spatial derivatives<br>
>>> 4) Project the "Ones" field, which should be 1.0 everywhere.<br>
>>><br>
>>> So these things are done, but it is also possible that the specific<br>
>>> quadtree functionality could be tested, in isolation from the<br>
>>> projection. I think this may be oneo f the things Anthony is talking<br>
>>> about -- answer testing can handle the big, complex items, and by<br>
>>> breaking down to the fundamentals we can address isolated items from a<br>
>>> unit testing perspective.<br>
>>><br>
>>> ><br>
>>> > So unfortunately, I think it would still come down to having a fake<br>
>>> > frontend. It's not ideal, but it seems like any more isolation would<br>
>>> > require<br>
>>> > big rewrites to yt.<br>
>>><br>
>>> One fun thing that is not usually known is that we have a fake<br>
>>> frontend already, it just doesn't get used much. It's called the<br>
>>> "Stream" frontend and it was designed originally to be used in<br>
>>> ParaView, but now gets used by the (new, not-yet-documented/released)<br>
>>> load_uniform_grid function as well as by Hyperion, the RT code by Tom<br>
>>> R. It can set up AMR as well as static mesh. It's not terribly well<br>
>>> documented, but there are examples on the wiki.<br>
>>><br>
>>> One thing I've been thinking about is actually creating a couple fake<br>
>>> outputs, which could be defined analytically with spheres of<br>
>>> overdensity inside them. In principle, if we added refinement<br>
>>> criteria, we could make this relatively complex data that was defined<br>
>>> with only a few lines of code, but spun up a big in-memory dataset.<br>
>>><br>
>>> (This exact thing is on my list of things to do and then to output in<br>
>>> GDF, by the way...)<br>
>>><br>
>>> That I think could come, down the road a bit. The refinement criteria<br>
>>> wouldn't be too bad to implement, especially since we already have the<br>
>>> grid splitting routines. I just don't think we should focus on it at<br>
>>> the moment. But the uniform grid creation and loading works already<br>
>>> -- I used it this morning. You can do it with:<br>
>>><br>
>>> from yt.frontends.stream.api import load_uniform_grid<br>
>>> ug = load_uniform_grid({"VelocityNorm":data1, "Density":data2}, [359,<br>
>>> 359, 359], 1.0)<br>
>>><br>
>>> the list is the dimensions of the data and the value is the to-cm<br>
>>> conversion.<br>
>>><br>
>>> ><br>
>>> > Of course, I could be missing something. Matt, can you think of a better<br>
>>> > way?<br>
>>><br>
>>> I think for this specific example (and your damningly complex tracing<br>
>>> of things through the source ...) the easiest thing to do is isolate<br>
>>> the Cython routine, which it seems I was able to do only because I<br>
>>> wrote it and which seems quite buried in the code, and to also provide<br>
>>> high-level machinery for faking a frontend.<br>
>>><br>
>>> -Matt<br>
>>><br>
>>> ><br>
>>> > - Casey<br>
>>> ><br>
>>> ><br>
>>> > On Mon, Sep 24, 2012 at 11:02 AM, Anthony Scopatz <<a href="mailto:scopatz@gmail.com">scopatz@gmail.com</a>><br>
>>> > wrote:<br>
>>> >><br>
>>> >> Helo Casey,<br>
>>> >><br>
>>> >> Sorry for taking the whole weekend to respond.<br>
>>> >><br>
>>> >>>> I would like to help with this, but it's difficult to figure out<br>
>>> >>>> where<br>
>>> >>>> to start.<br>
>>> >><br>
>>> >><br>
>>> >> Not to worry. I think that any of the items listed at the bottom of<br>
>>> >> Matt's<br>
>>> >> original email<br>
>>> >> would be a great place to start.<br>
>>> >><br>
>>> >>>><br>
>>> >>>><br>
>>> >>>> Say I want to test projections. I make a fake 3D density field, maybe<br>
>>> >>>> something as simple as np.arange(4**3).reshape((4, 4, 4)). I write<br>
>>> >>>> down the<br>
>>> >>>> answer to the x-projection. Now all I need to do is call<br>
>>> >>>> assert_allclose(yt_result, answer, rtol=1e-15), but I don't know what<br>
>>> >>>> pieces<br>
>>> >>>> of low-level yt stuff to call to get to `yt_result`. Hopefully that's<br>
>>> >>>> clear...<br>
>>> >>>><br>
>>> >>>> Maybe this comes down to creating a fake frontend we can attach<br>
>>> >>>> fields<br>
>>> >>>> to?<br>
>>> >><br>
>>> >><br>
>>> >> Actually, I disagree with this strategy, as I told Matt when we spoke<br>
>>> >> last<br>
>>> >> week.<br>
>>> >> What is important is that we test the science and math parts of the<br>
>>> >> code<br>
>>> >> before, if ever, dealing with the software architecture that surrounds<br>
>>> >> them.<br>
>>> >><br>
>>> >> Let's taking your example of projections. What we need to test is the<br>
>>> >> actual function<br>
>>> >> or method which actually slogs through the projection calculation. In<br>
>>> >> many cases in<br>
>>> >> yt these functions are not directly attached to the front end but live<br>
>>> >> in<br>
>>> >> analysis, visualization<br>
>>> >> or utilities subpackages. It is these such packages that we should<br>
>>> >> worry<br>
>>> >> about testing.<br>
>>> >> We can easily create routines to feed them sample data.<br>
>>> >><br>
>>> >> On the other hand, testing or mocking things like frontends should be a<br>
>>> >> very low priority.<br>
>>> >> At the end of the day what you are testing here is pulling in data from<br>
>>> >> disk or other<br>
>>> >> sources. Effectively, this is just re-testing functionality present in<br>
>>> >> h5py, etc. That is not<br>
>>> >> really our job. Yes, in a perfect world, front ends would be tested<br>
>>> >> too.<br>
>>> >> But I think that the<br>
>>> >> priority should be placed on things like the KDTree.<br>
>>> >><br>
>>> >> Be Well<br>
>>> >> Anthony<br>
>>> >><br>
>>> >>>><br>
>>> >>>><br>
>>> >>>> - Casey<br>
>>> >>>><br>
>>> >>>><br>
>>> >>>> On Fri, Sep 21, 2012 at 2:42 PM, Matthew Turk <<a href="mailto:matthewturk@gmail.com">matthewturk@gmail.com</a>><br>
>>> >>>> wrote:<br>
>>> >>>>><br>
>>> >>>>> Hi all,<br>
>>> >>>>><br>
>>> >>>>> As some of you have seen (at least Stephen), I filed a ticket this<br>
>>> >>>>> morning about increasing testing coverage. The other night Anthony<br>
>>> >>>>> and I met up in NYC and he had something of an "intervention" about<br>
>>> >>>>> the sufficiency of answer testing for yt; it didn't take too much<br>
>>> >>>>> work<br>
>>> >>>>> on his part to convince me that we should be testing not just<br>
>>> >>>>> against<br>
>>> >>>>> a gold standard, but also performing unit tests. In the past I had<br>
>>> >>>>> eschewed unit testing simply because the task of mocking data was<br>
>>> >>>>> quite tricky, and by adding tests that use smaller bits we could<br>
>>> >>>>> cover<br>
>>> >>>>> unit testable areas with answer testing.<br>
>>> >>>>><br>
>>> >>>>> But, this isn't really a good strategy. Let's move to having both.<br>
>>> >>>>> The testing infrastructure he recommends is the nearly-omnipresent<br>
>>> >>>>> nose:<br>
>>> >>>>><br>
>>> >>>>> <a href="http://nose.readthedocs.org/en/latest/" target="_blank">http://nose.readthedocs.org/en/latest/</a><br>
>>> >>>>><br>
>>> >>>>> The ticket to track this is here:<br>
>>> >>>>><br>
>>> >>>>><br>
>>> >>>>><br>
>>> >>>>> <a href="https://bitbucket.org/yt_analysis/yt/issue/426/increase-unit-test-coverage" target="_blank">https://bitbucket.org/yt_analysis/yt/issue/426/increase-unit-test-coverage</a><br>
>>> >>>>><br>
>>> >>>>> There are a couple sub-items here:<br>
>>> >>>>><br>
>>> >>>>> 1) NumPy's nose test plugins provide a lot of necessary<br>
>>> >>>>> functionality<br>
>>> >>>>> that we have reimplemented in the answer testing utilities. I'd<br>
>>> >>>>> like<br>
>>> >>>>> to start using the numpy plugins, which include things like<br>
>>> >>>>> conditional test execution, array comparisons, "slow" tests, etc<br>
>>> >>>>> etc.<br>
>>> >>>>> 2) We can evaluate, using conditional test execution, moving to nose<br>
>>> >>>>> for answer testing. But that's not on the agenda now.<br>
>>> >>>>> 3) Writing tests for nose is super easy, and running them is too.<br>
>>> >>>>> Just<br>
>>> >>>>> do:<br>
>>> >>>>><br>
>>> >>>>> nosetest -w yt/<br>
>>> >>>>><br>
>>> >>>>> when in your source directory.<br>
>>> >>>>><br>
>>> >>>>> 4) I've written a simple sample here:<br>
>>> >>>>><br>
>>> >>>>><br>
>>> >>>>><br>
>>> >>>>> <a href="https://bitbucket.org/yt_analysis/yt-3.0/src/da10ffc17f6d/yt/utilities/tests/test_interpolators.py" target="_blank">https://bitbucket.org/yt_analysis/yt-3.0/src/da10ffc17f6d/yt/utilities/tests/test_interpolators.py</a><br>
>>> >>>>><br>
>>> >>>>> 5) I'll handle writing up some mock data that doesn't require<br>
>>> >>>>> shipping<br>
>>> >>>>> lots of binary files, which can then be used for checking things<br>
>>> >>>>> that<br>
>>> >>>>> absolutely require hierarchies.<br>
>>> >>>>><br>
>>> >>>>> --<br>
>>> >>>>><br>
>>> >>>>> The way to organize tests is easy. Inside each directory with<br>
>>> >>>>> testable items create a new directory called "tests", and in here<br>
>>> >>>>> toss<br>
>>> >>>>> some scripts. You can stick a bunch of functions in those scripts.<br>
>>> >>>>><br>
>>> >>>>> Anyway, I'm going to start writing more of these (in the main yt<br>
>>> >>>>> repo,<br>
>>> >>>>> and this change will be grafted there as well) and I'll write back<br>
>>> >>>>> once the data mocking is ready. I'd like it if we started<br>
>>> >>>>> encouraging<br>
>>> >>>>> or even mandating simple tests (and/or answer tests) for<br>
>>> >>>>> functionality<br>
>>> >>>>> that gets added, but that's a discussion that should be held<br>
>>> >>>>> separately.<br>
>>> >>>>><br>
>>> >>>>> The items on the ticket:<br>
>>> >>>>><br>
>>> >>>>> * kD-tree for nearest neighbor<br>
>>> >>>>> * Geometric selection routines<br>
>>> >>>>> * Profiles<br>
>>> >>>>> * Projections -- underlying quadtree<br>
>>> >>>>> * Data object selection of data containers<br>
>>> >>>>> * Data object selection of points<br>
>>> >>>>> * Orientation class<br>
>>> >>>>> * Pixelization<br>
>>> >>>>> * Color maps<br>
>>> >>>>> * PNG writing<br>
>>> >>>>><br>
>>> >>>>> Is anyone willing to claim any additional items that they will help<br>
>>> >>>>> write unit tests for?<br>
>>> >>>>><br>
>>> >>>>> -Matt<br>
>>> >>>>> _______________________________________________<br>
>>> >>>>> yt-dev mailing list<br>
>>> >>>>> <a href="mailto:yt-dev@lists.spacepope.org">yt-dev@lists.spacepope.org</a><br>
>>> >>>>> <a href="http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org" target="_blank">http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org</a><br>
>>> >>>><br>
>>> >>>><br>
>>> >>>><br>
>>> >>>> _______________________________________________<br>
>>> >>>> yt-dev mailing list<br>
>>> >>>> <a href="mailto:yt-dev@lists.spacepope.org">yt-dev@lists.spacepope.org</a><br>
>>> >>>> <a href="http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org" target="_blank">http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org</a><br>
>>> >>>><br>
>>> >><br>
>>> >><br>
>>> >> _______________________________________________<br>
>>> >> yt-dev mailing list<br>
>>> >> <a href="mailto:yt-dev@lists.spacepope.org">yt-dev@lists.spacepope.org</a><br>
>>> >> <a href="http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org" target="_blank">http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org</a><br>
>>> >><br>
>>> ><br>
>>> ><br>
>>> > _______________________________________________<br>
>>> > yt-dev mailing list<br>
>>> > <a href="mailto:yt-dev@lists.spacepope.org">yt-dev@lists.spacepope.org</a><br>
>>> > <a href="http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org" target="_blank">http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org</a><br>
>>> ><br>
>>> _______________________________________________<br>
>>> yt-dev mailing list<br>
>>> <a href="mailto:yt-dev@lists.spacepope.org">yt-dev@lists.spacepope.org</a><br>
>>> <a href="http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org" target="_blank">http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org</a><br>
>><br>
>><br>
>><br>
>> _______________________________________________<br>
>> yt-dev mailing list<br>
>> <a href="mailto:yt-dev@lists.spacepope.org">yt-dev@lists.spacepope.org</a><br>
>> <a href="http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org" target="_blank">http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org</a><br>
>><br>
_______________________________________________<br>
yt-dev mailing list<br>
<a href="mailto:yt-dev@lists.spacepope.org">yt-dev@lists.spacepope.org</a><br>
<a href="http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org" target="_blank">http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org</a><br>
</div></div></blockquote></div><br></div>