Hi Matt.<div><br></div><div>Glad my example was useful in some way. I guess knowing exactly which Cython routines to test for what is what I meant about where to start.</div><div><br></div><div>Thanks for the tip about the stream frontend.</div>
<div><br></div><div>- Casey</div><div><br><br><div class="gmail_quote">On Mon, Sep 24, 2012 at 1:52 PM, Matthew Turk <span dir="ltr"><<a href="mailto:matthewturk@gmail.com" target="_blank">matthewturk@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hey Casey and Anthony,<br>
<div class="im"><br>
On Mon, Sep 24, 2012 at 4:20 PM, Casey W. Stark <<a href="mailto:caseywstark@gmail.com">caseywstark@gmail.com</a>> wrote:<br>
> Hi Anthony.<br>
><br>
> I completely agree that we should target the level of functions actually<br>
> performing the projection rather than yt's organization. The mock frontend<br>
> suggestion was just a hack to get there. I don't know if there's a way<br>
> around it though...<br>
><br>
> Here's an example of what I sorted through to get to projections:<br>
> - Load a test plotfile, check pf.h.proj to find it's source.<br>
> - Read through data_objects/hierarchy.py and<br>
> utilities/parallel_tools/parallel_analysis_interface.py to find where proj<br>
> is attached, can't find it.<br>
> - The proj docstring says it is a reference to AMRQuadProj. Can't find a<br>
> class by that name.<br>
> - Search data_objects sources for "proj", find AMRProjBase.<br>
><br>
> So it looks like the functionality is wrapped up in the __project_level and<br>
> _project_grid methods. I can't think of a way to test those without creating<br>
> an AMRProjBase, and that requires a staticoutput object.<br>
<br>
</div>You're right, the projection stuff as *projections* is not easy to<br>
test. But in terms of testing the underlying code, which is wrapped<br>
up in a Cython class called QuadTree, I think it could be done. The<br>
steps you're describing are actually all part of the existing answer<br>
testing machinery, which performs a couple things and verifies that<br>
they don't change over time:<br>
<br>
1) Project some fields from the disk<br>
2) Project a couple derived fields<br>
3) Project a derived field that requires spatial derivatives<br>
4) Project the "Ones" field, which should be 1.0 everywhere.<br>
<br>
So these things are done, but it is also possible that the specific<br>
quadtree functionality could be tested, in isolation from the<br>
projection. I think this may be oneo f the things Anthony is talking<br>
about -- answer testing can handle the big, complex items, and by<br>
breaking down to the fundamentals we can address isolated items from a<br>
unit testing perspective.<br>
<div class="im"><br>
><br>
> So unfortunately, I think it would still come down to having a fake<br>
> frontend. It's not ideal, but it seems like any more isolation would require<br>
> big rewrites to yt.<br>
<br>
</div>One fun thing that is not usually known is that we have a fake<br>
frontend already, it just doesn't get used much. It's called the<br>
"Stream" frontend and it was designed originally to be used in<br>
ParaView, but now gets used by the (new, not-yet-documented/released)<br>
load_uniform_grid function as well as by Hyperion, the RT code by Tom<br>
R. It can set up AMR as well as static mesh. It's not terribly well<br>
documented, but there are examples on the wiki.<br>
<br>
One thing I've been thinking about is actually creating a couple fake<br>
outputs, which could be defined analytically with spheres of<br>
overdensity inside them. In principle, if we added refinement<br>
criteria, we could make this relatively complex data that was defined<br>
with only a few lines of code, but spun up a big in-memory dataset.<br>
<br>
(This exact thing is on my list of things to do and then to output in<br>
GDF, by the way...)<br>
<br>
That I think could come, down the road a bit. The refinement criteria<br>
wouldn't be too bad to implement, especially since we already have the<br>
grid splitting routines. I just don't think we should focus on it at<br>
the moment. But the uniform grid creation and loading works already<br>
-- I used it this morning. You can do it with:<br>
<br>
from yt.frontends.stream.api import load_uniform_grid<br>
ug = load_uniform_grid({"VelocityNorm":data1, "Density":data2}, [359,<br>
359, 359], 1.0)<br>
<br>
the list is the dimensions of the data and the value is the to-cm conversion.<br>
<div class="im"><br>
><br>
> Of course, I could be missing something. Matt, can you think of a better<br>
> way?<br>
<br>
</div>I think for this specific example (and your damningly complex tracing<br>
of things through the source ...) the easiest thing to do is isolate<br>
the Cython routine, which it seems I was able to do only because I<br>
wrote it and which seems quite buried in the code, and to also provide<br>
high-level machinery for faking a frontend.<br>
<br>
-Matt<br>
<div class="HOEnZb"><div class="h5"><br>
><br>
> - Casey<br>
><br>
><br>
> On Mon, Sep 24, 2012 at 11:02 AM, Anthony Scopatz <<a href="mailto:scopatz@gmail.com">scopatz@gmail.com</a>> wrote:<br>
>><br>
>> Helo Casey,<br>
>><br>
>> Sorry for taking the whole weekend to respond.<br>
>><br>
>>>> I would like to help with this, but it's difficult to figure out where<br>
>>>> to start.<br>
>><br>
>><br>
>> Not to worry. I think that any of the items listed at the bottom of Matt's<br>
>> original email<br>
>> would be a great place to start.<br>
>><br>
>>>><br>
>>>><br>
>>>> Say I want to test projections. I make a fake 3D density field, maybe<br>
>>>> something as simple as np.arange(4**3).reshape((4, 4, 4)). I write down the<br>
>>>> answer to the x-projection. Now all I need to do is call<br>
>>>> assert_allclose(yt_result, answer, rtol=1e-15), but I don't know what pieces<br>
>>>> of low-level yt stuff to call to get to `yt_result`. Hopefully that's<br>
>>>> clear...<br>
>>>><br>
>>>> Maybe this comes down to creating a fake frontend we can attach fields<br>
>>>> to?<br>
>><br>
>><br>
>> Actually, I disagree with this strategy, as I told Matt when we spoke last<br>
>> week.<br>
>> What is important is that we test the science and math parts of the code<br>
>> before, if ever, dealing with the software architecture that surrounds<br>
>> them.<br>
>><br>
>> Let's taking your example of projections. What we need to test is the<br>
>> actual function<br>
>> or method which actually slogs through the projection calculation. In<br>
>> many cases in<br>
>> yt these functions are not directly attached to the front end but live in<br>
>> analysis, visualization<br>
>> or utilities subpackages. It is these such packages that we should worry<br>
>> about testing.<br>
>> We can easily create routines to feed them sample data.<br>
>><br>
>> On the other hand, testing or mocking things like frontends should be a<br>
>> very low priority.<br>
>> At the end of the day what you are testing here is pulling in data from<br>
>> disk or other<br>
>> sources. Effectively, this is just re-testing functionality present in<br>
>> h5py, etc. That is not<br>
>> really our job. Yes, in a perfect world, front ends would be tested too.<br>
>> But I think that the<br>
>> priority should be placed on things like the KDTree.<br>
>><br>
>> Be Well<br>
>> Anthony<br>
>><br>
>>>><br>
>>>><br>
>>>> - Casey<br>
>>>><br>
>>>><br>
>>>> On Fri, Sep 21, 2012 at 2:42 PM, Matthew Turk <<a href="mailto:matthewturk@gmail.com">matthewturk@gmail.com</a>><br>
>>>> wrote:<br>
>>>>><br>
>>>>> Hi all,<br>
>>>>><br>
>>>>> As some of you have seen (at least Stephen), I filed a ticket this<br>
>>>>> morning about increasing testing coverage. The other night Anthony<br>
>>>>> and I met up in NYC and he had something of an "intervention" about<br>
>>>>> the sufficiency of answer testing for yt; it didn't take too much work<br>
>>>>> on his part to convince me that we should be testing not just against<br>
>>>>> a gold standard, but also performing unit tests. In the past I had<br>
>>>>> eschewed unit testing simply because the task of mocking data was<br>
>>>>> quite tricky, and by adding tests that use smaller bits we could cover<br>
>>>>> unit testable areas with answer testing.<br>
>>>>><br>
>>>>> But, this isn't really a good strategy. Let's move to having both.<br>
>>>>> The testing infrastructure he recommends is the nearly-omnipresent<br>
>>>>> nose:<br>
>>>>><br>
>>>>> <a href="http://nose.readthedocs.org/en/latest/" target="_blank">http://nose.readthedocs.org/en/latest/</a><br>
>>>>><br>
>>>>> The ticket to track this is here:<br>
>>>>><br>
>>>>><br>
>>>>> <a href="https://bitbucket.org/yt_analysis/yt/issue/426/increase-unit-test-coverage" target="_blank">https://bitbucket.org/yt_analysis/yt/issue/426/increase-unit-test-coverage</a><br>
>>>>><br>
>>>>> There are a couple sub-items here:<br>
>>>>><br>
>>>>> 1) NumPy's nose test plugins provide a lot of necessary functionality<br>
>>>>> that we have reimplemented in the answer testing utilities. I'd like<br>
>>>>> to start using the numpy plugins, which include things like<br>
>>>>> conditional test execution, array comparisons, "slow" tests, etc etc.<br>
>>>>> 2) We can evaluate, using conditional test execution, moving to nose<br>
>>>>> for answer testing. But that's not on the agenda now.<br>
>>>>> 3) Writing tests for nose is super easy, and running them is too. Just<br>
>>>>> do:<br>
>>>>><br>
>>>>> nosetest -w yt/<br>
>>>>><br>
>>>>> when in your source directory.<br>
>>>>><br>
>>>>> 4) I've written a simple sample here:<br>
>>>>><br>
>>>>><br>
>>>>> <a href="https://bitbucket.org/yt_analysis/yt-3.0/src/da10ffc17f6d/yt/utilities/tests/test_interpolators.py" target="_blank">https://bitbucket.org/yt_analysis/yt-3.0/src/da10ffc17f6d/yt/utilities/tests/test_interpolators.py</a><br>
>>>>><br>
>>>>> 5) I'll handle writing up some mock data that doesn't require shipping<br>
>>>>> lots of binary files, which can then be used for checking things that<br>
>>>>> absolutely require hierarchies.<br>
>>>>><br>
>>>>> --<br>
>>>>><br>
>>>>> The way to organize tests is easy. Inside each directory with<br>
>>>>> testable items create a new directory called "tests", and in here toss<br>
>>>>> some scripts. You can stick a bunch of functions in those scripts.<br>
>>>>><br>
>>>>> Anyway, I'm going to start writing more of these (in the main yt repo,<br>
>>>>> and this change will be grafted there as well) and I'll write back<br>
>>>>> once the data mocking is ready. I'd like it if we started encouraging<br>
>>>>> or even mandating simple tests (and/or answer tests) for functionality<br>
>>>>> that gets added, but that's a discussion that should be held<br>
>>>>> separately.<br>
>>>>><br>
>>>>> The items on the ticket:<br>
>>>>><br>
>>>>> * kD-tree for nearest neighbor<br>
>>>>> * Geometric selection routines<br>
>>>>> * Profiles<br>
>>>>> * Projections -- underlying quadtree<br>
>>>>> * Data object selection of data containers<br>
>>>>> * Data object selection of points<br>
>>>>> * Orientation class<br>
>>>>> * Pixelization<br>
>>>>> * Color maps<br>
>>>>> * PNG writing<br>
>>>>><br>
>>>>> Is anyone willing to claim any additional items that they will help<br>
>>>>> write unit tests for?<br>
>>>>><br>
>>>>> -Matt<br>
>>>>> _______________________________________________<br>
>>>>> yt-dev mailing list<br>
>>>>> <a href="mailto:yt-dev@lists.spacepope.org">yt-dev@lists.spacepope.org</a><br>
>>>>> <a href="http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org" target="_blank">http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org</a><br>
>>>><br>
>>>><br>
>>>><br>
>>>> _______________________________________________<br>
>>>> yt-dev mailing list<br>
>>>> <a href="mailto:yt-dev@lists.spacepope.org">yt-dev@lists.spacepope.org</a><br>
>>>> <a href="http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org" target="_blank">http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org</a><br>
>>>><br>
>><br>
>><br>
>> _______________________________________________<br>
>> yt-dev mailing list<br>
>> <a href="mailto:yt-dev@lists.spacepope.org">yt-dev@lists.spacepope.org</a><br>
>> <a href="http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org" target="_blank">http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org</a><br>
>><br>
><br>
><br>
> _______________________________________________<br>
> yt-dev mailing list<br>
> <a href="mailto:yt-dev@lists.spacepope.org">yt-dev@lists.spacepope.org</a><br>
> <a href="http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org" target="_blank">http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org</a><br>
><br>
_______________________________________________<br>
yt-dev mailing list<br>
<a href="mailto:yt-dev@lists.spacepope.org">yt-dev@lists.spacepope.org</a><br>
<a href="http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org" target="_blank">http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org</a><br>
</div></div></blockquote></div><br></div>