[yt-dev] Frontend tests

[Matthew Turk]

Hi Nathan,

On Wed, Aug 28, 2013 at 4:53 PM, Nathan Goldbaum <nathan12343 at gmail.com> wrote:
> I don't think creating new answer tests is hard as such.  What's difficult
> is setting up the testing environment, uploading new answers to S3, and
> figuring out how to use the nose answer testing plugin to actually run the
> tests. I think Kacper mentioned that he could add some jenkins magic to
> automate most of that process which should ease the burden of adding new
> answer test results in the future. While I like your idea, I still do think
> we should be doing quantitative answer testing of all the frontends.

Okay, fair point.  Let's continue with answer testing as primary focus.

>
> Do we have datasets for the new 3.0 only frontends?  Can we put them on
> yt-project.org/data?

Yup.  I will upload them and add them to the data listing.

>
> I like your idea and think it nicely integrates some degree of lightweight
> testing and, more importantly, documents how to use the frontends and
> validates that accessing them works using the latest and greatest
> development version of yt.  I see no problem with hosting it under the main
> yt_analysis account.

Alright, cool.  I have started sketching it out and will put it up soon.

-Matt

>
>
> On Wed, Aug 28, 2013 at 1:46 PM, Matthew Turk <matthewturk at gmail.com> wrote:
>>
>> Hi all,
>>
>> I'd like to bring up the topic of frontend tests.
>>
>> As it stands, we have answer tests for a few frontends, and a large
>> amount of sample data for other frontends.  The answer tests are fine,
>> but they are also somewhat ... difficult to write and come up with.  I
>> believe that they should still exist inside the main repo.
>>
>> However, what I'd like to propose is a new repository
>> ("frontend_tests" perhaps?) that includes scripts for each frontend
>> that load data, save images, and that we will then *version* a set of
>> results images and data inside.  This repository will be allowed to
>> grow larger than we'd like the main yt repository to grow, and it
>> would also mean that we could use normal pull request mechanisms for
>> updating results, rather than AWS S3 uploads with keys and so on.
>>
>> My idea was that it would include something like a helper function
>> library (for common routines like "what is the current version of the
>> code" and "commit a new version of this image") and would also include
>> image-generating scripts and mechanisms for viewing images.  The idea
>> is that you would run a script at the top level and it would spit out
>> a bunch of images or data, and there would be templates of HTML that
>> you could view old-versus-new.  This could then be integrated into our
>> CI system (I spoke with Kacper about this previously).  It would serve
>> two purposes:
>>
>> 1) Display results as a function of the current iteration (these
>> results would not be quantitatively assessed; this would be the
>> function of the answer testing we already have)
>> 2) Tell us if loading a dataset or frontend breaks
>> 3) Light quantitative result analysis
>>
>> I'm planning to create a repository similar to this regardless (for
>> demonstrating frontend scripts) but I'm bringing it to the list to see
>> if it is alright to host under the main yt_analysis team on Bitbucket
>> and to integrate into testing.  Does this appeal to anyone?  I imagine
>> it could be much simpler than creating new answer tests.
>>
>> -Matt
>> _______________________________________________
>> yt-dev mailing list
>> yt-dev at lists.spacepope.org
>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>
>
>
> _______________________________________________
> yt-dev mailing list
> yt-dev at lists.spacepope.org
> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>