[yt-dev] 2012 Community Survey Summary

Matthew Turk matthewturk at gmail.com
Tue May 29 07:59:26 PDT 2012


Hi Nathan and Elizabeth (and everyone else),

Thanks very much for your comments.  My delay in replying was to allow
ample time for anyone else interested to chime in.  (And anyone still
thinking of doing so, please do!)

On Wed, May 23, 2012 at 3:00 AM, Nathan Goldbaum <goldbaum at ucolick.org> wrote:
> Hi all, just a few comments:
>
> I think the main issue is that yt is at its heart an extension of the python language.  If a new user is not familiar with python initially then they will have a much more difficult time following the docs and learning how to control yt.  At the same time, the fact that yt is embedded in a high level programming language means that it is very powerful and easily extensible to someone familiar with python and willing to wade through the decently-sized codebase.

I think that is definitely true.  And in the past, we've tried to ease
the process of getting people on board with Python -- through the
short orientation document, links to other places, and so on, but I
think trying to address this on our own may be a losing battle.
Python is rapidly gaining mindshare in astronomy and so perhaps this
will be less and less of a problem as time goes on.

This brings up Elizabeth's point, though, that there *are* easier
introductions to yt than strictly through the Python interface.  A
week and a half ago I gave a morning lecture in Sapporo, with Britton,
John and Elizabeth, on yt to a room full of people who hadn't done
much or any Python programming.  (There were a handful of Ruby
programmers, though, which was interesting.)  During that lecture, I
remembered just how weird some things are.  If you're not used to
Python, or even if you are but aren't used to yt, the following couple
of lines can be really daunting:

from yt.mods import *
pf = load("DD0076/DD0076")
pc = PlotCollection(pf, "c")
pc.add_slice("Density", "x")
pc.save()

Yes, each line makes sense by itself, but taken as a whole to swallow
with no exposure to the idioms or syntax of Python this can be
frustrating.  (And, let's not forget that this includes a lot of
organically developed yt-isms, which I'd like to see disappear --
making a "collection" of a single image, "c" implicitly indicating
domain center, how do you know which fields are available, blah blah.)

>
> Elizabeth's point about Reason in the docs is worth thinking about.  I don't think Reason is quite there yet, but if we can point to the gui in the docs before we even mention a python tutorial, I think far fewer people will be scared off in the same way as the negative commenter.

I agree.  I spent some time last week in a jetlagged stupor evaluating
where I thought Reason was weak, inspecting the code, and playing with
it a little bit.  I've come up with a plan, but I don't see it being
implementable in the next couple weeks, but rather sometime over the
next couple months.

 * Add a file open dialog.  I did this last week, and I'm actually
quite pleased with how it turned out.  Now you can specify files on
the command line, open from a window, or use -f to load parameter
files in Reason.
 * Refactor the javascript code.  This has no direct implications for
the user, but it will provide much more maintainable software.  This
includes:
  * Migrate to ExtJS4
  * Move to using classes provided by ExtJS for widgets, etc etc
(i.e., make creating a new widget *way* easier)
  * Reorganize .js files
  * Make ExtJS get loaded from a single .zip file rather than an
unzipped directory in $YT_DEST/src/
 * Add a few more widgets for 3D viz and improve the existing ones
 * Change the view to be parameter-file centric, rather than notebook
centric.  This should make choosing fields, making new widgets, etc
etc much easier.  The notebook will still be available, but rather
than having the user right-click on the pf, present a pf widget that
has all the available widgets as well as stats and whatnot about the
parameter file.
 * Test out linked view widgets (we used to have this in Reason v1).
See http://arxiv.org/abs/1205.4747 for more details.

Right now it's all a bit widget-centric, but I think moving to more
parameter file -centric would improve things.  This would then nicely
migrate into time series-centric analysis...  Way down the line we can
consider using the IPython infrastructure.  I thought we could try to
do that on a much shorter term, but I was wrong.

Jeff has a set of things he'd like to polish about reason, but I'm not
going to try to summarize or list them here.  I think it just needs
lots of polish.

>
> For that to happen It should be much easier to write new analysis modules for reason.  If there were an API to add new functionality in the same way as we can add new derived quantities or time series analysis tasks in pure python, I think reason would get much more attention.

I like this idea.  It would be great if we could have the main widgets
and then a set of contributed analysis modules.  Like, imagine we had
the usual slices and plots and stuff available, and then made it easy
to add on a new "canned analysis" that prompted the user for some
inputs, applied itself, and then returned either a display or a data
object back.  This could probably be done with introspection and
annotations.  I took a stab at something like this a while back:

http://bitbucket.org/MatthewTurk/yt.web_interactors

but that was in a standanlone environment.  You decorated your
function, and then yt figured out how to create widgets it showed to
the user in a web browser.  You set values, clicked "Go" and it
returned a new value back to you.

>
> One option that may be worth thinking about is a gui plotting interface separate from reason.   I know we want the new plotting interface to expose the matplotlib backend more readily - something I am completely for - but I think matplotlib can be quite confusing for newcomers.

Sigh ... yeah, I see where you're coming from with wanting a native
interface.  We have a simple interface to matplotlib that can do the
zooms and whatnot.  But I'm not sure what to do here, as even thinking
about dealing with GUI toolkits gives me headaches.  Can you elaborate
a bit on what you'd want to see?

>
> I'd like it if I could make a plot with nice-looking tick marks, labels, and colorbar by manipulating a plotting window and without writing a line of matplotlib code or python code.  It would also be cool to set up a plot interactively and save a script that could reproduce that plot or press a button to interactively make the exact same plot with a different dataset.  This is all stuff that visit does and I think there are some lessons to be learned with how visit handles data, even if the plotting and scripting is sometimes pretty awful (sorry visit developers…).

There's definitely a long list of things we could learn from visit.  I
see what you mean about making simple plots.

Perhaps we should envision an interface where we expose the matplotlib
interface readily, but we also provide a number of commonly done
idioms, named with plaintext names.  For instance, it's pretty common
to want to stagger the ticks every other.  So, make that a recipe that
can be applied to the matplotlib interface.

>
> One other thing that I've mentioned before that would make the docs much easier to use for a newcomer would be an improved search.  Right now, random API references tend to appear at the top of the search results while the narrative docs appear at the bottom which is very confusing.

I'm going to try to improve this.  Britton and I went over some of the
docs, and I think we're going to try to improve the examples/cookbook,
and also remove the API docs from the search function.  This is a
common complaint.

>
> Thanks for posting the results of the survey Matt.  I'm looking forward to working on this stuff this summer :)
>
> Cheers,
>
> Nathan
>
> On May 22, 2012, at 9:20 PM, Elizabeth Tasker wrote:
>
>> Hi everyone,
>>
>> I have a couple of comments related to the negative comment on the yt survey. In particular, there is one part I agree with and one I do not:
>>
>> The comment about yt being heavily cosmology based I do not agree with. As a non-cosmo simulator --one who still intends to torch a fellow developer's house for declaring all my research 'test problems' at a user meeting-- I use yt for all my simulation analysis and use a wide range of its capabilities. In fact, the only part I can think that I do not use is the halo finder, but I do use connected sets which is a similar idea for gas.
>>
>> I therefore feel this comment is not part of the general consensus and can be ignored.

:-)

>>
>> However, the comment about the documentation being from the user's rather than the programmer's, point of view is interesting and think worth some thought. For a while, I skirted around yt, aware that it was powerful but baulking at the effort needed to use it comfortably. Last summer, I had to write an entirely new analysis program for my simulations and I bit the bullet and starting working extensively with yt and python. I've never picked up an IDL script since, but there was definitely an energy bump. I also think that I do find the documentation **significantly** clearer now I approach it from a programmer's perspective.
>>
>> I have a couple of suggestions of what I think might help with this:
>>
>> (1) The quickest option is to keep everything the same but add more actual examples. For instance, the routine call, e.g.
>>
>> contour(self, field, ncont=5, factor=4, take_log=False, clim=None, plot_args=None):
>>
>> is confusion for a non-programmer. When I first came to yt, I spent a while with thoughts like:
>>
>> What is 'self'? Do I need to pass it like in C++ with Grid-> ?
>>
>> Although it's slightly painful, I think it would be much improved if there was an actual call example for each one of the callbacks and other tools. So we'd have:
>>
>> contour(self, field, ncont=5, factor=4, take_log=False, clim=None, plot_args=None):
>> (This is a proxy for ContourCallback.)
>> pc.modify["contours"]("Density", ncont=1, take_log=False)
>> Add contours in field to the plot. ncont governs the number of contours generated, factor governs the number of points used in the interpolation, take_log governs how it is contoured and clim gives the (upper, lower) limits for contouring.

This is a great suggestion, and I will work to implement this.  The
organization also needs some work, but I'm the wrong person for that
job.

I'll also see if we can have example images below each of these.

>>
>>
>> (2) My second suggestion involves the gui and concerns the way yt is presented. Currently, we talk about scripts and then mention there is a gui. How about we do it the other way around? So a new user's first port of call should be the gui. Then, we get to the section "beyond the gui" and talk about scripting and cookbook-ing and then finally onto full simulation analysis, i.e. writing whatever analysis package you like in python, but making use of the routines yt has.
>>
>> For instance, in the workshop we just had at Hokudai, we could have stood up and brought up the yt gui before anything else. *Then* gone onto scripts.
>>
>> The docs could be rearranged in a similar fashion.

This is an interesting idea that had not occurred to me.  While Nathan
brings up the notion that Reason is not ready, I think it is perhaps
worthwhile to figure out in what way it isn't, address those, and then
move forward with making this the standard method of interfacing with
yt for newcomers.  I'm going to be preparing a screencast of "yt for
beginners", just a short little demo, and I'll do it using Reason
first.  I think it would ease the transition, and also help those
people new to Python to avoid having the big daunting task of their
first experience being to write a script.  While of course that's
where (in an ideal world) they'd end up, maybe it shouldn't be where
they start.

>>
>> I think this might provide a much faster way for new users to start getting into yt and make the gui of much more interest. Currently, people come to it after they learn to script at which point, its value is much less because it's inevitably more limited.

Good point.

Again, thanks to everybody for their feedback, and any other comments
would be greatly appreciated.

Best,

Matt

>>
>>
>> Elizabeth
>>
>> PS today's google-doodle (or possible tomorrow's for the USA) is awesome.
>>
>>
>>
>>
>> !DSPAM:10175,4fbc659f1748213191834!
>> _______________________________________________
>> yt-dev mailing list
>> yt-dev at lists.spacepope.org
>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>>
>>
>> !DSPAM:10175,4fbc659f1748213191834!
>
> _______________________________________________
> yt-dev mailing list
> yt-dev at lists.spacepope.org
> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org



More information about the yt-dev mailing list