[yt-dev] 2012 Community Survey Summary

[Matthew Turk]

Hi everyone,

I've closed up the 2012 community survey.  We had a fair number of
responses (24), which I think is a good cross-section of the
community.  (There are 134 people on yt-users, so this is just under
20% of the list population, many of whom may not be actual or active
users.)

For the most part, it was very, very positive.  We're definitely doing
some things very right.  However, we had one *particularly* negative
response (explored in detail below), and a good amount of constructive
criticism scattered through the positive responses.  I'll detail the
particularly negative response, and then discuss the rest in bulk.  At
the end I've included the set of free-form comments left in the
survey.  I think the big takeaways there are pretty clear, and mostly
are on the roadmap!

I'm happy to make available the original responses, but I've collated
most of them here.  I have no doubt there is a *huge* selection
effect, and I'm *really* grateful that the strongly negative response
was left at all, so that we could get an insight into what went wrong!

Finally, I'd encourage everyone reading this to reply if you have
comments, suggestions, or any feedback that did not make it into the
survey.  I don't want this to be the end of the conversation, but
rather the start of the feedback process.

= Negative =

The negative response was, indeed, quite negative.  (And, it was
difficult to read, to be honest.)  Here are the comments, unedited.

--
* What should be changed about the documentation?

"I want more narrative documentation, It should be written from the
user's rather than the programmer's point of view.  Start with simple
tasks, and then build up to scripting and modification only slowly."

* Have you found anything in yt to be particularly annoying?

"Unreliability -- both times I've tried to use it, I had to invoke
developer support to even get basic work done.  Terribly written
documentation -- virtually incomprehensible if you aren't both a
cosmologist and a programmer."

* If there's one message about yt you want to send, what is it?

"The concepts are powerful, but it should either be stated to be
specifically for cosmological visualization only, or its general
purpose use should be better supported."

* How do you find the yt community to be?

"Helpful and friendly, but rather too convinced of yt's usefulness, so
that people get sucked in for whom the tool is not at all ready yet."

* Anything to add about deficiencies in yt?

"It is far too focused on the cosmology use case to be useful as a
general purpose visualization tool.  This impacts the documentation,
which only makes sense if you are in the mindset of halo-finding and
object property measurement, and the testing, which relies on users to
find such basic things as support for 2D model visualizations (the
most basic test problems) or non-square grids in nominally supported
codes."
--

Basically, my takeaways from this remarkably well-written, if blunt,
takedown were:

 * The submitter thinks we're pretty self-satisfied, which gets in the
way of supporting new users.
 * The submitter finds that we're cosmology-biased.  I don't know that
this is necessarily the case, but it seems to come across in our
supporting materials.
 * The submitter believes we have positioned ourselves as "general
purpose" but I'm not sure if this is in opposition to "astrophysically
focused" or in opposition to "cosmology focused"
 * Our support for simplified dimensionality and test problems seems
lacking.  I don't know how much to read in to the "nominally supported
codes" section.  I assume this means FLASH, but it's not clear which
version of FLASH.
 * We need much, much better and more descriptive error handling.

I am a bit sorry that this submitter did not include contact
information and instead chose to submit anonymously.  A followup
discussion would have been very productive, particularly because I
think this response will spur a number of changes.  I've shared this
response with a few other people on this list, and all of them have
been eager to make changes in response to the comments.  It would have
been nice to be able to discuss them with this person, and to start an
ongoing discussion.

However, regardless, this should act as a wakeup call.  I will
endeavor to ensure that we support reduced dimensionality test
problems, and during the next iteration of the documentation I hope we
can iterate on the tutorial, the narrative docs, and so on.  I suppose
in the past we have neglected this population of computational
astrophysicists that are not as at home with programming, so perhaps
either adding better educational materials *or* identifying existing
resources that exist (and linking to them) would be useful.

One plan I'd had had been to add screencasts.  I think organizing
these as how to install, then moving on to how to get going with
*each* simulation code type (and providing explicit sample scripts and
example data) independently would be a good strategy.

Does anyone have any additional thoughts on this?  Furthermore, does
anyone else want to pile on?

= All Results =

We had 24 replies.  Of these people:

--
20 (80%) said yt was easy to use
3 (12%) said yt was difficult to use
1 (4%) said yt was too hard to use
--

I think this is *great*.  And, I think it's a huge compliment to the
work we've all put in to make the code easier to use.

--
16 (64%) said they use yt all the time (!)
7 (28%) use it once in a while
1 (4%) tried it, it broke, they stopped
--

Again, wow.  While it's difficult to read too much into this because
of the selection bias, it's still great.  And the final response is
valuable to put into context the overall survey results.

--
The GUI Reason:

2 (8%) Use it often
10 (40%) Use it once in a while
11 (44%) Don't use it
1 (4%) Never heard of it
--

This is interesting, but I'd put it out to the rest of you to
interpret.  Sam, Jeff, thoughts?

--
The thing yt needs more than anything else:

2 (8%) More supported codes
3 (12%) Speed or parallelism improvements
6 (24%) More canned tasks for astrophysical analysis
1 (4%) Better/easier visualization not destined for publication
5 (20%) Better/easier visualization for publication
0 (0%) Improvements to the GUI
3 (12%) API documentation
3 (12%) Narrative documentation and tutorials
0 (0%) A more friendly development process or community
0 (0%) More reliability or error handling
0 (0%) Ability to conduct time series analysis
2 Other 2
--

It seems like the top two priorities are better/easier publication
quality viz, and canned astrophysical tasks.  This is excellent, as
they are both well within our wheelhouse.  One thing we should start
pushing harder on is encouraging users to submit analysis modules
upstream -- and to create uniform interfaces for analysis modules,
particularly ones that work on time series.  Britton and I have talked
about smoothing up the various interfaces, making things more
"Dataflow" or "Pipeline" oriented, and this is a good point of focus
for that.

One sad bit is that I really want to improve the GUI (and I think John
and Sam have expressed a similar desire, and I know Jeff has) but it
looks like either due to indifference or its capabilities, this isn't
a priority for others.

--
What should be changed about the documentation?

0 (0%) There's too much -- slim it down!
7 (32%) I want more narrative documentation
17 (77%) I want more example scripts
6 (27%) The organization should be improved
2 (9%) I'd like more screencasts
6 (27%) Other
--

More examples!  We have let the cookbook falter a bit.  That should be
reorganized and made more extensive.

Of the 24 responses, 14 had watched one or more videos from the
workshop.  This was the most surprising response.

--
Which of these pieces of functionality have you used:

10 (48%) yt pastebin (the yt pastebin)
6 (29%) Clump finder
11 (52%) Halo finder
17 (81%) Volume renderer
9 (43%) IRC for yt help
6 (29%) Time series analysis
--

I found this the most interesting.  More people have used the volume
renderer than have used the pastebin!  I guess it's a hit, and I would
guess it's also a big draw.

--
Volume rendering in yt is ...

15 (75%) Attractive
13 (65%) Hard to get right
2 (10%) Slow
2 (10%) Easy to set up
--

This completely jives with my experience.  VR is super hard to make
look nice, easy to make look terrible, and gives great results one
you've set it up.  I think coming up with new methods for rapid
iteration, pre-visualization, and so on are all going to be important
in the near future.

= Specific Comments =

Have you found anything in yt to be particularly annoying?

 * It may be related with python. Particular version of yt need
particular version of python. It particularly makes an issue with mac
OS X.. (it is partially mac OSX fault). Now days, yt install its own
python in to mac but it makes more work... For example, if I want to
make my own routine with scipy and yt, it may cause issue.
 * Deleting the *yt and *harrays for making some script work.
 * "Unreliability -- both times I've tried to use it, I had to invoke
developer support to even get basic work done.  Terribly written
documentation -- virtually incomprehensible if you aren't both a
cosmologist and a programmer."
 * Most of the "cookbook" relies on plot collections which are clunky
in some situations.  It would be nice to see more cookbook entries for
generating plots, figures, etc without relying on this data structure.
 * I get lost in a hurry if I do something non-standard and need to
look under the hood.  More under-the-hood docs would be cool.
 * Yes, I found that simple funtions like figure control (i.e.,
changing axis, symbles, grid, changing font and size of text etc) are
quite  annoying. I had difficulties in plotting multiple plots etc.
Size of simulation box is not displayed in YT and one has to manually
mention which could be useful.

If there's one message about yt you want to send, what is it?

 * You guys are doing a great job - keep it up!
 * Thanks!
 * It is good tool! I hope it will continuously and actively improve.
 * "Keep up the great job!
 * Keep developing this great project! Congrats
 * The concepts are powerful, but it should either be stated to be
specifically for cosmological visualization only, or its general
purpose use should be better supported.
 * yt makes my life easier! Keep it up.
 * Keep doing the good work.
 * Good work thus far.
 * yt is elegant.

 How do you find the yt community to be?

 * Very welcoming - no complaints!
 * Great!
 * friendly, helpful + dedicated
 * It's a great place for receiving and giving help and suggestions
about real work.
 * The friendliest community I know :)
 * Great!
 * Helpful and friendly, but rather too convinced of yt's usefulness,
so that people get sucked in for whom the tool is not at all ready
yet.
 * I haven't been active on the list at all, but it seems like
response times are (extremely) short, and everyone is very helpful.
 * Love it!
 * Friendly and helpful.
 * Invaluable!  The IRC channel and email list are incredibly helpful
resources when something is missing from the docs or a bug creeps in
the code.
 * Excellent.  Super helpful.
 * I found them very helpful.
 * Very responsive and helpful.

Have you ever contributed code to yt?  If so, what was good or bad
about that experience?

 * Not yet.
 * Not yet, but I look forward to it!
 * Yes, hg was a bit rough experience at the beginning, but as soons
as I've achieved rookie level, I'm fine with it.
 * No.
 * Haven't made any changes.
 * yes, a little bit. It was a very good experience.
 * Yes.  I felt it fixed a problem, and it was well-received.
 * No
 * Wish someone have short example of write code, add to doc, add test problem.

Anything to add about deficiencies in yt?

 * Most of the deficiencies that ails me are already covered in yt-3.0
doc: multiple fluids, handling non-cartesian grids, handling vector
fields
 * It is far too focused on the cosmology use case to be useful as a
general purpose visualization tool.  This impacts the documentation,
which only makes sense if you are in the mindset of halo-finding and
object property measurement, and the testing, which relies on users to
find such basic things as support for 2D model visualizations (the
most basic test problems) or non-square grids in nominally supported
codes.
 * It would be great to have more direct access to matplotlib axes
objects, say when you make a PlotCollection, to allow for easy
manipulation of tick marks, axes labels, etc.
 * I would like volume rendering to be easier to use correctly (i.e.
interactive isocontour finding).
 * I don't like the choices for the first question-- I think yt is
very easy for some things, and gets more difficult as you get off the
beaten path in a non-linear manner.
 * Figure controls, 3-D visualization, Better GUI not just web based,
Importing data into YT with simple formats like ASCII or Binary etc,
Imroved halofinder like T_vir
 * I would like to see it read Gadget snapshots.
 * need transfer function alongside volume rendering in log space, or
make it customizable for the user.  Right now it's printed in real
space so if you have multiple add gaussian, the smaller ones are
barely noticeable, and the background being white is not good for if
the volume rendering is done with a background.  Would be nice to be
able to customize it.  Maybe add transfer function and ability to add
strings of text to describe the volume rendering off to the sides or
top/bottom.