[yt-dev] Request comments/vote on use of notebooks in yt docs

Cameron Hummels chummels at gmail.com
Mon Oct 28 06:42:18 PDT 2013


Hey Nathan,

First of all, let me thank you for putting together this functionality.  It
looks great!

Secondly, I'm confused about something.  Notebooks can be as simple as we
want to make them.  I don't see why we cannot just convert the existing
scripts into notebooks without interspersing them with lots of comments and
intermediate images.  That way we retain the current short scripts that
everyone (including me) is pleased with, while having the ability to make
more advanced topics into longer or more detailed notebooks.  Then it's all
in your proposed new infrastructure as well.  Am I missing something here?

Cameron


On Sun, Oct 27, 2013 at 9:20 PM, Nathan Goldbaum <nathan12343 at gmail.com>wrote:

> Hi all,
>
> I've finished up a prototpye for the sphinx directive and issued a PR
> about this:
> https://bitbucket.org/yt_analysis/yt-doc/pull-request/109/new-notebook-sphinx-directive-and-adding/diff
>
> To get a taste for what the new embedded notebooks will look like, take a
> look at the docs build I've put on my webserver:
>
> http://ngoldbaum.net/docs_build/bootcamp/simple_visualization.html
>
> After creating the extension and playing around a bit with the existing
> cookbook recipes, I think I've come around more to Elizabeth, Brian, and
> Matt's point of view. Many of the example scripts show off functionality
> that is most useful in the context of a script. If one were trying to do
> the same thing in a notebook, they would most likely use similar, but
> somewhat different code.
>
> I think instead of wholesale converting all the example scripts, we should
> include example notebooks in addition to the scripts.  With the new sphinx
> plugin I've written, these example notebooks will be directly embedded in
> the docs and should flow nicely with the example scripts. Some of the
> scripts probably work better as notebooks and we probably should think
> about converting them as they come up.
>
> One place where I could see the notebooks working very nicely is in the
> narrative docs. Since the notebook directive produces output that's
> indistinguishable from the output generated from normal sphinx .rst files,
> we could adapt the content of the existing narrative docs to be notebooks.
>  The major benefit here is that output will be embedded in the docs and
> that docs will be tested every time they are built.
>
> I've got a couple of example notebooks to show off movie-making and
> parallelism under IPython's parallelism that I'd like to add in the next
> couple of days.  If any of you have some notebooks that you think would
> work well in the documentation, please consider adding them!
>
> -Nathan
>
>
> On Sat, Oct 26, 2013 at 5:58 PM, Elizabeth Tasker <
> tasker at astro1.sci.hokudai.ac.jp> wrote:
>
>> why not flip it around and have the script be the standard, compact way
>> of viewing the data, but one can click on a link to take the user to a
>> notebook that they can look at, and which can be downloaded?
>>
>>
>> I'm +1 on this idea. When I work with my students during our meeting, we
>> use python notebooks to explore the data, but when they or I do serious
>> analysis, we use scripts.
>>
>> Given this is the main way of using yt, I think there's a lot of benefit
>> in demonstrating it as the standard and keeping the python notebooks as a
>> teaching tool that one could click on to view. (Rather like the 'hint'
>> system in online homework systems!).
>>
>> Elizabeth
>>
>>
>>
>> On Oct 27, 2013, at 5:08 AM, Brian O'Shea <bwoshea at gmail.com> wrote:
>>
>> Hi folks,
>>
>> I'm sorry for not responding to Nathan and Cameron's emails from a couple
>> of days ago in a timely fashion.
>>
>> I believe that people who use yt to do data analysis in a production
>> setting are primarily using scripts, as Britton said.  This is certainly
>> true in my own research group, and I think generally true when people are
>> doing batch analysis of large datasets (and/or lots of datasets) on
>> supercomputers.  If the cookbook is truly meant to help users that are new
>> to yt, it makes sense to present the cookbook code in the manner that most
>> closely approximates how a new user will actually use it, so presenting the
>> code as scripts in the cookbook makes the most sense to me.  Also, the
>> current implementation of the cookbook is relatively compact, so it's easy
>> to scroll through and extract what one needs, which is useful for the OTHER
>> use of the cookbook, which is for more advanced users to quickly zip
>> through and remind themselves of how the heck a piece of functionality
>> works.  Using iPython notebooks would make the cookbook much longer, or
>> force us to break it up into a bunch of segments.  Either way, it would
>> also be harder to quickly parse through because the actual commands the
>> user cares about are embedded inside quite a bit of extraneous (for this
>> purpose) text.  I'm concerned that this mailing list's enthusiasm for the
>> new, shiny thing, while commendable in many ways, is driving us away from
>> the original purposes of the cookbook, at least as I understand them.
>>
>> That said, I do think that iPython notebooks are very cool, and I intend
>> to use them as a teaching tool in a class I'm teaching next semester. It's
>> also pretty clear which way the wind is blowing on this mailing list, so
>> putting iPython notebooks into the cookbook in some way appears to be
>> inevitable.  So, why not flip it around and have the script be the
>> standard, compact way of viewing the data, but one can click on a link to
>> take the user to a notebook that they can look at, and which can be
>> downloaded?  That allows the cookbook to be relatively easy to parse, and
>> also lets people who want to see the notebooks do so.
>>
>> --Brian (though possibly "Grampa Brian" is a more appropriate signoff,
>> given the tone of my email)
>>
>>
>> On Sat, Oct 26, 2013 at 12:09 PM, Britton Smith <brittonsmith at gmail.com>wrote:
>>
>>> There seems to be enough utility in having both the notebooks and at
>>> least some way of downloading pure python scripts.  On the one hand, the
>>> notebooks are illustrative for both yt usage and for learning ipython
>>> notebooks themselves.  As a teaching tool, they seem great.  On the other,
>>> I think the primary usage of yt for analysis is through scripting and we
>>> should try to preserve that notion and make these available as seeds for
>>> new users looking to build their own scripts.  I think that presenting the
>>> recipes with the notebook, but with a button to download as a script
>>> nearby, would be an option that satisfies everyone.
>>>
>>> Britton
>>>
>>>
>>> On Thu, Oct 24, 2013 at 9:54 PM, Chris Malone <chris.m.malone at gmail.com>wrote:
>>>
>>>> I'm +1 on this, with the possibility of providing also just the .py
>>>> files from nbconvert for those that won't/can't use notebooks.
>>>>
>>>> Chris
>>>>
>>>>
>>>> On Thu, Oct 24, 2013 at 12:28 PM, Sam Skillman <samskillman at gmail.com>wrote:
>>>>
>>>>> Hi all,
>>>>>
>>>>> I think if we don't take this opportunity to use notebooks during this
>>>>> major update to the documentation, we'll be kicking ourselves in about a
>>>>> year for missing it.  I'd like to voice my support for the following layout:
>>>>>
>>>>> 1) Write the cookbook examples in a notebook, annotating it with
>>>>> comments and reasoning in markdown cells. Use some amount of conventions
>>>>> for data loading so that with minimal work users could change the path to
>>>>> the data and run themselves.
>>>>> 2) Display the notebooks in the docs
>>>>> 3) Allow for download of both a stripped down (no images included)
>>>>> .ipynb and nbconverted script.
>>>>>
>>>>> The neat thing is that now you have all these .ipynb files in the doc
>>>>> repo. It would be stupid simple to then show people how to go to that
>>>>> folder, launch yt notebook, you can then interactively execute the examples
>>>>> after pointing to the data locations.  This would be really really nice in
>>>>> my opinion.
>>>>>
>>>>> Cheers,
>>>>> Sam
>>>>>
>>>>>
>>>>>
>>>>> On Wed, Oct 23, 2013 at 10:02 PM, Elizabeth Tasker <
>>>>> tasker at astro1.sci.hokudai.ac.jp> wrote:
>>>>>
>>>>>> Hi everyone,
>>>>>>
>>>>>> I agree with Brian. I've only recently discovered python notebooks
>>>>>> and I love them for my own analysis and when I'm looking at data together
>>>>>> with my students and we're exploring data sets, but I feel they are
>>>>>> unnecessary cumbersome for code examples.
>>>>>>
>>>>>> Most of the cookbook snippets have only one simple product (image,
>>>>>> plot etc), so you don't gain a great deal by showing the results from each
>>>>>> line in the notebook.
>>>>>>
>>>>>> Additionally, however great notebooks are, they're not as handy for
>>>>>> tasks that you need to perform repeatedly or for writing full length
>>>>>> analysis scripts. In my opinion, using yt as a front end to more detailed
>>>>>> analysis is one of its major strengths. If you take away the cookbook
>>>>>> scripts, we lack examples of yt in python programs.
>>>>>>
>>>>>> The notebooks are also not quite as easy to use as a downloaded code
>>>>>> snippet.
>>>>>>
>>>>>> Elizabeth
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Oct 23, 2013, at 7:23 PM, Brian O'Shea <bwoshea at gmail.com> wrote:
>>>>>>
>>>>>> Hi Cameron,
>>>>>>
>>>>>> For what it's worth, as a user of yt I find the current cookbook
>>>>>> format to be incredibly useful.  I don't think notebooks would add to the
>>>>>> utility - it's easy enough for me to download the script and load it into a
>>>>>> notebook on my own machine if that's what I want to do.  It definitely
>>>>>> seems that the challenges (and possible downsides) substantially outweigh
>>>>>> the benefits, at least for me and my usage patterns.
>>>>>>
>>>>>> --Brian
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Tue, Oct 22, 2013 at 4:13 PM, Cameron Hummels <chummels at gmail.com>wrote:
>>>>>>
>>>>>>> Hey everyone,
>>>>>>>
>>>>>>> The documentation sprint is next Monday and Tuesday for those of you
>>>>>>> who want to participate.  I'll send out another email regarding that in the
>>>>>>> next day or so.
>>>>>>>
>>>>>>> In preparation for that, though, I wanted to request input from the
>>>>>>> developer community on something related to the docs.
>>>>>>>
>>>>>>> Right now, the cookbook page contains a lot of recipes for doing
>>>>>>> various things, and I think it is hugely beneficial to the community to
>>>>>>> maintain this (I personally use this page a lot too!).  However, with the
>>>>>>> advent of ipython notebooks over the last year, we are faced with a
>>>>>>> question: should we move toward incorporating more notebooks into our
>>>>>>> documentation, and specifically, do you we want to transfer the existing
>>>>>>> cookbook to a series of notebooks for each task?
>>>>>>>
>>>>>>> Benefits:
>>>>>>> --Portability: users can download an entire notebook for both
>>>>>>> viewing how it should work as well as being able to execute it locally on
>>>>>>> their own datasets
>>>>>>> --Illustrative: Interim steps in a cookbook can produce output that
>>>>>>> can show up inside the notebook, instead of being a single script which
>>>>>>> generates an image/output at the end (as is the case in the current
>>>>>>> paradigm)
>>>>>>> --Narrative: notebooks provide more space for narrating each step,
>>>>>>> instead of confining any narrative to comments in the recipe itself
>>>>>>>
>>>>>>> Disadvantages:
>>>>>>> --Work: it is going to take a decent amount of work to move all of
>>>>>>> the recipes over from the existing cookbook to individual notebooks
>>>>>>> --Bulking of repo: In the current paradigm, images associated with
>>>>>>> each recipe are generated dynamically on the server by executing each
>>>>>>> script, thereby minimizing the number of files that need to be tracked by
>>>>>>> mercurial.  By moving to a notebook with images that are embedded in each
>>>>>>> notebook, we'd potentially increase the footprint of the repository
>>>>>>> substantially, especially if there were frequent updates of individual
>>>>>>> recipes.
>>>>>>>
>>>>>>> I also like the yt bootcamp notebooks that Matt put together a year
>>>>>>> ago.  I think they are great for getting new users up to speed on how to
>>>>>>> use various aspects of the code.  Perhaps this notebook could make its way
>>>>>>> into the beginning of the cookbook for a more streamlined approach to the
>>>>>>> documentation?
>>>>>>>
>>>>>>> So now is your chance to vote:
>>>>>>>
>>>>>>> Move cookbook to ipython notebooks? +/- 0-1?
>>>>>>>
>>>>>>> Move yt bootcamp to cookbook? +/- 0-1?
>>>>>>>
>>>>>>> Comments?  Suggestions?
>>>>>>>
>>>>>>> Cameron
>>>>>>>
>>>>>>> --
>>>>>>> Cameron Hummels
>>>>>>> Postdoctoral Researcher
>>>>>>> Steward Observatory
>>>>>>> University of Arizona
>>>>>>> http://chummels.org
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> yt-dev mailing list
>>>>>>> yt-dev at lists.spacepope.org
>>>>>>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>>>>>>>
>>>>>>>
>>>>>> _______________________________________________
>>>>>> yt-dev mailing list
>>>>>> yt-dev at lists.spacepope.org
>>>>>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>>>>>>
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> yt-dev mailing list
>>>>>> yt-dev at lists.spacepope.org
>>>>>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>>>>>>
>>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> yt-dev mailing list
>>>>> yt-dev at lists.spacepope.org
>>>>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>>>>>
>>>>>
>>>>
>>>> _______________________________________________
>>>> yt-dev mailing list
>>>> yt-dev at lists.spacepope.org
>>>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>>>>
>>>>
>>>
>>> _______________________________________________
>>> yt-dev mailing list
>>> yt-dev at lists.spacepope.org
>>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>>>
>>>
>> _______________________________________________
>> yt-dev mailing list
>> yt-dev at lists.spacepope.org
>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>>
>>
>>
>> _______________________________________________
>> yt-dev mailing list
>> yt-dev at lists.spacepope.org
>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>>
>>
>
> _______________________________________________
> yt-dev mailing list
> yt-dev at lists.spacepope.org
> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>
>


-- 
Cameron Hummels
Postdoctoral Researcher
Steward Observatory
University of Arizona
http://chummels.org
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.spacepope.org/pipermail/yt-dev-spacepope.org/attachments/20131028/a388412c/attachment.html>


More information about the yt-dev mailing list