[Yt-dev] New installation mechanism: the RAFT

[Matthew Turk]

Hi everyone,

As some of you know, I've been working on a fork/port/rebranding of
the FEMhub(dot org) framework.  FEMhub itself is a fork of SAGE, which
you can find at sagemath.org.  Basically, SAGE and FEMhub are designed
to be fully self-contained, web-based interfaces to finite element
computation systems (FEMhub) and symbolic math systems (SAGE).  Both
are pretty amazing, and SAGE itself is a strong competitor with Maple
and Mathematica.

With that out of the way, I'd like to take a moment to focus on a
couple aspects of this.  I saw this talk on FEMhub at SciPy 2009 by
Ondrej Certik (more from Ondrej here:
http://ondrejcertik.blogspot.com/2009/08/scipy-2009-conference.html )
where he described the development of FEMhub from a project called SPD
(source python distribution.)  These projects started as stripped down
versions of SAGE, but now serve as generic Python installation
mechanisms.  The upside of this is that these projects can easily
install complete python distributions, there is a large selection of
existent packages available, and they even can (optionally) present
web-interfaces to the complete python installation.  I'll return to
the idea of the web-interface in a moment, because at first I didn't
really care about that but I have since come to.  FEMhub actually has
created and includes off-screen rendering mechanisms for VTK and
OpenGL which work through the web notebook, which I think is pretty
neat.  However, you don't *have* to use it through the web -- all the
python and everything you're familiar with works out of the box, and
(best yet) it handles all the setting of library-loading and
environment variables.

So here's what's up: I've taken FEMhub and rebranded it with a new
name, RAFT.  I've stripped out a bunch of the finite element codes and
added in HDF5, h5py, and yt.  In doing so, I believe I have completely
replicated *and supplemented* the installation script, using a
framework that has been externally developed and which continues to be
developed externally.  My installation scripts pale in comparison to
the quality control that the SAGE project demands -- their system is
much more robust than what we use, and it includes more packages.  The
RAFT project now benefits from this QA effort.

In the past, we've avoided including any code in yt that even *uses*
scipy, simply because scipy is traditionally so difficult to install.
FEMhub/SAGE/RAFT simply *work*, and they come with SciPy.
Additionally, they come with the entire MayaVi distribution, which is
not only able to render, but to render off-screen the need be.  We get
a lot of fun stuff with this, and it works on a ton of different
systems without any hackery on the part of the user.

Now, if I can speak for a moment about the web notebook.  This is
something I still haven't figured out my feelings on, but I can tell
you all this: it works seamlessly with yt with no modifications, which
is more than I can say for other web-based interaction mechanisms I
have tried.  I was able to import yt, load a data file, add plots, and
then whenever "save" is called they just show up in the web browser.
Additionally, because it binds only to localhost and contains
authentication mechanisms, this is accessible through ssh forwarding
and should be considered feasible for running at supercomputer
centers.

Okay.  So.  I think the modifications all work, and I've rebranded
much of the Notebook.  I have a couple more decisions to make on the
back end about package distribution.

  * Do we install VTK/MayaVi by default?  This adds some compilation
time.  But, it also enables us to do 3D rendering more easily.  I'm
inclined to say no, because it can be automatically
downloaded/installed by the raft command.
  * Do we completely replicate the SAGE spkg distribution mechanism on
enzotools.org?  This would enable us to have finer-grained control as
well as to distribute personal packages, but we'd also have to stay up
to date with upstream bug fixes *manually*.
  * What role should the notebook play?  I was immediately and
extremely opposed to this when I first saw it; but I have since warmed
to it considerably, because of the ease-of-development, inline help,
repeatable analysis, export/import capabilities and PDF output.  It
seems likely to me that this would be a very easy mechanism for new
users, and we can distribute tutorial sheets *with* the RAFT, which
users can step through on their own to learn both Python and yt.
  * Does anyone else want to help keep up with changes upstream, help
with rebranding, and all-in-all test this out?

SO!  On to the testing part.

You can test the RAFT 0.1 alpha distribution by snagging it here:

http://raft.enzotools.org/files/raft-0.1a.tar.bz2

You should ensure that no YT files are in your PATH or LD_LIBRARY_PATH
-- we're doing this all fresh.  Then,

$ tar xvfj raft-0.1a.tar.bz2
$ cd raft-0.1a
$ make

and you should get an install, a little bit later.  On triton it was
pretty fast.  You can now set your PATH to include
/wherever/raft-0.1a/local/bin and when you run python2.6 you should
get a yt installation.  Now, you should be able to run anything you
like.  Running the command "raft" inside the raft-0.1a directory will
give you a modified IPython.  It might be fun to start that up, and
type:

notebook(port=[something])

where [something] is some unused port.  I like 7741.  On your local
machine, you can then type:

$ ssh -L 8000:localhost:7741 remote-host.org

and when you open up a web browser and point it to localhost:8000,
you'll get your notebook interface.  It's authenticated, so you'll
need a u/pw combo, but the first time you start the notebook it'll
prompt you.  You can then add plots or whatever and anaylze any data
accessible to you on that system.  When a new PNG file is added (i.e.,
via the save() command on a PlotCollection) it will get included in
the cell directly.  Be sure to ctrl-c out when you're done.  The
"raft" command is pretty powerful, and you should also be able to
install things with it.

Okay, that's a long enough email.  I'm going to work on this a little
bit more, but I'd *REALLY* appreciate it if you guys could just try
installing and using it on a machine or two that you have access to.
I intend for this to replace the installation script, but until it's
definitely working, we can't really do that.  :)

Thanks!

-Matt