[yt-users] Development branch now requires Cython

[Matthew Turk]

Hi everyone,

tl;dr summary: In the unstable branch, Cython is now required, but it
should be auto-installed, and email me directly and off-list if you
run into any problems.

Longer:

I've just pushed a change (8e477b655c96) to the development branch,
but not the stable branch, that changes the way Cython code is handled
in yt.  Cython is a Python-like dialect that compiles down to C code.
It understands NumPy arrays and Python objects natively, and can
produce code that operates at speeds competitive with raw,
hand-written C code.  We use it in yt to write things like the volume
renderer, fast interpolation, level set identification, PNG writing,
and so on.  You can see most of the Cython code in
yt/utilities/_amr_utils/*.pyx .

In the past, the Cython code was converted to C before being put into
the repository.  This is visible in yt/utilities/amr_utils.c.  If you
take a second to open this up, you'll note a couple things about it.
The first is that it's generated code: notoriously awful to read, and
100% impossible to maintain.  That's why it's never touched directly,
and only generated by Cython.  But, the problem with that is that if
we want to make small changes, they cascade into ridiculously long
changesets and diffs, which end up growing the size of the mercurial
repository by far more than is appropriate.

To get around this, I have added an install-time dependency on the
Cython package.  To ensure that this will cause no problems during the
transition, installation of Cython was added to the install script a
while ago, and I have additionally added a check to setup.py.  If
Cython is not found, it should install it.  The only cases where this
should cause a problem are those where yt was installed with elevated
(sudo) permissions.  Now, every time yt is rebuilt, the C code that
was previously in yt/utilities/amr_utils.c is regenerated from the
Cython code in yt/utilities/_amr_utils/*.pyx.  This means that we no
longer need to update amr_utils.c in the mercurial repository.

Adding Cython brings with it a number of interesting things we can do;
these include much faster iteration on improvements to, say, the
volume renderer.  Additionally, there are some other very interesting
things one can do with the speed improvements Cython brings, which
hopefully we can explore in the future.

Please email me directly if you have any problems.

For a bit more about this, and where we discussed it, you can see the
yt-dev mailing list thread about it:

http://lists.spacepope.org/pipermail/yt-dev-spacepope.org/2011-January/001080.html

For more about Cython:

http://www.cython.org/

Best,

Matt