[yt-dev] Fw: Re: [numfocus] Community build service (starting with Windows)

Nathan Goldbaum nathan12343 at gmail.com
Thu Nov 14 00:25:31 PST 2013


There is a very interesting conversation going on over on the numfocus list about a distributed build service for the numfocus ecosystem: https://groups.google.com/forum/#!topic/numfocus/mVNakFqfpZg

I’m forwarding this message from Anthony since he mentions yt.  I’m hopeful that in the end this discussion leads to something concrete and useful for our users.

On November 14, 2013 at 12:16:44 AM, Anthony Scopatz (scopatz at gmail.com) wrote:

Hello All, 

I'd like to volunteer to be the person to spearhead this effort, or at least be the manager/organizer of this effort. Normally I wouldn't do this (I have enough on my plate) but the build and distribution problems have become the central issue for PyNE in the past 10 days [1]. On Monday Nov. 4th, Katy hosted a PyNE workshop at UC Berkeley.  Out of the ~15 attendees, 2 could successfully install PyNE...even using anaconda and/or canopy and ignoring our optional dependencies. We absolutely have to solve this problem for the project to survive. The PyNE stack fairly standard for this community.  The fact that we have had such terrible install problems is really a serious issue.  We are going to dedicate the next release to fixing this problem rather than doing any nuclear engineering.

Furthermore, I don't think that this is a wholly unique situation as I know other codes, such as yt, have experienced similar difficulties in this space.

I believe that to fix this problem sufficiently we must have empathy for our users, rather than for ourselves as developers.  We need to be open to installing packages in the way the user is comfortable with even though it may make more work for us.  

Many of the problems I feel stem from wanting to support a public API for many languages simultaneously (C/C++/Python/Fortran).  PyNE is not unique in this way.  NumPy also does this.  However, the dependency pool for PyNE is much broader and deeper than for NumPy.  Additionally as Aaron brings up, we aren't really just talking about supporting Python & Cython code.  By virtue of our dependencies, we also have to support packages in these compiled libraries.

I have many ideas about how to go about this process, though it will require help from many people.  I believe all of the pieces are there to do this right, but what we need is a coordinated strategy, someone to carve up tasks appropriately, and someone to make sure that this process doesn't take too long.  Again, I am happy to be the cat-wrangler, if for no other reason than that PyNE will be going through this shortly.

Travis-ci, vms, etc, have come up quite a bit so far on this thread.  As others have noted, this is the right idea but insufficiently executed.  Therefore, I'd like to direct folks to the Build & Test Lab (BaTLab). This is an NSF funded organization whose purpose it is to provide an integrated suite of platforms for building and testing scientific software.  Even though it is at UW-Madison, it is open to everyone and free to use.  The advantage here is that BaTLab jobs can take as long as they need to and it already supports the platforms we need in many flavors: Windows, Mac, Linux.  For another project (cyclus), I am working on a little web app which exposes Travis-CI-like functionality but is backended by BaTLab.  Hopefully this will be in a working state by this weekend.  Basically, there is no need for us to build the infrastructure! Or at lease no need to do so in the short- to medium-term.

I think if every project were able to donate a person or two to work on this for the next month we really could get something working that truly addressed or routed around some of the fundamental flaws with the current system.

In summary, we (PyNE +/- yt) can run off and solve this for ourselves -OR- we (NumFOCUS) can help pool our resources and try to create a quality solution that applies to everyone.  Sorry if this is a bit ranty, but it has become difficult for me to sell Python & PyNE because of building and distribution.  This is long overdue and I am willing to put my money where my mouth is.  If someone has better ideas, I'd love to hear them!

Be Well
Anthony

1. https://groups.google.com/d/topic/pyne-dev/RxNCHvZAAEQ/discussion
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.spacepope.org/pipermail/yt-dev-spacepope.org/attachments/20131114/88de02f1/attachment.htm>


More information about the yt-dev mailing list