[yt-dev] Fw: Re: [numfocus] Community build service (starting with Windows)

Anthony Scopatz scopatz at gmail.com
Fri Nov 15 10:44:05 PST 2013


Hey J,

Sadly there are no primers on this particular issue because it is not a
solved problem!  Of course, you can pick up the cmake / pip / autoconf /
brew / ports manuals and read through those.  This will provide a basis for
understanding how people have solves this problem when they have complete
control over the environment.  What it doesn't provide is the context for
users running into issues trying to install shard object libraries on
platforms that don't natively support a compiler ;).  For that, you need
experience and to join mailing lists where this comes up, such as yt and
(shameless plug) NumFOCUS.

Be Well
Anthony


On Thu, Nov 14, 2013 at 12:12 PM, j s oishi <jsoishi at gmail.com> wrote:

> I am very, very interested in this discussion, but I am an absolute
> neophyte when it comes to build systems, installing, and so on. Is there a
> good primer on these issues that I could look at?
>
> j
>
>
> On Thu, Nov 14, 2013 at 11:59 AM, Anthony Scopatz <scopatz at gmail.com>wrote:
>
>> Thanks for fwding this along Nathan.
>> On Nov 14, 2013 3:25 AM, "Nathan Goldbaum" <nathan12343 at gmail.com> wrote:
>>
>>> There is a very interesting conversation going on over on the numfocus
>>> list about a distributed build service for the numfocus ecosystem:
>>> https://groups.google.com/forum/#!topic/numfocus/mVNakFqfpZg
>>>
>>> I’m forwarding this message from Anthony since he mentions yt.  I’m
>>> hopeful that in the end this discussion leads to something concrete and
>>> useful for our users.
>>>
>>> On November 14, 2013 at 12:16:44 AM, Anthony Scopatz (scopatz at gmail.com<//scopatz at gmail.com>)
>>> wrote:
>>>
>>> Hello All,
>>>
>>> I'd like to volunteer to be the person to spearhead this effort, or at
>>> least be the manager/organizer of this effort. Normally I wouldn't do this
>>> (I have enough on my plate) but the build and distribution problems have
>>> become *the* central issue for PyNE <http://pynesim.org/> in the past
>>> 10 days [1]. On Monday Nov. 4th, Katy hosted a PyNE workshop at UC
>>> Berkeley.  Out of the ~15 attendees, 2 could successfully install
>>> PyNE...even using anaconda and/or canopy and ignoring our optional
>>> dependencies. *We absolutely have to solve this problem for the project
>>> to survive.* The PyNE stack fairly standard for this community.  The
>>> fact that we have had such terrible install problems is really a serious
>>> issue.  We are going to dedicate the next release to fixing this problem *rather
>>> than* doing any nuclear engineering.
>>>
>>> Furthermore, I don't think that this is a wholly unique situation as I
>>> know other codes, such as yt, have experienced similar difficulties in this
>>> space.
>>>
>>> I believe that to fix this problem sufficiently we must have empathy for
>>> our users, rather than for ourselves as developers.  We need to be open to
>>> installing packages in the way the user is comfortable with even though it
>>> may make more work for us.
>>>
>>> Many of the problems I feel stem from wanting to support a public API
>>> for many languages simultaneously (C/C++/Python/Fortran).  PyNE is not
>>> unique in this way.  NumPy also does this.  However, the dependency pool
>>> for PyNE is much broader and deeper than for NumPy.  Additionally as Aaron
>>> brings up, we aren't really just talking about supporting Python & Cython
>>> code.  By virtue of our dependencies, we also have to support packages in
>>> these compiled libraries.
>>>
>>> I have many ideas about how to go about this process, though it will
>>> require help from many people.  I believe all of the pieces are there to do
>>> this right, but what we need is a coordinated strategy, someone to carve up
>>> tasks appropriately, and someone to make sure that this process doesn't
>>> take too long.  Again, I am happy to be the cat-wrangler, if for no other
>>> reason than that PyNE will be going through this shortly.
>>>
>>> Travis-ci, vms, etc, have come up quite a bit so far on this thread.  As
>>> others have noted, this is the right idea but insufficiently executed.
>>>  Therefore, I'd like to direct folks to the Build & Test Lab (BaTLab)<https://www.batlab.org/>.
>>> This is an NSF funded organization whose purpose it is to provide an
>>> integrated suite of platforms <https://www.batlab.org/?q=platforms> for
>>> building and testing scientific software.  Even though it is at UW-Madison,
>>> it is open to everyone and free to use.  The advantage here is that BaTLab
>>> jobs can take as long as they need to and it already supports the platforms
>>> we need in many flavors: Windows, Mac, Linux.  For another project
>>> (cyclus), I am working on a little web app<https://github.com/cyclus/polyphemus>which exposes Travis-CI-like functionality but is backended by BaTLab.
>>>  Hopefully this will be in a working state by this weekend.  *Basically,
>>> there is no need for us to build the infrastructure!* Or at lease no
>>> need to do so in the short- to medium-term.
>>>
>>> I think if every project were able to donate a person or two to work on
>>> this for the next month we really could get something working that truly
>>> addressed or routed around some of the fundamental flaws with the current
>>> system.
>>>
>>> In summary, we (PyNE +/- yt) can run off and solve this for ourselves
>>> -OR- we (NumFOCUS) can help pool our resources and try to create a quality
>>> solution that applies to everyone.  Sorry if this is a bit ranty, but it
>>> has become difficult for me to sell Python & PyNE because of building and
>>> distribution.  This is long overdue and I am willing to put my money where
>>> my mouth is.  If someone has better ideas, I'd love to hear them!
>>>
>>> Be Well
>>> Anthony
>>>
>>> 1. https://groups.google.com/d/topic/pyne-dev/RxNCHvZAAEQ/discussion
>>>
>>>
>>> _______________________________________________
>>> yt-dev mailing list
>>> yt-dev at lists.spacepope.org
>>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>>>
>>>
>> _______________________________________________
>> yt-dev mailing list
>> yt-dev at lists.spacepope.org
>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>>
>>
>
> _______________________________________________
> yt-dev mailing list
> yt-dev at lists.spacepope.org
> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.spacepope.org/pipermail/yt-dev-spacepope.org/attachments/20131115/aec0e5b7/attachment.html>


More information about the yt-dev mailing list