[yt-dev] conda and external libraries

Matthew Turk matthewturk at gmail.com
Tue Oct 22 09:30:58 PDT 2013


Hi Jeff,

I agree. This is a common use case and something we should consider as we
evaluate distributing via Conda.

The separation between user and developer has, by design, always been low
for yt. Part of this has been shipping the VCS, cloning, providing
auto-update and so on. From what I can tell, this is harder with conda,
because building a new package and installing it is requires using Conda
itself. (Please correct me if I am wrong.) This also affects what you are
describing: Conda as an environment only works internally, and I think this
may even be by design.

So I am in principle in agreement with Nathan: Conda works when we are
attempting to distribute yt, as its own thing. It doesn't seem to work as a
generic scientific environment, and whether that is by design or by
coincidence I can't quite say. It all seems to come down to the linking of
external libraries.

I think that regardless of conda, we should continue to maintain a
slimmed-down install script.

But as for Conda itself, it does seem likely to me that for isolated
instances one could create build recipes. But that's not sustainable.
(First example: mpi4py with SC-center built interconnect support.)

So I don't know. This seems like a good thing to being up with Continuum,
using the concrete examples of linking of non-conda against conda or
linking conda packages against SC packages. I would support exploring this
with them, but the more specific the issue the better it can be described.

I wonder if this could be fixed if the whole conda environment could be
bootstrapped from source?

Matt
On Oct 22, 2013 12:15 PM, "j s oishi" <jsoishi at gmail.com> wrote:

> Nathan,
>
> I'm sorry. I was not at all clear. Thanks for pointing out that FFTW could
> have been built against the libm that conda provides. The problem I was
> trying to get at is that there are probably a lot of scenarios where we
> *don't* want to rely on whatever library conda provides but we *do* want to
> link that library into the yt python somehow. I was thinking specifically
> about deploying on supercomputing resources where, for example, we'd want
> mpi4py to interface NOT with whatever mpi library conda provides but with
> the one the supercomputing center provides. Another key example (perhaps
> not super relevant to yt at the moment) is linear algebra libraries. I
> guess what I was really asking is if there is a way to make conda talk to
> system libraries better than it does, or if that's simply not part of how
> it is intended to operate. From what I understand, conda is designed to
> deploy python and its scientific appendages easily on a given hardware
> resource; I would think this would include interfacing with provided high
> performance libraries, but perhaps that is just not part of their use case.
>
> If it is the case that conda cannot help with this problem, I wonder how
> this is going to shake out for yt, since we need to *both* have an easy
> install for a diverse set of people where performance may not be driven by
> external libraries, but we also have a situation where it might: for
> example, we increasingly will want to do analysis on machines with somewhat
> unusual MPI implementations (kraken, for example). Relying on the centers
> to deploy these resources is one way of dealing with it, but perhaps not
> the best one given the rapid updates to yt.
>
> j
>
>
> On Tue, Oct 22, 2013 at 11:59 AM, Nathan Goldbaum <nathan12343 at gmail.com>wrote:
>
>> I believe the issue is that your FFTW wrappers are linked against your
>> system libm while you should be linking then against the libraries provided
>> by conda if you want to use it with that python install.
>>
>> You might want to consider writing a conda recipe for your wrappers.
>>  Building a conda package from such a recipe will ensure that it is linking
>> with anaconda's libraries since conda package builds happen in an isolated
>> environment.
>>
>> I agree that there are issues on Linux with anaconda and binary
>> compatibility and that it's not totally obvious to newcomers how to use
>> anaconda python if they also want to build their own packages under it.
>>  However, given that it simplifies the installation process and gives users
>> an out-of-the box yt install with significantly less hassle than the
>> install script and with very little maintenance on our end, I still
>> think it's the way to go moving forward.
>>
>> If you're going to be doing a lot of development, anaconda's python may
>> not be the best choice.  Instead, you might consider building yt and all
>> it's dependencies using the Ubuntu's system python and packages.
>>
>> Cheers,
>>
>> Nathan
>>
>> On Tuesday, October 22, 2013, j s oishi wrote:
>>
>>> daikaiju:/data/../96_Ra10000 [10:39]$ python 2d_gmode_kx1_kz1.py
>>>
>>>
>>> Traceback (most recent call last):
>>>   File "2d_gmode_kx1_kz1.py", line 26, in <module>
>>>     from dedalus.mods import *
>>>   File "/home/jsoishi/hg-projects/dedalus/dedalus/mods.py", line 30, in
>>> <module>
>>>     from dedalus.utils.logger import mylog
>>>   File "/home/jsoishi/hg-projects/dedalus/dedalus/utils/logger.py", line
>>> 30, in <module>
>>>     from dedalus.utils.parallelism import com_sys
>>>   File "/home/jsoishi/hg-projects/dedalus/dedalus/utils/parallelism.py",
>>> line 52, in <module>
>>>     com_sys = CommunicationSystem()
>>>   File "/home/jsoishi/hg-projects/dedalus/dedalus/utils/parallelism.py",
>>> line 43, in __init__
>>>     from dedalus.utils.fftw import fftw
>>> ImportError: /home/jsoishi/build/yt-conda/bin/../lib/libm.so.6: version
>>> `GLIBC_2.15' not found (required by /home/jsoishi/build/lib/libfftw3.so.3)
>>>
>>>
>>>
>>> On Tue, Oct 22, 2013 at 11:39 AM, Nathan Goldbaum <nathan12343 at gmail.com
>>> > wrote:
>>>
>>>> Hi Jeff,
>>>>
>>>> Can you paste the error messages you're seeing?
>>>>
>>>> Nathan
>>>>
>>>>
>>>> On Tuesday, October 22, 2013, j s oishi wrote:
>>>>
>>>>> Hi all,
>>>>>
>>>>> So I've been playing with the new conda installed yt. In principle, I
>>>>> really like the idea: a simple, completely isolated python distribution
>>>>> that does dependency management. However, I'm running into some problems
>>>>> that are potentially big problems for yt down the road. My issue is that
>>>>> I'm trying to install Dedalus into the yt-provided conda install. However,
>>>>> to do that, I need to interface with external libraries--ones that I do
>>>>> *not* want to have built as conda packages. The two that come up first are
>>>>> MPI and FFTW. Since the Dedalus FFTW bindings are actually used in yt for
>>>>> its FFT analysis, this is relevant for yt too.
>>>>>
>>>>> On a supercomputing center, one would very much want to use their MPI
>>>>> and FFT libraries, rather than some binary blob from conda, I think we need
>>>>> to figure out how to do this. The problem I'm getting specifically is that
>>>>> if I build FFTW (for example) myself, I want to link against the system
>>>>> libm. But conda provides its OWN libm, which, it turns out is not GLIBC
>>>>> compatible with what FFTW was built against. I believe this is because libm
>>>>> from conda was not built against the same glibc as my libm, which FFTW was
>>>>> built against. I don't know how to solve this problem, since I don't really
>>>>> understand dynamic libraries, nor conda. Does anyone have any ideas about
>>>>> this? Is this something I should take up with the conda list?
>>>>>
>>>>> My apologies if this is unclear. I am not fully sure I understand what
>>>>> is happening myself.
>>>>>
>>>>> thanks,
>>>>>
>>>>> j
>>>>>
>>>>
>>>> _______________________________________________
>>>> yt-dev mailing list
>>>> yt-dev at lists.spacepope.org
>>>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>>>>
>>>>
>>>
>> _______________________________________________
>> yt-dev mailing list
>> yt-dev at lists.spacepope.org
>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>>
>>
>
> _______________________________________________
> yt-dev mailing list
> yt-dev at lists.spacepope.org
> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.spacepope.org/pipermail/yt-dev-spacepope.org/attachments/20131022/466a3d79/attachment.htm>


More information about the yt-dev mailing list