[yt-dev] conda and external libraries

[j s oishi]

Nathan,

I'm sorry. I was not at all clear. Thanks for pointing out that FFTW could
have been built against the libm that conda provides. The problem I was
trying to get at is that there are probably a lot of scenarios where we
*don't* want to rely on whatever library conda provides but we *do* want to
link that library into the yt python somehow. I was thinking specifically
about deploying on supercomputing resources where, for example, we'd want
mpi4py to interface NOT with whatever mpi library conda provides but with
the one the supercomputing center provides. Another key example (perhaps
not super relevant to yt at the moment) is linear algebra libraries. I
guess what I was really asking is if there is a way to make conda talk to
system libraries better than it does, or if that's simply not part of how
it is intended to operate. From what I understand, conda is designed to
deploy python and its scientific appendages easily on a given hardware
resource; I would think this would include interfacing with provided high
performance libraries, but perhaps that is just not part of their use case.

If it is the case that conda cannot help with this problem, I wonder how
this is going to shake out for yt, since we need to *both* have an easy
install for a diverse set of people where performance may not be driven by
external libraries, but we also have a situation where it might: for
example, we increasingly will want to do analysis on machines with somewhat
unusual MPI implementations (kraken, for example). Relying on the centers
to deploy these resources is one way of dealing with it, but perhaps not
the best one given the rapid updates to yt.

j


On Tue, Oct 22, 2013 at 11:59 AM, Nathan Goldbaum <nathan12343 at gmail.com>wrote:

> I believe the issue is that your FFTW wrappers are linked against your
> system libm while you should be linking then against the libraries provided
> by conda if you want to use it with that python install.
>
> You might want to consider writing a conda recipe for your wrappers.
>  Building a conda package from such a recipe will ensure that it is linking
> with anaconda's libraries since conda package builds happen in an isolated
> environment.
>
> I agree that there are issues on Linux with anaconda and binary
> compatibility and that it's not totally obvious to newcomers how to use
> anaconda python if they also want to build their own packages under it.
>  However, given that it simplifies the installation process and gives users
> an out-of-the box yt install with significantly less hassle than the
> install script and with very little maintenance on our end, I still think
> it's the way to go moving forward.
>
> If you're going to be doing a lot of development, anaconda's python may
> not be the best choice.  Instead, you might consider building yt and all
> it's dependencies using the Ubuntu's system python and packages.
>
> Cheers,
>
> Nathan
>
> On Tuesday, October 22, 2013, j s oishi wrote:
>
>> daikaiju:/data/../96_Ra10000 [10:39]$ python 2d_gmode_kx1_kz1.py
>>
>>
>> Traceback (most recent call last):
>>   File "2d_gmode_kx1_kz1.py", line 26, in <module>
>>     from dedalus.mods import *
>>   File "/home/jsoishi/hg-projects/dedalus/dedalus/mods.py", line 30, in
>> <module>
>>     from dedalus.utils.logger import mylog
>>   File "/home/jsoishi/hg-projects/dedalus/dedalus/utils/logger.py", line
>> 30, in <module>
>>     from dedalus.utils.parallelism import com_sys
>>   File "/home/jsoishi/hg-projects/dedalus/dedalus/utils/parallelism.py",
>> line 52, in <module>
>>     com_sys = CommunicationSystem()
>>   File "/home/jsoishi/hg-projects/dedalus/dedalus/utils/parallelism.py",
>> line 43, in __init__
>>     from dedalus.utils.fftw import fftw
>> ImportError: /home/jsoishi/build/yt-conda/bin/../lib/libm.so.6: version
>> `GLIBC_2.15' not found (required by /home/jsoishi/build/lib/libfftw3.so.3)
>>
>>
>>
>> On Tue, Oct 22, 2013 at 11:39 AM, Nathan Goldbaum <nathan12343 at gmail.com>wrote:
>>
>>> Hi Jeff,
>>>
>>> Can you paste the error messages you're seeing?
>>>
>>> Nathan
>>>
>>>
>>> On Tuesday, October 22, 2013, j s oishi wrote:
>>>
>>>> Hi all,
>>>>
>>>> So I've been playing with the new conda installed yt. In principle, I
>>>> really like the idea: a simple, completely isolated python distribution
>>>> that does dependency management. However, I'm running into some problems
>>>> that are potentially big problems for yt down the road. My issue is that
>>>> I'm trying to install Dedalus into the yt-provided conda install. However,
>>>> to do that, I need to interface with external libraries--ones that I do
>>>> *not* want to have built as conda packages. The two that come up first are
>>>> MPI and FFTW. Since the Dedalus FFTW bindings are actually used in yt for
>>>> its FFT analysis, this is relevant for yt too.
>>>>
>>>> On a supercomputing center, one would very much want to use their MPI
>>>> and FFT libraries, rather than some binary blob from conda, I think we need
>>>> to figure out how to do this. The problem I'm getting specifically is that
>>>> if I build FFTW (for example) myself, I want to link against the system
>>>> libm. But conda provides its OWN libm, which, it turns out is not GLIBC
>>>> compatible with what FFTW was built against. I believe this is because libm
>>>> from conda was not built against the same glibc as my libm, which FFTW was
>>>> built against. I don't know how to solve this problem, since I don't really
>>>> understand dynamic libraries, nor conda. Does anyone have any ideas about
>>>> this? Is this something I should take up with the conda list?
>>>>
>>>> My apologies if this is unclear. I am not fully sure I understand what
>>>> is happening myself.
>>>>
>>>> thanks,
>>>>
>>>> j
>>>>
>>>
>>> _______________________________________________
>>> yt-dev mailing list
>>> yt-dev at lists.spacepope.org
>>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>>>
>>>
>>
> _______________________________________________
> yt-dev mailing list
> yt-dev at lists.spacepope.org
> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.spacepope.org/pipermail/yt-dev-spacepope.org/attachments/20131022/b1cd8212/attachment.htm>