<html><head><meta http-equiv="content-type" content="text/html; charset=utf-8"></head><body dir="auto"><div>For the reasons expressed very well by Nathan, I am +1 on this change.<br><br>John ZuHone<div>Laboratory for High-Energy Astrophysics<br><div>NASA/Goddard Space Flight Center</div><div>8800 Greenbelt Rd., Code 662</div><div>Greenbelt, MD 20771</div><div>(w) 301-286-2531</div><div>(m) 773-758-0172</div></div><div><a href="mailto:jzuhone@gmail.com">jzuhone@gmail.com</a></div><div><a href="mailto:john.zuhone@nasa.gov">john.zuhone@nasa.gov</a></div></div><div><br>On Oct 8, 2013, at 5:10 PM, Nathan Goldbaum <<a href="mailto:nathan12343@gmail.com">nathan12343@gmail.com</a>> wrote:<br><br></div><blockquote type="cite"><div><div dir="ltr">With anaconda in particular, it's a real pain if we have to link against external libraries due to binary compatibility issues they don't fully deal with. For example, hdf5 linking is broken with the version of hdf5 delivered with anaconda on OS X for (at least) OS X 10.8 because the anaconda buildbot Continuum uses for anaconda releases is a 10.5 machine.<div>
<br></div><div>What this means in practice is that yt will be both pip-installable or conda-buildable with a minimum of fuss. It's still possible there will be binary compatibility issues with libpng, but my experiments with anaconda-based packaging point to hdf5 as the primary pain point.</div>
<div><br></div><div>Since ContinuumIO packages hdf5 as well as h5py and pytables, we don't need to worry about linking against hdf5 at all in a universe where we move to miniconda or anaconda-based yt installations.</div>
</div><div class="gmail_extra"><br><br><div class="gmail_quote">On Tue, Oct 8, 2013 at 2:02 PM, Matthew Turk <span dir="ltr"><<a href="mailto:matthewturk@gmail.com" target="_blank">matthewturk@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="im">On Tue, Oct 8, 2013 at 5:00 PM, Britton Smith <<a href="mailto:brittonsmith@gmail.com">brittonsmith@gmail.com</a>> wrote:<br>
> Hi Matt,<br>
><br>
> I'm definitely for this. However, won't we will still have hdf5 as a<br>
> dependency since we need it to install h5py?<br>
><br>
<br>
</div>Yup -- but it means it's an indirect dependency, and also a runtime<br>
rather than buildtime dependency, so it means we don't need to worry<br>
about having particular types of the library around and whatnot. So<br>
it'll still be in the install_script.sh, but when we use Conda we can<br>
simplify things considerably as we can just use h5py on the system.<br>
<div class="HOEnZb"><div class="h5"><br>
> Britton<br>
><br>
><br>
> On Tue, Oct 8, 2013 at 9:44 PM, Matthew Turk <<a href="mailto:matthewturk@gmail.com">matthewturk@gmail.com</a>> wrote:<br>
>><br>
>> Well, I should be clear -- loading a gigantic dataset in yt-2 is<br>
>> slower than either forms of the yt-3 loading, with hdf5_light_reader<br>
>> or with h5py. So the order goes:<br>
>><br>
>> yt-3.0 + hdf5_light_reader (fastest)<br>
>> yt-3.0 + h5py (not bad)<br>
>> yt-2.x + hdf5_light_reader (slowest)<br>
>><br>
>> (Also, I'd encourage anybody who wants to see "write only code" to<br>
>> take a look at hdf5_light_reader.)<br>
>><br>
>> -Matt<br>
>><br>
>> On Tue, Oct 8, 2013 at 4:30 PM, Cameron Hummels <<a href="mailto:chummels@gmail.com">chummels@gmail.com</a>><br>
>> wrote:<br>
>> > I'm all for it if it results in faster IO for large datasets without any<br>
>> > significant downsides for medium/small datasets. Plus if it simplifies<br>
>> > things, that's always better too. Thanks for looking into this, Matt!<br>
>> ><br>
>> > +1<br>
>> ><br>
>> > Cameron<br>
>> ><br>
>> ><br>
>> > On Tue, Oct 8, 2013 at 1:20 PM, Matthew Turk <<a href="mailto:matthewturk@gmail.com">matthewturk@gmail.com</a>><br>
>> > wrote:<br>
>> >><br>
>> >> Hi all,<br>
>> >><br>
>> >> One of the biggest pain points in our distribution is linking against<br>
>> >> external libraries. We do this in three places by default:<br>
>> >><br>
>> >> * HDF5, for the hdf5_light_reader<br>
>> >> * libpng, for write_png<br>
>> >> * freetype, for freetype_writer<br>
>> >><br>
>> >> hdf5_light_reader was originally written (by me) something like five<br>
>> >> years and change ago, because at the time PyTables was slower than I<br>
>> >> wanted. I've been working on improving IO and found that we can get<br>
>> >> nearly the same speed from h5py; when combined with overall<br>
>> >> performance improvements in yt-3, this still translates to faster<br>
>> >> speeds for very large datasets. On small to medium datasets, the<br>
>> >> performance is not noticeably different. I'd like to propose that we<br>
>> >> stop using hdf5_light_reader, which may result in a slight degradation<br>
>> >> of performance for (only) Enzo IO.<br>
>> >><br>
>> >> Additionally, I don't think *anyone* uses the freetype writer, and I'd<br>
>> >> like to remove it. I'm actually not sure if anyone has ever used it.<br>
>> >><br>
>> >> I think with these two changes we can likely improve our packaging<br>
>> >> situation by a good amount, as hdf5 specifically is the main problem<br>
>> >> with distribution in Conda. These changes would only be in yt 3.0,<br>
>> >> and would not affect 2.x.<br>
>> >><br>
>> >> [+-][01]?<br>
>> >><br>
>> >> -Matt<br>
>> >> _______________________________________________<br>
>> >> yt-dev mailing list<br>
>> >> <a href="mailto:yt-dev@lists.spacepope.org">yt-dev@lists.spacepope.org</a><br>
>> >> <a href="http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org" target="_blank">http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org</a><br>
>> ><br>
>> ><br>
>> ><br>
>> ><br>
>> > --<br>
>> > Cameron Hummels<br>
>> > Postdoctoral Researcher<br>
>> > Steward Observatory<br>
>> > University of Arizona<br>
>> > <a href="http://chummels.org" target="_blank">http://chummels.org</a><br>
>> ><br>
>> > _______________________________________________<br>
>> > yt-dev mailing list<br>
>> > <a href="mailto:yt-dev@lists.spacepope.org">yt-dev@lists.spacepope.org</a><br>
>> > <a href="http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org" target="_blank">http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org</a><br>
>> ><br>
>> _______________________________________________<br>
>> yt-dev mailing list<br>
>> <a href="mailto:yt-dev@lists.spacepope.org">yt-dev@lists.spacepope.org</a><br>
>> <a href="http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org" target="_blank">http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org</a><br>
><br>
><br>
><br>
> _______________________________________________<br>
> yt-dev mailing list<br>
> <a href="mailto:yt-dev@lists.spacepope.org">yt-dev@lists.spacepope.org</a><br>
> <a href="http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org" target="_blank">http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org</a><br>
><br>
_______________________________________________<br>
yt-dev mailing list<br>
<a href="mailto:yt-dev@lists.spacepope.org">yt-dev@lists.spacepope.org</a><br>
<a href="http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org" target="_blank">http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org</a><br>
</div></div></blockquote></div><br></div>
</div></blockquote><blockquote type="cite"><div><span>_______________________________________________</span><br><span>yt-dev mailing list</span><br><span><a href="mailto:yt-dev@lists.spacepope.org">yt-dev@lists.spacepope.org</a></span><br><span><a href="http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org">http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org</a></span><br></div></blockquote></body></html>