[yt-dev] Get rid of hdf5_light_reader and freetype

John ZuHone jzuhone at gmail.com
Tue Oct 8 14:13:07 PDT 2013


For the reasons expressed very well by Nathan, I am +1 on this change.

John ZuHone
Laboratory for High-Energy Astrophysics
NASA/Goddard Space Flight Center
8800 Greenbelt Rd., Code 662
Greenbelt, MD 20771
(w) 301-286-2531
(m) 773-758-0172
jzuhone at gmail.com
john.zuhone at nasa.gov

> On Oct 8, 2013, at 5:10 PM, Nathan Goldbaum <nathan12343 at gmail.com> wrote:
> 
> With anaconda in particular, it's a real pain if we have to link against external libraries due to binary compatibility issues they don't fully deal with.  For example, hdf5 linking is broken with the version of hdf5 delivered with anaconda on OS X for (at least) OS X 10.8 because the anaconda buildbot Continuum uses for anaconda releases is a 10.5 machine.
> 
> What this means in practice is that yt will be both pip-installable or conda-buildable with a minimum of fuss.  It's still possible there will be binary compatibility issues with libpng, but my experiments with anaconda-based packaging point to hdf5 as the primary pain point.
> 
> Since ContinuumIO packages hdf5 as well as h5py and pytables, we don't need to worry about linking against hdf5 at all in a universe where we move to miniconda or anaconda-based yt installations.
> 
> 
>> On Tue, Oct 8, 2013 at 2:02 PM, Matthew Turk <matthewturk at gmail.com> wrote:
>> On Tue, Oct 8, 2013 at 5:00 PM, Britton Smith <brittonsmith at gmail.com> wrote:
>> > Hi Matt,
>> >
>> > I'm definitely for this.  However, won't we will still have hdf5 as a
>> > dependency since we need it to install h5py?
>> >
>> 
>> Yup -- but it means it's an indirect dependency, and also a runtime
>> rather than buildtime dependency, so it means we don't need to worry
>> about having particular types of the library around and whatnot.  So
>> it'll still be in the install_script.sh, but when we use Conda we can
>> simplify things considerably as we can just use h5py on the system.
>> 
>> > Britton
>> >
>> >
>> > On Tue, Oct 8, 2013 at 9:44 PM, Matthew Turk <matthewturk at gmail.com> wrote:
>> >>
>> >> Well, I should be clear -- loading a gigantic dataset in yt-2 is
>> >> slower than either forms of the yt-3 loading, with hdf5_light_reader
>> >> or with h5py.  So the order goes:
>> >>
>> >> yt-3.0 + hdf5_light_reader (fastest)
>> >> yt-3.0 + h5py (not bad)
>> >> yt-2.x + hdf5_light_reader (slowest)
>> >>
>> >> (Also, I'd encourage anybody who wants to see "write only code" to
>> >> take a look at hdf5_light_reader.)
>> >>
>> >> -Matt
>> >>
>> >> On Tue, Oct 8, 2013 at 4:30 PM, Cameron Hummels <chummels at gmail.com>
>> >> wrote:
>> >> > I'm all for it if it results in faster IO for large datasets without any
>> >> > significant downsides for medium/small datasets.  Plus if it simplifies
>> >> > things, that's always better too.  Thanks for looking into this, Matt!
>> >> >
>> >> > +1
>> >> >
>> >> > Cameron
>> >> >
>> >> >
>> >> > On Tue, Oct 8, 2013 at 1:20 PM, Matthew Turk <matthewturk at gmail.com>
>> >> > wrote:
>> >> >>
>> >> >> Hi all,
>> >> >>
>> >> >> One of the biggest pain points in our distribution is linking against
>> >> >> external libraries.  We do this in three places by default:
>> >> >>
>> >> >>  * HDF5, for the hdf5_light_reader
>> >> >>  * libpng, for write_png
>> >> >>  * freetype, for freetype_writer
>> >> >>
>> >> >> hdf5_light_reader was originally written (by me) something like five
>> >> >> years and change ago, because at the time PyTables was slower than I
>> >> >> wanted.  I've been working on improving IO and found that we can get
>> >> >> nearly the same speed from h5py; when combined with overall
>> >> >> performance improvements in yt-3, this still translates to faster
>> >> >> speeds for very large datasets.  On small to medium datasets, the
>> >> >> performance is not noticeably different.  I'd like to propose that we
>> >> >> stop using hdf5_light_reader, which may result in a slight degradation
>> >> >> of performance for (only) Enzo IO.
>> >> >>
>> >> >> Additionally, I don't think *anyone* uses the freetype writer, and I'd
>> >> >> like to remove it.  I'm actually not sure if anyone has ever used it.
>> >> >>
>> >> >> I think with these two changes we can likely improve our packaging
>> >> >> situation by a good amount, as hdf5 specifically is the main problem
>> >> >> with distribution in Conda.  These changes would only be in yt 3.0,
>> >> >> and would not affect 2.x.
>> >> >>
>> >> >> [+-][01]?
>> >> >>
>> >> >> -Matt
>> >> >> _______________________________________________
>> >> >> yt-dev mailing list
>> >> >> yt-dev at lists.spacepope.org
>> >> >> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>> >> >
>> >> >
>> >> >
>> >> >
>> >> > --
>> >> > Cameron Hummels
>> >> > Postdoctoral Researcher
>> >> > Steward Observatory
>> >> > University of Arizona
>> >> > http://chummels.org
>> >> >
>> >> > _______________________________________________
>> >> > yt-dev mailing list
>> >> > yt-dev at lists.spacepope.org
>> >> > http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>> >> >
>> >> _______________________________________________
>> >> yt-dev mailing list
>> >> yt-dev at lists.spacepope.org
>> >> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>> >
>> >
>> >
>> > _______________________________________________
>> > yt-dev mailing list
>> > yt-dev at lists.spacepope.org
>> > http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>> >
>> _______________________________________________
>> yt-dev mailing list
>> yt-dev at lists.spacepope.org
>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
> 
> _______________________________________________
> yt-dev mailing list
> yt-dev at lists.spacepope.org
> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.spacepope.org/pipermail/yt-dev-spacepope.org/attachments/20131008/082b9fe0/attachment.html>


More information about the yt-dev mailing list