[yt-dev] Get rid of hdf5_light_reader and freetype

[Matthew Turk]

Now in pull request form:

https://bitbucket.org/yt_analysis/yt-3.0/pull-request/114/removing-freetype-and-hdf5-wrappers/diff

Also, for what it's worth, I dropped Enzo down to using the low-level
h5py API and it's now speed-competitive with hdf5_light_reader; it's
something like 10% slower is all.  And, still faster than 2.X.

-Matt

On Tue, Oct 8, 2013 at 5:13 PM, John ZuHone <jzuhone at gmail.com> wrote:
> For the reasons expressed very well by Nathan, I am +1 on this change.
>
> John ZuHone
> Laboratory for High-Energy Astrophysics
> NASA/Goddard Space Flight Center
> 8800 Greenbelt Rd., Code 662
> Greenbelt, MD 20771
> (w) 301-286-2531
> (m) 773-758-0172
> jzuhone at gmail.com
> john.zuhone at nasa.gov
>
> On Oct 8, 2013, at 5:10 PM, Nathan Goldbaum <nathan12343 at gmail.com> wrote:
>
> With anaconda in particular, it's a real pain if we have to link against
> external libraries due to binary compatibility issues they don't fully deal
> with.  For example, hdf5 linking is broken with the version of hdf5
> delivered with anaconda on OS X for (at least) OS X 10.8 because the
> anaconda buildbot Continuum uses for anaconda releases is a 10.5 machine.
>
> What this means in practice is that yt will be both pip-installable or
> conda-buildable with a minimum of fuss.  It's still possible there will be
> binary compatibility issues with libpng, but my experiments with
> anaconda-based packaging point to hdf5 as the primary pain point.
>
> Since ContinuumIO packages hdf5 as well as h5py and pytables, we don't need
> to worry about linking against hdf5 at all in a universe where we move to
> miniconda or anaconda-based yt installations.
>
>
> On Tue, Oct 8, 2013 at 2:02 PM, Matthew Turk <matthewturk at gmail.com> wrote:
>>
>> On Tue, Oct 8, 2013 at 5:00 PM, Britton Smith <brittonsmith at gmail.com>
>> wrote:
>> > Hi Matt,
>> >
>> > I'm definitely for this.  However, won't we will still have hdf5 as a
>> > dependency since we need it to install h5py?
>> >
>>
>> Yup -- but it means it's an indirect dependency, and also a runtime
>> rather than buildtime dependency, so it means we don't need to worry
>> about having particular types of the library around and whatnot.  So
>> it'll still be in the install_script.sh, but when we use Conda we can
>> simplify things considerably as we can just use h5py on the system.
>>
>> > Britton
>> >
>> >
>> > On Tue, Oct 8, 2013 at 9:44 PM, Matthew Turk <matthewturk at gmail.com>
>> > wrote:
>> >>
>> >> Well, I should be clear -- loading a gigantic dataset in yt-2 is
>> >> slower than either forms of the yt-3 loading, with hdf5_light_reader
>> >> or with h5py.  So the order goes:
>> >>
>> >> yt-3.0 + hdf5_light_reader (fastest)
>> >> yt-3.0 + h5py (not bad)
>> >> yt-2.x + hdf5_light_reader (slowest)
>> >>
>> >> (Also, I'd encourage anybody who wants to see "write only code" to
>> >> take a look at hdf5_light_reader.)
>> >>
>> >> -Matt
>> >>
>> >> On Tue, Oct 8, 2013 at 4:30 PM, Cameron Hummels <chummels at gmail.com>
>> >> wrote:
>> >> > I'm all for it if it results in faster IO for large datasets without
>> >> > any
>> >> > significant downsides for medium/small datasets.  Plus if it
>> >> > simplifies
>> >> > things, that's always better too.  Thanks for looking into this,
>> >> > Matt!
>> >> >
>> >> > +1
>> >> >
>> >> > Cameron
>> >> >
>> >> >
>> >> > On Tue, Oct 8, 2013 at 1:20 PM, Matthew Turk <matthewturk at gmail.com>
>> >> > wrote:
>> >> >>
>> >> >> Hi all,
>> >> >>
>> >> >> One of the biggest pain points in our distribution is linking
>> >> >> against
>> >> >> external libraries.  We do this in three places by default:
>> >> >>
>> >> >>  * HDF5, for the hdf5_light_reader
>> >> >>  * libpng, for write_png
>> >> >>  * freetype, for freetype_writer
>> >> >>
>> >> >> hdf5_light_reader was originally written (by me) something like five
>> >> >> years and change ago, because at the time PyTables was slower than I
>> >> >> wanted.  I've been working on improving IO and found that we can get
>> >> >> nearly the same speed from h5py; when combined with overall
>> >> >> performance improvements in yt-3, this still translates to faster
>> >> >> speeds for very large datasets.  On small to medium datasets, the
>> >> >> performance is not noticeably different.  I'd like to propose that
>> >> >> we
>> >> >> stop using hdf5_light_reader, which may result in a slight
>> >> >> degradation
>> >> >> of performance for (only) Enzo IO.
>> >> >>
>> >> >> Additionally, I don't think *anyone* uses the freetype writer, and
>> >> >> I'd
>> >> >> like to remove it.  I'm actually not sure if anyone has ever used
>> >> >> it.
>> >> >>
>> >> >> I think with these two changes we can likely improve our packaging
>> >> >> situation by a good amount, as hdf5 specifically is the main problem
>> >> >> with distribution in Conda.  These changes would only be in yt 3.0,
>> >> >> and would not affect 2.x.
>> >> >>
>> >> >> [+-][01]?
>> >> >>
>> >> >> -Matt
>> >> >> _______________________________________________
>> >> >> yt-dev mailing list
>> >> >> yt-dev at lists.spacepope.org
>> >> >> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>> >> >
>> >> >
>> >> >
>> >> >
>> >> > --
>> >> > Cameron Hummels
>> >> > Postdoctoral Researcher
>> >> > Steward Observatory
>> >> > University of Arizona
>> >> > http://chummels.org
>> >> >
>> >> > _______________________________________________
>> >> > yt-dev mailing list
>> >> > yt-dev at lists.spacepope.org
>> >> > http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>> >> >
>> >> _______________________________________________
>> >> yt-dev mailing list
>> >> yt-dev at lists.spacepope.org
>> >> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>> >
>> >
>> >
>> > _______________________________________________
>> > yt-dev mailing list
>> > yt-dev at lists.spacepope.org
>> > http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>> >
>> _______________________________________________
>> yt-dev mailing list
>> yt-dev at lists.spacepope.org
>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>
>
> _______________________________________________
> yt-dev mailing list
> yt-dev at lists.spacepope.org
> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>
>
> _______________________________________________
> yt-dev mailing list
> yt-dev at lists.spacepope.org
> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>