[yt-dev] Get rid of hdf5_light_reader and freetype

Matthew Turk matthewturk at gmail.com
Tue Oct 8 13:44:38 PDT 2013


Well, I should be clear -- loading a gigantic dataset in yt-2 is
slower than either forms of the yt-3 loading, with hdf5_light_reader
or with h5py.  So the order goes:

yt-3.0 + hdf5_light_reader (fastest)
yt-3.0 + h5py (not bad)
yt-2.x + hdf5_light_reader (slowest)

(Also, I'd encourage anybody who wants to see "write only code" to
take a look at hdf5_light_reader.)

-Matt

On Tue, Oct 8, 2013 at 4:30 PM, Cameron Hummels <chummels at gmail.com> wrote:
> I'm all for it if it results in faster IO for large datasets without any
> significant downsides for medium/small datasets.  Plus if it simplifies
> things, that's always better too.  Thanks for looking into this, Matt!
>
> +1
>
> Cameron
>
>
> On Tue, Oct 8, 2013 at 1:20 PM, Matthew Turk <matthewturk at gmail.com> wrote:
>>
>> Hi all,
>>
>> One of the biggest pain points in our distribution is linking against
>> external libraries.  We do this in three places by default:
>>
>>  * HDF5, for the hdf5_light_reader
>>  * libpng, for write_png
>>  * freetype, for freetype_writer
>>
>> hdf5_light_reader was originally written (by me) something like five
>> years and change ago, because at the time PyTables was slower than I
>> wanted.  I've been working on improving IO and found that we can get
>> nearly the same speed from h5py; when combined with overall
>> performance improvements in yt-3, this still translates to faster
>> speeds for very large datasets.  On small to medium datasets, the
>> performance is not noticeably different.  I'd like to propose that we
>> stop using hdf5_light_reader, which may result in a slight degradation
>> of performance for (only) Enzo IO.
>>
>> Additionally, I don't think *anyone* uses the freetype writer, and I'd
>> like to remove it.  I'm actually not sure if anyone has ever used it.
>>
>> I think with these two changes we can likely improve our packaging
>> situation by a good amount, as hdf5 specifically is the main problem
>> with distribution in Conda.  These changes would only be in yt 3.0,
>> and would not affect 2.x.
>>
>> [+-][01]?
>>
>> -Matt
>> _______________________________________________
>> yt-dev mailing list
>> yt-dev at lists.spacepope.org
>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>
>
>
>
> --
> Cameron Hummels
> Postdoctoral Researcher
> Steward Observatory
> University of Arizona
> http://chummels.org
>
> _______________________________________________
> yt-dev mailing list
> yt-dev at lists.spacepope.org
> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>



More information about the yt-dev mailing list