[yt-dev] Soliciting opinions about how to handle data tables for spectral integrator port

Nathan Goldbaum nathan12343 at gmail.com
Wed Dec 7 12:12:09 PST 2016


On Wed, Dec 7, 2016 at 1:30 PM, John Zuhone <jzuhone at gmail.com> wrote:

> Hi everyone,
>
> I just issued PR 2465 (https://bitbucket.org/yt_analysis/yt/pull-requests/
> 2465/wip-port-the-spectral_integrator-analysis/) to port the spectral
> integrator analysis module (originally written by Britton) under yt.fields.
> spectral_integrator is an analysis module that creates X-ray emission
> fields in user-specified energy bands.
>
> The hitch is that spectral_integrator uses HDF5 tables to compute the
> emissivity, since they are not analytical functions. We currently host
> those tables on yt-project.org/data. spectral_integrator downloads them
> automatically if the analysis module is used and they are not present.
>
> There was some discussion on Slack as to whether or not this is the
> correct approach, since it’s not ideal for certain computing environments
> (e.g., HPC).
>

Also a problem for e.g. packaging yt on linux distros.


> The files are a total of about 2.4 MB in size, so there is some reticence
> to bundling them with yt.
>

So my main concern here is adding them to the repo. If you could figure out
a way to add them to the source distribution without including them in the
repo (and then issue an error with a URL for non-sdist builds of yt [e.g.
from the repo]), it wouldn't be a problem.

This isn't really a generic solution for all binary files we might want to
send to users, but 2.4 MB isn't all that big to be part of an sdist or
wheel.


> There is also the issue that uploading new versions of the tables breaks
> backward-compatibility. I have sacrificed backwards-compatiblity for
> considerable simplification of code.
>

I think the solution is just to use different names for the files or the
hdf5 datasets or groups here so the code can tell the difference.


>
> Does anyone have any ideas about the best way to handle this issue?
>
> Best,
>
> John Z
>
> _______________________________________________
> yt-dev mailing list
> yt-dev at lists.spacepope.org
> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.spacepope.org/pipermail/yt-dev-spacepope.org/attachments/20161207/3dac61ee/attachment.htm>


More information about the yt-dev mailing list