[yt-dev] Soliciting opinions about how to handle data tables for spectral integrator port

Jason Galyardt jason.galyardt at gmail.com
Thu Dec 8 08:30:56 PST 2016


I agree with Cameron. I would simply add that, from a user's perspective,
it's nice to have a reminder to download that data if it's not currently in
the expected location.

Cheers,
Jason

----

Jason Galyardt
Dept. of Physics and Astronomy
University of Georgia


On Thu, Dec 8, 2016 at 11:26 AM, Cameron Hummels <chummels at gmail.com> wrote:

> Agreed that these should stay out of the repo.  Trident has this issue as
> well, and it automatically downloads them when the code is used the first
> time.  It also has a permanent location where HPC users and such can
> download them directly and put them in the appropriate directory.  I think
> this is about as good as you can get to making it work for everyone.  Just
> document it and hope people read the docs before using.
>
> Cameron
>
> On Thu, Dec 8, 2016 at 7:34 AM, Matthew Turk <matthewturk at gmail.com>
> wrote:
>
>> Hi,
>>
>> I'm okay with this, but let's try to make it obvious where things go.
>> (which you have all suggested.)
>>
>> I think astropy does this and has helper functions for it.
>>
>> Matt
>>
>> On Dec 8, 2016 9:28 AM, "Britton Smith" <brittonsmith at gmail.com> wrote:
>>
>> HI all,
>>
>> I agree that it would be best to keep the data files out of the repo.
>> What about adding some config variables to control where the data gets
>> stored?  We could even have it default to storing things in .config/yt.
>> Perhaps, on the first import when the data is not there, you just get a
>> notice with links to wget/curl them.
>>
>> Britton
>>
>> On Wed, Dec 7, 2016 at 12:12 PM, Nathan Goldbaum <nathan12343 at gmail.com>
>> wrote:
>>
>>>
>>>
>>> On Wed, Dec 7, 2016 at 1:30 PM, John Zuhone <jzuhone at gmail.com> wrote:
>>>
>>>> Hi everyone,
>>>>
>>>> I just issued PR 2465 (https://bitbucket.org/yt_anal
>>>> ysis/yt/pull-requests/2465/wip-port-the-spectral_integrator-analysis/)
>>>> to port the spectral integrator analysis module (originally written by
>>>> Britton) under yt.fields. spectral_integrator is an analysis module that
>>>> creates X-ray emission fields in user-specified energy bands.
>>>>
>>>> The hitch is that spectral_integrator uses HDF5 tables to compute the
>>>> emissivity, since they are not analytical functions. We currently host
>>>> those tables on yt-project.org/data. spectral_integrator downloads
>>>> them automatically if the analysis module is used and they are not present.
>>>>
>>>> There was some discussion on Slack as to whether or not this is the
>>>> correct approach, since it’s not ideal for certain computing environments
>>>> (e.g., HPC).
>>>>
>>>
>>> Also a problem for e.g. packaging yt on linux distros.
>>>
>>>
>>>> The files are a total of about 2.4 MB in size, so there is some
>>>> reticence to bundling them with yt.
>>>>
>>>
>>> So my main concern here is adding them to the repo. If you could figure
>>> out a way to add them to the source distribution without including them in
>>> the repo (and then issue an error with a URL for non-sdist builds of yt
>>> [e.g. from the repo]), it wouldn't be a problem.
>>>
>>> This isn't really a generic solution for all binary files we might want
>>> to send to users, but 2.4 MB isn't all that big to be part of an sdist or
>>> wheel.
>>>
>>>
>>>> There is also the issue that uploading new versions of the tables
>>>> breaks backward-compatibility. I have sacrificed backwards-compatiblity for
>>>> considerable simplification of code.
>>>>
>>>
>>> I think the solution is just to use different names for the files or the
>>> hdf5 datasets or groups here so the code can tell the difference.
>>>
>>>
>>>>
>>>> Does anyone have any ideas about the best way to handle this issue?
>>>>
>>>> Best,
>>>>
>>>> John Z
>>>>
>>>> _______________________________________________
>>>> yt-dev mailing list
>>>> yt-dev at lists.spacepope.org
>>>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>>>>
>>>>
>>>
>>> _______________________________________________
>>> yt-dev mailing list
>>> yt-dev at lists.spacepope.org
>>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>>>
>>>
>>
>> _______________________________________________
>> yt-dev mailing list
>> yt-dev at lists.spacepope.org
>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>>
>>
>>
>> _______________________________________________
>> yt-dev mailing list
>> yt-dev at lists.spacepope.org
>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>>
>>
>
>
> --
> Cameron Hummels
> NSF Postdoctoral Fellow
> Department of Astronomy
> California Institute of Technology
> http://chummels.org
>
> _______________________________________________
> yt-dev mailing list
> yt-dev at lists.spacepope.org
> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.spacepope.org/pipermail/yt-dev-spacepope.org/attachments/20161208/0b940bd3/attachment.html>


More information about the yt-dev mailing list