[yt-dev] Data downloader

Nathan Goldbaum nathan12343 at gmail.com
Tue Jul 24 11:13:19 PDT 2012


Hi Casey,

The direct links are of course still available (just not publicly visible): http://yt-project.org/data/

-Nathan

On Jul 24, 2012, at 11:09 AM, Casey W. Stark wrote:

> Sounds good. I would support a tarball per simulation also. Maybe this is just me, but I would prefer a direct link and using my own tools.
> 
> - Casey
> 
> 
> On Tue, Jul 24, 2012 at 11:01 AM, Britton Smith <brittonsmith at gmail.com> wrote:
> Even one tar file per simulation would be ok, at least for the smaller ones.  The enzo_tiny_cosmology simulation is designed to showcase time series and things using multiple datasets, so having a tarfile for each dataset is probably unnecessary.  Perhaps we could just evaluate what files are meant to be used in groups and put those together in single tarfiles.
> 
> Britton
> 
> On Tue, Jul 24, 2012 at 1:55 PM, Casey W. Stark <caseywstark at gmail.com> wrote:
> I would be in favor of one tarball for simplicity. Are the example files that large?
> 
> - Casey
> 
> 
> On Tue, Jul 24, 2012 at 10:48 AM, Matthew Turk <matthewturk at gmail.com> wrote:
> Hi John,
> 
> Hm, that's puzzling.  Stephen, any ideas?
> 
> After Stephen's email, I went from +0 on keeping the downloader to +1,
> because I think having it from the command line is a much simpler
> solution than what we had tried before, which was the
> download-by-hand.  So let's see if we can address this, and then check
> it in to scripts/ .
> 
> -Matt
> 
> On Tue, Jul 24, 2012 at 1:45 PM, John ZuHone <jzuhone at gmail.com> wrote:
> > Hi all,
> >
> > What I don't like about the downloader is the directory structure it creates. At least on my machine, if I download only the sloshing dataset, I get:
> >
> > GasSloshing/GasSloshing/sloshing_nomag2*
> >
> > as the location of the files. Is there any reason why it ended up this way?
> >
> > John
> >
> > On Jul 24, 2012, at 11:26 AM, Stephen Skory wrote:
> >
> >> Hi Matt,
> >>
> >>> One question that's come up is: do we want to continue using
> >>> download.py?  The burden on uploaders is moderately higher, in that
> >>> the files all have to be tarred up in a particular way, and they have
> >>> to be added to download.py, but it does provide a measure of
> >>> robustness.  If we do this, can we move download.py into the main
> >>> distribution, under scripts/ ?  Stephen, John, others who have used
> >>> the script, what do you think about this?
> >>
> >> I will not be insulted if we do away with the download script. If a
> >> web page of download links is easier for everyone around, that's fine
> >> by me. The best argument I can think of for keeping it, or something
> >> similar, it is forces some kind of uniformity so that the datasets are
> >> in an expected layout. Also, downloading tens of data dumps one at at
> >> time from a webpage is kind of tedious. That could be solved by having
> >> both the separate data dumps and a big 'ol tarball of the whole thing
> >> so people could get exactly what they want, but that doubles disk
> >> space on someone's computer.
> >>
> >> But this isn't HIPPA medical data, so we can be loosie goosie if that
> >> makes things easier for everyone!
> >>
> >> --
> >> Stephen Skory
> >> s at skory.us
> >> http://stephenskory.com/
> >> 510.621.3687 (google voice)
> >> _______________________________________________
> >> yt-dev mailing list
> >> yt-dev at lists.spacepope.org
> >> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
> >
> > _______________________________________________
> > yt-dev mailing list
> > yt-dev at lists.spacepope.org
> > http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
> _______________________________________________
> yt-dev mailing list
> yt-dev at lists.spacepope.org
> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
> 
> 
> _______________________________________________
> yt-dev mailing list
> yt-dev at lists.spacepope.org
> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
> 
> 
> 
> _______________________________________________
> yt-dev mailing list
> yt-dev at lists.spacepope.org
> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
> 
> 
> _______________________________________________
> yt-dev mailing list
> yt-dev at lists.spacepope.org
> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.spacepope.org/pipermail/yt-dev-spacepope.org/attachments/20120724/f8bd0729/attachment.html>


More information about the yt-dev mailing list