[Yt-dev] Simulation Database

Matthew Turk matthewturk at gmail.com
Tue Sep 6 08:55:34 PDT 2011


Hi Stephen,

On Tue, Sep 6, 2011 at 11:50 AM, Stephen Skory <s at skory.us> wrote:
> Britton,
>
>> If we're going minimal here, then I don't think we need the topgrid
>> entries.  In theory, everything one needs to know about The Simulation can
>> come from the simulation_uuid.
>
> The idea I had about this database is it may be worth something to
> someone someday to be able to easily contrast and compare all the
> datasets tracked by the database. In which case, having more
> information than just the simulation_uuid would be useful. But I do
> agree overall with the idea that less is more, in most cases.

Okay -- so maybe for now, we can remove topgrid[012].

>
> I just had a thought. Wouldn't it be cool to have a universal
> database, shared by all machines you use? Then, for example, you could
> use that to look up where you stored a dataset (don't tell me you've
> never logged into several machines for just that reason). You could
> ask "do I have a dataset that's in a box about 500 Mpc/h at z=3, and
> where do I have it? And what else can you tell me about those datasets
> even though I'm not on that computer right now?" I can't think of any
> really good way to do it, other than some ad-hoc Dropbox sync, which
> probably raises other issues to do with concurrency and setup costs.
> But still, I think it would be cool!

Yes ... I completely agree.  That's where I'd like this to lead.  In
fact, what I would like to do is to provide something that could be
rolled out on a server by any interested party for uploading to.
Having a codified, flexible format is the first step toward that.  We
should talk more about this, but perhaps a bit down the line?  There's
quite a bit of interest in this from several parties.  The IVOA has
this on their agenda, but I think there is a place for something
that's immediately useful with low overhead and complexity, whereas
they are designing something much more ambitious.

Ultimately, I think the best way to do this would be to provide a web
application that could be deployed to EC2 or (better yet, as I suspect
it will be way below free quotas) google app engine.  Provide an API
for uploads, and and API for queries.  AGPL license it, and provide it
to anyone who wants to deploy.  In principle this is straightforward,
but in practice it may present challenges.  It's also a project unto
itself, even if it endsu p only being a few thousand lines of code.
Would be ideal for a GSoC student or other interested party!

-Matt

>
> --
> Stephen Skory
> s at skory.us
> http://stephenskory.com/
> 510.621.3687 (google voice)
> _______________________________________________
> Yt-dev mailing list
> Yt-dev at lists.spacepope.org
> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>



More information about the yt-dev mailing list