[Yt-dev] Simulation Database

Tom Abel tabel at slac.stanford.edu
Tue Sep 6 11:25:08 PDT 2011


Matt, that's great stuff! I only have two cents to add not more at this point … 

SQLite should work very well for this. 

Some parameters one could already gauge already some
basic properties of the simulation would be in this order:
ProblemType
BoxSize
NumberOfBaryonFields
TotalNumberOfGridCells 
TotalNumberOfParticles

Almost all of this is in the ParameterFiles.
So I'd include those basic things only if there is a 
good chance it will help in making very fast queries
down the road of what already exists. 

What do you think?
Tom


On Sep 6, 2011, at 10:34 AM, Matthew Turk wrote:

> Forgot to include this link, about pricing:
> 
> http://aws.amazon.com/simpledb/#pricing
> 
> take note of two things.  1) all data transfer IN is free, data
> transfer OUT is free for first 1GB and $0.12/gb after that.  2) There
> are substantial Amazon education grants (Galacticus benefits from
> these, for instance) which are basically given away.
> 
> On Tue, Sep 6, 2011 at 1:33 PM, Matthew Turk <matthewturk at gmail.com> wrote:
>> Hi Stephen,
>> 
>> On Tue, Sep 6, 2011 at 12:03 PM, Stephen Skory <s at skory.us> wrote:
>>> Matt,
>>> 
>>>>  There's
>>>> quite a bit of interest in this from several parties.  The IVOA has
>>>> this on their agenda, but I think there is a place for something
>>>> that's immediately useful with low overhead and complexity, whereas
>>>> they are designing something much more ambitious.
>>> 
>>> Do you know if they're (IVOA) designing something that has both a
>>> personal database and a public database?
>> 
>> I honestly have no idea, but I assume first the latter, then the
>> former.  I have done a small amount of work setting up Enzo as a
>> "Simulator" for their databases, but after the workshop I was at where
>> I was meeting with some VO people, work on that sort of tapered off
>> abruptly.
>> 
>>> Because what I was
>>> envisioning is a personal database; an extension of the
>>> parameter_files.(csv/db) file to cover more than one machine. It could
>>> be as simple as writing something to use something to use this in
>>> tandem or as an alterative to the SQLite file:
>>> 
>>> http://aws.amazon.com/simpledb/
>> 
>> Okay, this is better than I realized it was.  This is actually all we
>> would need, I think.  Federating between instances might be more
>> complex, but this should cover almost all of our needs.  Boto supports
>> it:
>> 
>> http://code.google.com/p/boto/
>> 
>> with code like:
>> 
>> http://code.google.com/p/boto/wiki/SimpleDbIntro
>> 
>> This is awesome.  We actually would not really need a webapp if we
>> used this, and furthermore, it is easily extensible and not very much
>> money.  The free tier is ample to cover storing information about all
>> the simulations you run, and it could be extended to support storing
>> extracted data products from yt.  This is perfect.
>> 
>> Thanks for the pointer -- this is super cool.  How would you see this
>> working, in practice?
>> 
>> -Matt
>> 
>>> 
>>> Something to think about...
>>> 
>>> --
>>> Stephen Skory
>>> s at skory.us
>>> http://stephenskory.com/
>>> 510.621.3687 (google voice)
>>> _______________________________________________
>>> Yt-dev mailing list
>>> Yt-dev at lists.spacepope.org
>>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>>> 
>> 
> _______________________________________________
> Yt-dev mailing list
> Yt-dev at lists.spacepope.org
> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org




More information about the yt-dev mailing list