[Yt-dev] Simulation Database
Tom Abel
tabel at slac.stanford.edu
Tue Sep 6 11:25:08 PDT 2011
Matt, that's great stuff! I only have two cents to add not more at this point …
SQLite should work very well for this.
Some parameters one could already gauge already some
basic properties of the simulation would be in this order:
ProblemType
BoxSize
NumberOfBaryonFields
TotalNumberOfGridCells
TotalNumberOfParticles
Almost all of this is in the ParameterFiles.
So I'd include those basic things only if there is a
good chance it will help in making very fast queries
down the road of what already exists.
What do you think?
Tom
On Sep 6, 2011, at 10:34 AM, Matthew Turk wrote:
> Forgot to include this link, about pricing:
>
> http://aws.amazon.com/simpledb/#pricing
>
> take note of two things. 1) all data transfer IN is free, data
> transfer OUT is free for first 1GB and $0.12/gb after that. 2) There
> are substantial Amazon education grants (Galacticus benefits from
> these, for instance) which are basically given away.
>
> On Tue, Sep 6, 2011 at 1:33 PM, Matthew Turk <matthewturk at gmail.com> wrote:
>> Hi Stephen,
>>
>> On Tue, Sep 6, 2011 at 12:03 PM, Stephen Skory <s at skory.us> wrote:
>>> Matt,
>>>
>>>> There's
>>>> quite a bit of interest in this from several parties. The IVOA has
>>>> this on their agenda, but I think there is a place for something
>>>> that's immediately useful with low overhead and complexity, whereas
>>>> they are designing something much more ambitious.
>>>
>>> Do you know if they're (IVOA) designing something that has both a
>>> personal database and a public database?
>>
>> I honestly have no idea, but I assume first the latter, then the
>> former. I have done a small amount of work setting up Enzo as a
>> "Simulator" for their databases, but after the workshop I was at where
>> I was meeting with some VO people, work on that sort of tapered off
>> abruptly.
>>
>>> Because what I was
>>> envisioning is a personal database; an extension of the
>>> parameter_files.(csv/db) file to cover more than one machine. It could
>>> be as simple as writing something to use something to use this in
>>> tandem or as an alterative to the SQLite file:
>>>
>>> http://aws.amazon.com/simpledb/
>>
>> Okay, this is better than I realized it was. This is actually all we
>> would need, I think. Federating between instances might be more
>> complex, but this should cover almost all of our needs. Boto supports
>> it:
>>
>> http://code.google.com/p/boto/
>>
>> with code like:
>>
>> http://code.google.com/p/boto/wiki/SimpleDbIntro
>>
>> This is awesome. We actually would not really need a webapp if we
>> used this, and furthermore, it is easily extensible and not very much
>> money. The free tier is ample to cover storing information about all
>> the simulations you run, and it could be extended to support storing
>> extracted data products from yt. This is perfect.
>>
>> Thanks for the pointer -- this is super cool. How would you see this
>> working, in practice?
>>
>> -Matt
>>
>>>
>>> Something to think about...
>>>
>>> --
>>> Stephen Skory
>>> s at skory.us
>>> http://stephenskory.com/
>>> 510.621.3687 (google voice)
>>> _______________________________________________
>>> Yt-dev mailing list
>>> Yt-dev at lists.spacepope.org
>>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>>>
>>
> _______________________________________________
> Yt-dev mailing list
> Yt-dev at lists.spacepope.org
> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
More information about the yt-dev
mailing list