[Yt-dev] Simulation Database

[Matthew Turk]

Hi Tom and Stephen,

Tom -- your field list looks great.  I think what it (and Stephen's
suggestion of topgrid[012]) convinced me is that we need a Simulation
entry, not just the SimulationOutput, as the only variable we expect
to change is the number of cells and potentially the number of
particles.  I like including them, particularly as they are also good
proxies to the simulation size.  I'll work on manipulating to support
these items, possibly indirectly with variable length fields, and
adding the simulation entries (which might be more work and which I
don't know how to directly support non-Enzo datasets with yet.  Any
suggestions from other code devs would be greatly appreciated!)

On Tue, Sep 6, 2011 at 2:49 PM, Stephen Skory <s at skory.us> wrote:
> Matt,
>
>> Thanks for the pointer -- this is super cool.  How would you see this
>> working, in practice?
>
> I don't think that we should abandon a local database. There are
> enough firewalled machines out there, which would block SimpleDB
> acces, meaning a local copy is still useful. I think SimpleDB should
> be complimentary to local database(s). I'm not sure that the SimpleDB
> copy should be pulled from in the same way that the local copy is when
> yt runs.

Right, of course -- I wasn't suggesting we only use SimpleDB, just
that it works for the "remote" component, in a complementary fashion
as you suggest.  The rest that you outlined looks awesome, and I am
100% on board.  One module I stumbled across recently that might end
up being of help is urlparse, which might ease the issue of addressing
remote hosts.

I really encourage you to think about this and explore it, and bring
it to the list to share and discuss.  Even a small support of this
kind of interaction would be ... amazing.

>
> In practice, I think we should simply push whatever would go into the
> local database to SimpleDB, along with a machine identifier (*). This
> should be transparent to the user once they've entered their AWS
> credentials (**).
>
> I envision a webpage (say, mydb.yt-project.org) where users can enter
> their AWS creds to do simple queries or just see all their datasets.
> There are lots of web-based tools out there already, but it may not be
> difficult to write our own if the available tools are not suitable.
>
> It would be cool to add a command like "yt crawl DD????/DD????" that
> would automatically load all the dataset info into the databases
> without having to load() all of them manually. Another command like
> "yt dbclean" would delete datasets from the database (or mark them as
> stale or broken) that are no longer on disk and do the same online.
>
> I don't think we need to over think this. Just mirror the local
> database to SimpleDB, and possibly write a simple python webapp that
> uses Boto and whatever else to look at the datasets in a manageable
> way hosted on yt-project.org. Any fancier features will come if people
> need them, but I think this alone is pretty killer.
>
> I'm willing to contribute time to this stuff.
>
> (*) I think that this may prove to be more difficult than one might
> think to automate, so perhaps we shouldn't even try. The first place
> one would look for the name is 'hostname', but this could be as useful
> as 'login03.machine.institution.edu' or even as useless 'node0284' for
> an internal 10.x.x.x address. I think that the best way would be to
> query the user for their preferred name for this machine when yt is
> installed, and stored similarly to the hdf5.cfg file in a plain text
> file.
>
> (**) This raises the question of whether we want yt to store AWS
> credentials or not. The simplest solution is to have  them stored in
> the same file as the machine name with 'chmod 0600'. More secure
> solutions involving hashes get more complicated past that...
>
> --
> Stephen Skory
> s at skory.us
> http://stephenskory.com/
> 510.621.3687 (google voice)
> _______________________________________________
> Yt-dev mailing list
> Yt-dev at lists.spacepope.org
> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>