[Yt-dev] "Database" of Parameter Files

[Matthew Turk]

Hi all,

This week, after Cameron announced and Jeff demo'd the new GUI for yt,
we've had some thinking and thoughts about some other obvious places
to go with the interface.  Tonight Tom and I did some brainstorming
about what a *personal* database of simulation outputs might look
like; this isn't so much about large, comprehensive databases of
simulations, but rather about a pragmatic convenience for people who
run their data on a single file system.

As it stands, yt dumps the last 200 parameter files into a .csv file
in your ~/.yt directory.  These include the unique id, the parameter
file name on the file system, and some other misc data that's not
terribly important.

We were thinking it might be neat to have a really simple one that the
files never really left.  It could be ephemeral -- so that files that
get removed or moved around would simply be removed from the DB -- and
maybe it would contain information (if available from the simulation
code) about the previous output in the simulation.  Enzo has this, and
it may be coming to other codes, in the form of UUIDs that get tracked
and carried along by the sim.

So there are a couple ways this might work.  We came up with a few:

 * A "locate" command that just returns all the basic parameter files
/ outputs that could be loaded into yt (or other tools), maybe based
on a quick search
 * A registering of these outputs into the database at write time by
the simulation code
 * An intentional import of, say, a directory -- simply type "yt
import" in the base dir and it'll find all the outputs below that.
(This used to exist in the form of "fimport" but it has bitrot a bit.)
 * An unlimited parameter file storage, where all loaded parameter
files get added (unlike the 200 limit we have)

What do people think?  With the new GUI, this becomes I think a lot
more interesting -- particularly if you can re-assemble a graph of the
outputs, maybe even querying them, if you were to store the parameter
file in full.

Anyway, just a handful of thoughts.  Stephen, you have quite a bit of
experience with SQLite; do you think this could be a fun application
of them?  Or would that be overkill?  Maybe the whole system could be
handled with just filenames?

-Matt