[Yt-dev] A fun interface for data

Matthew Turk matthewturk at gmail.com
Wed May 18 10:52:51 PDT 2011


Thanks, Sam!

As a quick note, I changed all of the PNG writing to not use tempfile
but instead write directly to in-memory, which was a pretty glaring
problem waiting to rear its head.

I was idly thinking about this yesterday, and after some conversations
I've had with Tom and others I think that one could imagine a very
cool project coming out of this.  The idea would be that the minimum
amount of information to do an entire pan-n-scan is really just the
5xN array for a given projection or slice: (px, py, pdx, pdy, z).
This is how it currently works -- it simply repixelizes that array for
each tile segment, and returns a PNG of that newly created tile.  The
storage on the backend is really quite small; these 5xN arrays are
much more efficient than creating all the possible tiles in advance.
They're not free in terms of web-server memory, like the tiles would
be, but they're quite inexpensive.  I believe that even for the 1024^3
L7 dataset it was on the order of a hundred megs or so.

So imagine having a server that lived somewhere that stored a bunch of
these 5xN arrays in memory, along with appropriate metadata.  Go to
the frontend of the server, select one to view, and then having that
dataset pop up.  Although they aren't in the mapserver command, one
can supply annotations and even drawings with the leaflet library.

(I've basically just described a number of existing services.  Except,
ours would be based on simulation data exclusively and would probably
be able to scale to having many, many datasets!)

You'd have to implement a frontend, a backend that stored these
datasets, and likely have some kind of garbage collection -- if the
last request for a given dataset was more than N minutes ago, remove
it from memory.  I think this could be a viable, fun project, and
would be really excellent to use as supplemental data for papers and
presentations -- particularly for star formation and cosmology, where
there are a number of points of interest within relatively high
dynamical range.

Anyway, if someone is interested in working on this, it could be VERY
fun and possibly quite straightforward to implement.  I will confess I
don't know enough about web apps, server deployment and database
backends to move forward on it, but I'd be more than happy to help out
with the data transport and visualization aspects, which I think the
mapserver already has a good start on.  Coming up with money to run
this on, say, EC2 is probably also possible; it may even be reasonable
to think we could make this a project that people in the community
could upload to and use.

-Matt

On Wed, May 18, 2011 at 1:37 PM, Sam Skillman <samskillman at gmail.com> wrote:
> Matt,
> This is awesome, and I'm definitely a +1 on rolling this into reason in the
> future.  I'm very tempted to throw this up on a big set of displays, and up
> the window size :).
> Well done!
> Sam
> On Wed, May 18, 2011 at 1:42 AM, Matthew Turk <matthewturk at gmail.com> wrote:
>>
>> Thanks for the positive feedback, Stephen -- I'm pretty excited about
>> this, too.
>>
>> As a stopgap before it gets rolled into reason, which may end up being
>> sort of tricky, I have added a "mapserver" command to the yt command
>> line utility.  If you update your installation, you can do:
>>
>> yt mapserver DD0030/DD0030
>>
>> and it'll spawn on http://127.0.0.1:8080/  , which you can then open
>> in your browser.  There are a couple options for this command, too --
>> axis, field, projection, and weight, which you can see with --help.
>>
>> -Matt
>>
>> On Tue, May 17, 2011 at 4:15 PM, Stephen Skory <s at skory.us> wrote:
>> > Hi Matt,
>> >
>> >> I'd love it if people could download and try it out.  If you have yt
>> >> installed, then you don't need anything other than this repo:
>> >
>> > I just gave it a go on a couple datasets and it worked pretty well!
>> > The periodicity is handled interestingly - just like Google Maps where
>> > you can pan left and right and get the same thing, but up and down
>> > cuts off the data. Really neat!
>> >
>> > --
>> > Stephen Skory
>> > s at skory.us
>> > http://stephenskory.com/
>> > 510.621.3687 (google voice)
>> > _______________________________________________
>> > Yt-dev mailing list
>> > Yt-dev at lists.spacepope.org
>> > http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>> >
>> _______________________________________________
>> Yt-dev mailing list
>> Yt-dev at lists.spacepope.org
>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>
>
> _______________________________________________
> Yt-dev mailing list
> Yt-dev at lists.spacepope.org
> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>
>



More information about the yt-dev mailing list