[Yt-dev] A fun interface for data

Thu May 19 10:52:10 PDT 2011

Hi Sam,

I am totally on board.  I wrote up a quick script that does this -- it
requires both the parameter file and a .yt file, but nothing else.
Here's what the source looks like:

http://paste.enzotools.org/show/1645/

I tested this with the 1024 L7, for which I only have the .yt and
parameter file, and it works.  :)

-Matt

On Thu, May 19, 2011 at 1:45 PM, Sam Skillman <samskillman at gmail.com> wrote:
> I was just thinking about this a bit, and I do think that while we will want
> to roll this into reason, I could also see a use case for a very minimal
> stand-alone viewer that doesn't know about anything outside of a .yt file.
>  I think it could be neat to set up something where you could do yt explore
> DD1234.yt, which brings up all of the data objects that have been stored,
> allowing you to navigate through your saved projections.  Anyways, just
> throwing this out there...I haven't thought it through very much.
> A totally agree with Cameron, this is going to be super awesome.
> Sam
>
> On Wed, May 18, 2011 at 2:13 PM, Cameron Hummels
> <chummels at astro.columbia.edu> wrote:
>>
>> This is a super cool idea, especially given what I saw when you demo'd
>> it to me, Matt.  So what you're proposing is a sort of "standalone"
>> version of the applet that allows pan-n-scan in our plot windows, yes?
>> One so that a person could just go to a remote website that had the data
>> on it and be able to move around in that data interactively for
>> presentation to an audience, or basically like a googlemaps interace for
>> a few datasets?
>>
>> I think this would be very useful, however, we need to take things one
>> step at a time, I think.  Why don't we implement this in the existing
>> reason for now, and see how that works.  After all, reason will be able
>> to deal with more than just projections of the dataset and more than
>> just looking at it from one perspective, right?  Correct me if I'm
>> wrong, but what you're describing is a scaled down version of this which
>> would only be able to look at the tiles for one perspective's
>> projections?  Because if we want to do more than one perspective, or
>> more than one type of data (slices, projections, off-axis projections,
>> etc.), then we need to store more tiles, right?  Then we need more room
>> for the data, and we need a means in the window of switching between the
>> various views and modes, and then it becomes closer and closer to the
>> original reason.  So what I think we should do is first focus on
>> implementing it with the full suite of tools that we're doing in reason,
>> and THEN if things are slow for the full reason implementation, we could
>> make a reason-light version that scales down the feature set to only
>> pan-n-scan for one mode and one perspective.
>>
>> What do you think?
>>
>> Oh, btw, I totally want to be involved in this, because it is super
>> awesome.
>>
>> Cameron
>>
>> On 05/18/2011 01:52 PM, Matthew Turk wrote:
>> > Thanks, Sam!
>> >
>> > As a quick note, I changed all of the PNG writing to not use tempfile
>> > but instead write directly to in-memory, which was a pretty glaring
>> > problem waiting to rear its head.
>> >
>> > I was idly thinking about this yesterday, and after some conversations
>> > I've had with Tom and others I think that one could imagine a very
>> > cool project coming out of this.  The idea would be that the minimum
>> > amount of information to do an entire pan-n-scan is really just the
>> > 5xN array for a given projection or slice: (px, py, pdx, pdy, z).
>> > This is how it currently works -- it simply repixelizes that array for
>> > each tile segment, and returns a PNG of that newly created tile.  The
>> > storage on the backend is really quite small; these 5xN arrays are
>> > much more efficient than creating all the possible tiles in advance.
>> > They're not free in terms of web-server memory, like the tiles would
>> > be, but they're quite inexpensive.  I believe that even for the 1024^3
>> > L7 dataset it was on the order of a hundred megs or so.
>> >
>> > So imagine having a server that lived somewhere that stored a bunch of
>> > these 5xN arrays in memory, along with appropriate metadata.  Go to
>> > the frontend of the server, select one to view, and then having that
>> > dataset pop up.  Although they aren't in the mapserver command, one
>> > can supply annotations and even drawings with the leaflet library.
>> >
>> > (I've basically just described a number of existing services.  Except,
>> > ours would be based on simulation data exclusively and would probably
>> > be able to scale to having many, many datasets!)
>> >
>> > You'd have to implement a frontend, a backend that stored these
>> > datasets, and likely have some kind of garbage collection -- if the
>> > last request for a given dataset was more than N minutes ago, remove
>> > it from memory.  I think this could be a viable, fun project, and
>> > would be really excellent to use as supplemental data for papers and
>> > presentations -- particularly for star formation and cosmology, where
>> > there are a number of points of interest within relatively high
>> > dynamical range.
>> >
>> > Anyway, if someone is interested in working on this, it could be VERY
>> > fun and possibly quite straightforward to implement.  I will confess I
>> > don't know enough about web apps, server deployment and database
>> > backends to move forward on it, but I'd be more than happy to help out
>> > with the data transport and visualization aspects, which I think the
>> > mapserver already has a good start on.  Coming up with money to run
>> > this on, say, EC2 is probably also possible; it may even be reasonable
>> > to think we could make this a project that people in the community
>> > could upload to and use.
>> >
>> > -Matt
>> >
>> > On Wed, May 18, 2011 at 1:37 PM, Sam Skillman <samskillman at gmail.com>
>> > wrote:
>> >> Matt,
>> >> This is awesome, and I'm definitely a +1 on rolling this into reason in
>> >> the
>> >> future.  I'm very tempted to throw this up on a big set of displays,
>> >> and up
>> >> the window size :).
>> >> Well done!
>> >> Sam
>> >> On Wed, May 18, 2011 at 1:42 AM, Matthew Turk <matthewturk at gmail.com>
>> >> wrote:
>> >>> Thanks for the positive feedback, Stephen -- I'm pretty excited about
>> >>> this, too.
>> >>>
>> >>> As a stopgap before it gets rolled into reason, which may end up being
>> >>> sort of tricky, I have added a "mapserver" command to the yt command
>> >>> line utility.  If you update your installation, you can do:
>> >>>
>> >>> yt mapserver DD0030/DD0030
>> >>>
>> >>> and it'll spawn on http://127.0.0.1:8080/  , which you can then open
>> >>> in your browser.  There are a couple options for this command, too --
>> >>> axis, field, projection, and weight, which you can see with --help.
>> >>>
>> >>> -Matt
>> >>>
>> >>> On Tue, May 17, 2011 at 4:15 PM, Stephen Skory <s at skory.us> wrote:
>> >>>> Hi Matt,
>> >>>>
>> >>>>> I'd love it if people could download and try it out.  If you have yt
>> >>>>> installed, then you don't need anything other than this repo:
>> >>>> I just gave it a go on a couple datasets and it worked pretty well!
>> >>>> The periodicity is handled interestingly - just like Google Maps
>> >>>> where
>> >>>> you can pan left and right and get the same thing, but up and down
>> >>>> cuts off the data. Really neat!
>> >>>>
>> >>>> --
>> >>>> Stephen Skory
>> >>>> s at skory.us
>> >>>> http://stephenskory.com/
>> >>>> 510.621.3687 (google voice)
>> >>>> _______________________________________________
>> >>>> Yt-dev mailing list
>> >>>> Yt-dev at lists.spacepope.org
>> >>>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>> >>>>
>> >>> _______________________________________________
>> >>> Yt-dev mailing list
>> >>> Yt-dev at lists.spacepope.org
>> >>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>> >>
>> >> _______________________________________________
>> >> Yt-dev mailing list
>> >> Yt-dev at lists.spacepope.org
>> >> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>> >>
>> >>
>> > _______________________________________________
>> > Yt-dev mailing list
>> > Yt-dev at lists.spacepope.org
>> > http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>> >
>>
>> --
>> Cameron Hummels
>> PhD Candidate, Astronomy Department of Columbia University
>> Public Outreach Director, Astronomy Department of Columbia University
>> NASA IYA New York State Student Ambassador
>> http://outreach.astro.columbia.edu
>> PGP: 0x06F886E3
>>
>> _______________________________________________
>> Yt-dev mailing list
>> Yt-dev at lists.spacepope.org
>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>
>
> _______________________________________________
> Yt-dev mailing list
> Yt-dev at lists.spacepope.org
> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>
>