[yt-dev] Fwd: [yt_analysis/yt] Volume Rendering Refactor (pull request #168)

Matthew Turk matthewturk at gmail.com
Fri Jun 15 08:43:36 PDT 2012 

Hi all,

So we've updated the PR a couple times now, and I think we're about
set.  Does anyone have any other comments?

-Matt

On Thu, Jun 14, 2012 at 12:00 AM, Sam Skillman <samskillman at gmail.com> wrote:
> Hi Matt,
>
> Thanks for the kind words.  I'm glad that I'm in a community of developers
> who 1) care about results, 2) Have buy-in because it's what we use to get
> work done, 3) are cool people!
>
> Chris -- I think the delay between the building and the casting is when the
> data is loaded into the kdtree bricks.  Both the building of the kdtree and
> the reading of the data are serial processes.  So the relavent speedup is
> the one during the ray-casting.  One other thing to note is that if you are
> doing camera movements, the data should all stay in memory so these serial
> costs should go away.  Anyways, thanks for testing!  That's some great
> performance!
>
> Cheers,
> Sam
>
>
> On Wed, Jun 13, 2012 at 9:40 PM, Matthew Turk <matthewturk at gmail.com> wrote:
>>
>> Hi Sam,
>>
>> Thanks for doing this.
>>
>> I'll update my PR shortly; this needs to be a part of it before it's
>> accepted.
>>
>> This has also highlighted a part of the process that I think we're all
>> trying to do -- peer review not just of the quantitative changes to the code
>> (i.e., breaking tests!) but also qualitative.  In this case, Cameron saw
>> something that would change functionality, and he stood up for keeping the
>> ability to retain backwards compatibility.  I think this is a valid point
>> and a valuable position; in the case of the off-axis projection, when I
>> initially made this change it was because I felt this was truer to the
>> underlying data.  (There are extremely few places in the yt codebase where
>> it interpolates.)
>>
>> However, retaining a solid foundation for users is also extremely
>> important.  The growth of yt over the last couple years has put us in a
>> sticky situation; we've developed a community, but it's a community that
>> relies on functionality not changing without good reason, and more
>> importantly that relies on keeping a solid basis for repeatable analysis and
>> consistent results.  This change broke that -- even though it's arguably an
>> improvement of the solution -- and the old method was no longer available.
>>
>> I'm really happy that this system has worked; while I was initially -1 on
>> restoring interpolation, Cameron swayed me in IRC by bringing up this
>> remarkably important point.  I think turning the default to non-interpolated
>> but retaining the interpolated mode are both the right decisions, and I'm
>> +1.  And more than that, I'm glad this all happened before the PR was
>> accepted.
>>
>> So thank you, Sam, and Cameron, for restoring the old functionality.  I'm
>> excited about moving forward.  The results in this thread have really gotten
>> me excited about the success of this new set of changes.
>>
>> -Matt
>>
>> On Jun 13, 2012 11:26 PM, "Sam Skillman" <samskillman at gmail.com> wrote:
>>>
>>> Hi all,
>>>
>>> Earlier today Cameron brought up some concerns about removing the
>>> behavior of the off-axis-projection using interpolated data from
>>> vertex-centered data.  If you recall from earlier discussions on the impact
>>> of the new threading + opaque renderings + refactoring, one of the things
>>> we've done is switch from the interpolation to using the raw values within
>>> each cell a given ray traverses for use in the off-axis-projection.  This is
>>> mathematically a more accurate representation of the data, but can sometimes
>>> lead to blocky looking images.
>>>
>>> What I've done is go through and add an optional
>>> interpolated=(False/True) argument to the off_axis_projection call, where
>>> the default is False.  Additionally, I've set this up so that it now wraps
>>> the ProjectionCamera class, which is derived from the Camera class, so it
>>> has things like .rotate, etc.
>>>
>>> Just for reference, I'm including below both timings, results, and
>>> example images, made using this script:
>>> http://paste.yt-project.org/show/2462/
>>>
>>> Here is the output:
>>> Comparison of Raw vs Interpolated
>>> Density
>>> Raw -- Time: 1.173431e+00 seconds, min: 8.046089e-06, max: 1.177795e-02
>>> Int -- Time: 6.393837e+00 seconds, min: 8.322453e-06, max: 1.113421e-02
>>>
>>> Density-weighted Temperature
>>> Raw -- Time: 8.513930e-01 seconds, min: 4.681834e+00, max: 1.237399e+07
>>> Int -- Time: 8.450805e+00 seconds, min: 1.105447e+01, max: 1.213922e+07
>>>
>>> The main difference that you see here is that the interpolated doesn't
>>> capture the minima and maxima as well, which makes sense since it is
>>> interpolating the data and smoothing out the peaks.  Note that the
>>> Interpolated use the no_ghost=False option, which brings it much closer to
>>> the Raw values.  This highlights the improvement off the off_axis_projection
>>> for quantitative results, as it is both more accurate and faster.  This is
>>> also a fairly small dataset.  Improvements for larger simulations are even
>>> more impressive.
>>>
>>> You can see the diff between this changeset and Matt's changeset that is
>>> currently the PR changeset here:
>>>
>>> https://bitbucket.org/samskillman/yt-refactor/compare/..MatthewTurk/yt-refactor
>>>
>>> Please take a look.  I'm hopeful that with these changes pulled into
>>> Matt's PR, we should be closer to this being accepted.
>>>
>>> Along with these changes come some updated docs, as well as getting rid
>>> of those pesky print statements that Britton saw.
>>>
>>> Cheers,
>>> Sam
>>>
>>>
>>> On Wed, Jun 13, 2012 at 8:36 PM, Sam Skillman <samskillman at gmail.com>
>>> wrote:
>>>>
>>>> Hi Chris,
>>>>
>>>> Hmm,  maybe Pleiades sets OMP_NUM_THREADS=1 by default.  Can you check:
>>>> echo $OMP_NUM_THREADS
>>>>
>>>> If it is either unset or set to 1, could you give it a shot with export
>>>> OMP_NUM_THREADS=12?  The HT shouldn't help too much.
>>>>
>>>> Best,
>>>> Sam
>>>>
>>>>
>>>> On Wed, Jun 13, 2012 at 8:14 PM, Christopher Moody <cemoody at ucsc.edu>
>>>> wrote:
>>>>>
>>>>> Hi Matt,
>>>>> Yep, this is an ART dataset. I had a node running with 12 (24 with HT)
>>>>> cores at NASA Pleiades. It didn't go above 4% (=1/24) usage, which
>>>>> indicates means it likely did not  use more than one core (but I
>>>>> didn't tell it to). If there's a parallel version of the script I can
>>>>> try it out tonight, and see the scaling.
>>>>>
>>>>> chris
>>>>>
>>>>> On Wed, Jun 13, 2012 at 1:01 PM, Matthew Turk <matthewturk at gmail.com>
>>>>> wrote:
>>>>> > Chris, is this an ART dataset rendering?  I love it.  Great work.
>>>>> > And, did you note on your machine what the load went up to?  Did it
>>>>> > scale with the numebr of cores okay?
>>>>> >
>>>>> > On Wed, Jun 13, 2012 at 3:47 PM, Christopher Moody <cemoody at ucsc.edu>
>>>>> > wrote:
>>>>> >> Works for me!
>>>>> >>
>>>>> >> On one core, no_ghost=True, kd-Tree took 5 seconds, ray casting 56
>>>>> >> seconds. Impressive.
>>>>> >>
>>>>> >> chris
>>>>> >>
>>>>> >> On Wed, Jun 13, 2012 at 10:45 AM, Matthew Turk
>>>>> >> <matthewturk at gmail.com> wrote:
>>>>> >>> Also, for what it's worth, John, Sam and I have been working on
>>>>> >>> creating a movie sequence with this in HiDef for a display project.
>>>>> >>> Once that's done we're going to be posting the movie, the original
>>>>> >>> scripts, and potentially even a screencast of how one goes about
>>>>> >>> doing
>>>>> >>> a complex volume rendering.
>>>>> >>>
>>>>> >>> On Wed, Jun 13, 2012 at 12:34 PM, Sam Skillman
>>>>> >>> <samskillman at gmail.com> wrote:
>>>>> >>>> Hi all,
>>>>> >>>>
>>>>> >>>> Here's a sample script that exposes a few of the new features such
>>>>> >>>> as
>>>>> >>>> opacity and the map_to_colormap function.
>>>>> >>>> http://paste.yt-project.org/show/2458/
>>>>> >>>>
>>>>> >>>> I'll see if I can get some more scripts written in a portable way.
>>>>> >>>>
>>>>> >>>> Cheers,
>>>>> >>>> Sam
>>>>> >>>>
>>>>> >>>> On Wed, Jun 13, 2012 at 10:22 AM, John ZuHone <jzuhone at gmail.com>
>>>>> >>>> wrote:
>>>>> >>>>>
>>>>> >>>>> Matt,
>>>>> >>>>>
>>>>> >>>>> I'll give it a whirl... it may take a day or two.
>>>>> >>>>>
>>>>> >>>>> John Z
>>>>> >>>>>
>>>>> >>>>> On Jun 13, 2012, at 11:12 AM, Matthew Turk wrote:
>>>>> >>>>>
>>>>> >>>>> > Sam and I chatted, and we'd like to request three people test
>>>>> >>>>> > this and
>>>>> >>>>> > if they all agree, we can accept it.
>>>>> >>>>> >
>>>>> >>>>> > Can three people give it a go?
>>>>> >>>>> >
>>>>> >>>>> >
>>>>> >>>>> > ---------- Forwarded message ----------
>>>>> >>>>> > From: Matthew Turk <pullrequests-noreply at bitbucket.org>
>>>>> >>>>> > Date: Wed, Jun 13, 2012 at 11:12 AM
>>>>> >>>>> > Subject: [yt_analysis/yt] Volume Rendering Refactor (pull
>>>>> >>>>> > request #168)
>>>>> >>>>> > To: yt at enzotools.org
>>>>> >>>>> >
>>>>> >>>>> >
>>>>> >>>>> > A new pull request has been opened by Matthew Turk.
>>>>> >>>>> >
>>>>> >>>>> > MatthewTurk/yt-refactor has changes to be pulled into
>>>>> >>>>> > yt_analysis/yt.
>>>>> >>>>> >
>>>>> >>>>> >
>>>>> >>>>> >
>>>>> >>>>> > https://bitbucket.org/yt_analysis/yt/pull-request/168/volume-rendering-refactor
>>>>> >>>>> >
>>>>> >>>>> > Title: Volume Rendering Refactor
>>>>> >>>>> >
>>>>> >>>>> > This includes all of the changes to the volume rendering system
>>>>> >>>>> > that
>>>>> >>>>> > were discussed here:
>>>>> >>>>> >
>>>>> >>>>> >
>>>>> >>>>> >
>>>>> >>>>> > http://lists.spacepope.org/pipermail/yt-dev-spacepope.org/2012-June/002039.html
>>>>> >>>>> >
>>>>> >>>>> > The main user-facing changes that could cause problems:
>>>>> >>>>> >
>>>>> >>>>> > 1) Cython 0.16 is now required.
>>>>> >>>>> > 2) Off-axis projections no longer interpolate.
>>>>> >>>>> >
>>>>> >>>>> > Changes to be pulled:
>>>>> >>>>> >
>>>>> >>>>> >
>>>>> >>>>> >
>>>>> >>>>> > --
>>>>> >>>>> > This is an issue notification from bitbucket.org.
>>>>> >>>>> > You are receiving this either because you are the participating
>>>>> >>>>> > in a pull request, or you are following it.
>>>>> >>>>> > _______________________________________________
>>>>> >>>>> > yt-dev mailing list
>>>>> >>>>> > yt-dev at lists.spacepope.org
>>>>> >>>>> > http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>>>>> >>>>>
>>>>> >>>>> _______________________________________________
>>>>> >>>>> yt-dev mailing list
>>>>> >>>>> yt-dev at lists.spacepope.org
>>>>> >>>>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>>>>> >>>>
>>>>> >>>>
>>>>> >>>>
>>>>> >>>> _______________________________________________
>>>>> >>>> yt-dev mailing list
>>>>> >>>> yt-dev at lists.spacepope.org
>>>>> >>>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>>>>> >>>>
>>>>> >>> _______________________________________________
>>>>> >>> yt-dev mailing list
>>>>> >>> yt-dev at lists.spacepope.org
>>>>> >>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>>>>> >>
>>>>> >> _______________________________________________
>>>>> >> yt-dev mailing list
>>>>> >> yt-dev at lists.spacepope.org
>>>>> >> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>>>>> >>
>>>>> > _______________________________________________
>>>>> > yt-dev mailing list
>>>>> > yt-dev at lists.spacepope.org
>>>>> > http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>>>>> _______________________________________________
>>>>> yt-dev mailing list
>>>>> yt-dev at lists.spacepope.org
>>>>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>>>>
>>>>
>>>
>>>
>>> _______________________________________________
>>> yt-dev mailing list
>>> yt-dev at lists.spacepope.org
>>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>>>
>>
>> _______________________________________________
>> yt-dev mailing list
>> yt-dev at lists.spacepope.org
>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>>
>
>
> _______________________________________________
> yt-dev mailing list
> yt-dev at lists.spacepope.org
> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>

More information about the yt-dev mailing list