[yt-dev] Fwd: [yt_analysis/yt] Volume Rendering Refactor (pull request #168)

Sam Skillman samskillman at gmail.com
Wed Jun 13 21:00:37 PDT 2012 

Hi Matt,

Thanks for the kind words.  I'm glad that I'm in a community of developers
who 1) care about results, 2) Have buy-in because it's what we use to get
work done, 3) are cool people!

Chris -- I think the delay between the building and the casting is when the
data is loaded into the kdtree bricks.  Both the building of the kdtree and
the reading of the data are serial processes.  So the relavent speedup is
the one during the ray-casting.  One other thing to note is that if you are
doing camera movements, the data should all stay in memory so these serial
costs should go away.  Anyways, thanks for testing!  That's some great
performance!

Cheers,
Sam

On Wed, Jun 13, 2012 at 9:40 PM, Matthew Turk <matthewturk at gmail.com> wrote:

> Hi Sam,
>
> Thanks for doing this.
>
> I'll update my PR shortly; this needs to be a part of it before it's
> accepted.
>
> This has also highlighted a part of the process that I think we're all
> trying to do -- peer review not just of the quantitative changes to the
> code (i.e., breaking tests!) but also qualitative.  In this case, Cameron
> saw something that would change functionality, and he stood up for keeping
> the ability to retain backwards compatibility.  I think this is a valid
> point and a valuable position; in the case of the off-axis projection, when
> I initially made this change it was because I felt this was truer to the
> underlying data.  (There are extremely few places in the yt codebase where
> it interpolates.)
>
> However, retaining a solid foundation for users is also extremely
> important.  The growth of yt over the last couple years has put us in a
> sticky situation; we've developed a community, but it's a community that
> relies on functionality not changing without good reason, and more
> importantly that relies on keeping a solid basis for repeatable analysis
> and consistent results.  This change broke that -- even though it's
> arguably an improvement of the solution -- and the old method was no longer
> available.
>
> I'm really happy that this system has worked; while I was initially -1 on
> restoring interpolation, Cameron swayed me in IRC by bringing up this
> remarkably important point.  I think turning the default to
> non-interpolated but retaining the interpolated mode are both the right
> decisions, and I'm +1.  And more than that, I'm glad this all happened
> before the PR was accepted.
>
> So thank you, Sam, and Cameron, for restoring the old functionality.  I'm
> excited about moving forward.  The results in this thread have really
> gotten me excited about the success of this new set of changes.
>
> -Matt
> On Jun 13, 2012 11:26 PM, "Sam Skillman" <samskillman at gmail.com> wrote:
>
>> Hi all,
>>
>> Earlier today Cameron brought up some concerns about removing the
>> behavior of the off-axis-projection using interpolated data from
>> vertex-centered data.  If you recall from earlier discussions on the impact
>> of the new threading + opaque renderings + refactoring, one of the things
>> we've done is switch from the interpolation to using the raw values within
>> each cell a given ray traverses for use in the off-axis-projection.  This
>> is mathematically a more accurate representation of the data, but can
>> sometimes lead to blocky looking images.
>>
>> What I've done is go through and add an optional
>> interpolated=(False/True) argument to the off_axis_projection call, where
>> the default is False.  Additionally, I've set this up so that it now wraps
>> the ProjectionCamera class, which is derived from the Camera class, so it
>> has things like .rotate, etc.
>>
>> Just for reference, I'm including below both timings, results, and
>> example images, made using this script:
>> http://paste.yt-project.org/show/2462/
>>
>> Here is the output:
>> Comparison of Raw vs Interpolated
>> Density
>> Raw -- Time: 1.173431e+00 seconds, min: 8.046089e-06, max: 1.177795e-02
>> Int -- Time: 6.393837e+00 seconds, min: 8.322453e-06, max: 1.113421e-02
>>
>> Density-weighted Temperature
>> Raw -- Time: 8.513930e-01 seconds, min: 4.681834e+00, max: 1.237399e+07
>> Int -- Time: 8.450805e+00 seconds, min: 1.105447e+01, max: 1.213922e+07
>>
>> The main difference that you see here is that the interpolated doesn't
>> capture the minima and maxima as well, which makes sense since it is
>> interpolating the data and smoothing out the peaks.  Note that the
>> Interpolated use the no_ghost=False option, which brings it much closer to
>> the Raw values.  This highlights the improvement off the
>> off_axis_projection for quantitative results, as it is both more accurate
>> and faster.  This is also a fairly small dataset.  Improvements for larger
>> simulations are even more impressive.
>>
>> You can see the diff between this changeset and Matt's changeset that is
>> currently the PR changeset here:
>>
>> https://bitbucket.org/samskillman/yt-refactor/compare/..MatthewTurk/yt-refactor
>>
>> Please take a look.  I'm hopeful that with these changes pulled into
>> Matt's PR, we should be closer to this being accepted.
>>
>> Along with these changes come some updated docs, as well as getting rid
>> of those pesky print statements that Britton saw.
>>
>> Cheers,
>> Sam
>>
>>
>> On Wed, Jun 13, 2012 at 8:36 PM, Sam Skillman <samskillman at gmail.com>wrote:
>>
>>> Hi Chris,
>>>
>>> Hmm,  maybe Pleiades sets OMP_NUM_THREADS=1 by default.  Can you check:
>>> echo $OMP_NUM_THREADS
>>>
>>> If it is either unset or set to 1, could you give it a shot with export
>>> OMP_NUM_THREADS=12?  The HT shouldn't help too much.
>>>
>>> Best,
>>> Sam
>>>
>>>
>>> On Wed, Jun 13, 2012 at 8:14 PM, Christopher Moody <cemoody at ucsc.edu>wrote:
>>>
>>>> Hi Matt,
>>>> Yep, this is an ART dataset. I had a node running with 12 (24 with HT)
>>>> cores at NASA Pleiades. It didn't go above 4% (=1/24) usage, which
>>>> indicates means it likely did not  use more than one core (but I
>>>> didn't tell it to). If there's a parallel version of the script I can
>>>> try it out tonight, and see the scaling.
>>>>
>>>> chris
>>>>
>>>> On Wed, Jun 13, 2012 at 1:01 PM, Matthew Turk <matthewturk at gmail.com>
>>>> wrote:
>>>> > Chris, is this an ART dataset rendering?  I love it.  Great work.
>>>> > And, did you note on your machine what the load went up to?  Did it
>>>> > scale with the numebr of cores okay?
>>>> >
>>>> > On Wed, Jun 13, 2012 at 3:47 PM, Christopher Moody <cemoody at ucsc.edu>
>>>> wrote:
>>>> >> Works for me!
>>>> >>
>>>> >> On one core, no_ghost=True, kd-Tree took 5 seconds, ray casting 56
>>>> >> seconds. Impressive.
>>>> >>
>>>> >> chris
>>>> >>
>>>> >> On Wed, Jun 13, 2012 at 10:45 AM, Matthew Turk <
>>>> matthewturk at gmail.com> wrote:
>>>> >>> Also, for what it's worth, John, Sam and I have been working on
>>>> >>> creating a movie sequence with this in HiDef for a display project.
>>>> >>> Once that's done we're going to be posting the movie, the original
>>>> >>> scripts, and potentially even a screencast of how one goes about
>>>> doing
>>>> >>> a complex volume rendering.
>>>> >>>
>>>> >>> On Wed, Jun 13, 2012 at 12:34 PM, Sam Skillman <
>>>> samskillman at gmail.com> wrote:
>>>> >>>> Hi all,
>>>> >>>>
>>>> >>>> Here's a sample script that exposes a few of the new features such
>>>> as
>>>> >>>> opacity and the map_to_colormap function.
>>>> >>>> http://paste.yt-project.org/show/2458/
>>>> >>>>
>>>> >>>> I'll see if I can get some more scripts written in a portable way.
>>>> >>>>
>>>> >>>> Cheers,
>>>> >>>> Sam
>>>> >>>>
>>>> >>>> On Wed, Jun 13, 2012 at 10:22 AM, John ZuHone <jzuhone at gmail.com>
>>>> wrote:
>>>> >>>>>
>>>> >>>>> Matt,
>>>> >>>>>
>>>> >>>>> I'll give it a whirl... it may take a day or two.
>>>> >>>>>
>>>> >>>>> John Z
>>>> >>>>>
>>>> >>>>> On Jun 13, 2012, at 11:12 AM, Matthew Turk wrote:
>>>> >>>>>
>>>> >>>>> > Sam and I chatted, and we'd like to request three people test
>>>> this and
>>>> >>>>> > if they all agree, we can accept it.
>>>> >>>>> >
>>>> >>>>> > Can three people give it a go?
>>>> >>>>> >
>>>> >>>>> >
>>>> >>>>> > ---------- Forwarded message ----------
>>>> >>>>> > From: Matthew Turk <pullrequests-noreply at bitbucket.org>
>>>> >>>>> > Date: Wed, Jun 13, 2012 at 11:12 AM
>>>> >>>>> > Subject: [yt_analysis/yt] Volume Rendering Refactor (pull
>>>> request #168)
>>>> >>>>> > To: yt at enzotools.org
>>>> >>>>> >
>>>> >>>>> >
>>>> >>>>> > A new pull request has been opened by Matthew Turk.
>>>> >>>>> >
>>>> >>>>> > MatthewTurk/yt-refactor has changes to be pulled into
>>>> yt_analysis/yt.
>>>> >>>>> >
>>>> >>>>> >
>>>> >>>>> >
>>>> https://bitbucket.org/yt_analysis/yt/pull-request/168/volume-rendering-refactor
>>>> >>>>> >
>>>> >>>>> > Title: Volume Rendering Refactor
>>>> >>>>> >
>>>> >>>>> > This includes all of the changes to the volume rendering system
>>>> that
>>>> >>>>> > were discussed here:
>>>> >>>>> >
>>>> >>>>> >
>>>> >>>>> >
>>>> http://lists.spacepope.org/pipermail/yt-dev-spacepope.org/2012-June/002039.html
>>>> >>>>> >
>>>> >>>>> > The main user-facing changes that could cause problems:
>>>> >>>>> >
>>>> >>>>> > 1) Cython 0.16 is now required.
>>>> >>>>> > 2) Off-axis projections no longer interpolate.
>>>> >>>>> >
>>>> >>>>> > Changes to be pulled:
>>>> >>>>> >
>>>> >>>>> >
>>>> >>>>> >
>>>> >>>>> > --
>>>> >>>>> > This is an issue notification from bitbucket.org.
>>>> >>>>> > You are receiving this either because you are the participating
>>>> >>>>> > in a pull request, or you are following it.
>>>> >>>>> > _______________________________________________
>>>> >>>>> > yt-dev mailing list
>>>> >>>>> > yt-dev at lists.spacepope.org
>>>> >>>>> > http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>>>> >>>>>
>>>> >>>>> _______________________________________________
>>>> >>>>> yt-dev mailing list
>>>> >>>>> yt-dev at lists.spacepope.org
>>>> >>>>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>>>> >>>>
>>>> >>>>
>>>> >>>>
>>>> >>>> _______________________________________________
>>>> >>>> yt-dev mailing list
>>>> >>>> yt-dev at lists.spacepope.org
>>>> >>>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>>>> >>>>
>>>> >>> _______________________________________________
>>>> >>> yt-dev mailing list
>>>> >>> yt-dev at lists.spacepope.org
>>>> >>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>>>> >>
>>>> >> _______________________________________________
>>>> >> yt-dev mailing list
>>>> >> yt-dev at lists.spacepope.org
>>>> >> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>>>> >>
>>>> > _______________________________________________
>>>> > yt-dev mailing list
>>>> > yt-dev at lists.spacepope.org
>>>> > http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>>>> _______________________________________________
>>>> yt-dev mailing list
>>>> yt-dev at lists.spacepope.org
>>>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>>>>
>>>
>>>
>>
>> _______________________________________________
>> yt-dev mailing list
>> yt-dev at lists.spacepope.org
>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>>
>>
> _______________________________________________
> yt-dev mailing list
> yt-dev at lists.spacepope.org
> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.spacepope.org/pipermail/yt-dev-spacepope.org/attachments/20120613/c28107b4/attachment.htm>

More information about the yt-dev mailing list