[yt-dev] Near- and mid-term roadmap: 2.4, 2.x, 3.0

[Matthew Turk]

On Tue, Aug 7, 2012 at 11:59 AM, Samuel W Skillman
<Samuel.Skillman at colorado.edu> wrote:
> Matt,
>
> On Tue, Aug 7, 2012 at 9:55 AM, Matthew Turk <matthewturk at gmail.com> wrote:
>>
>> Sam,
>>
>> On Tue, Aug 7, 2012 at 11:51 AM, Sam Skillman <samskillman at gmail.com>
>> wrote:
>> > My goals for 2.5:
>> >
>> > Re-written, modularized, amr_kdtree.  This will allow for kd-tree decomp
>> > of
>> > generalized collections of grids as well as future optimization, as well
>> > as
>> > parallelization over arbitrary N processors, not just power of 2!
>> > Alpha channel integration for the rendering, allowing for easier control
>> > of
>> > opaque and transparent features in the same image.
>>
>> This sounds really awesome.
>>
>> >
>> > Neither of these necessarily belong in the 3.0 development IMO, but I
>> > could
>> > be convinced to work in that sandbox.  Both of these are working in an
>> > alpha
>> > state right now.
>>
>> I agree, these don't *need* to be in 3.0, so let's wait till that
>> branch is feature complete before I start selling you on moving the
>> dev there.  :)
>
>
> Sounds good.
>
>>
>>
>> ...but ... do you think the walk and the decomp could be separated, so
>> we could feed it an octree?  In the 3.0 branch I've got an oct
>> geometry handler which could potentially be instrumented with the
>> necessary additional info to allow fast traversal...
>
>
> Yes.  I think it should be straightforward to write a modified decomp for
> octree that retains the reduction methods for combining the final image.
> Then we can just pass each unit of octree to the fast traversal, whatever
> that ends up being.

Maybe this is what you are already thinking -- but perhaps
restructuring the way the camera is setup could follow something like:

1) Decompose volume based on data source
2) Setup image plane
3) Traverse volume

And that step 1 could call a method on the geometry handler, which
would dispatch differently for the Octs versus for the Patch
refinement.

>
>>
>>
>> -Matt
>>
>> >
>> > Sam
>> >
>> >
>> > On Tue, Aug 7, 2012 at 9:46 AM, Matthew Turk <matthewturk at gmail.com>
>> > wrote:
>> >>
>> >> Hi all,
>> >>
>> >> Now that 2.4 is out, I wanted to update you on where this roadmap
>> >> (from early June, two months ago today!) is.  It slipped a bit, but I
>> >> think we're still broadly on track.
>> >>
>> >> My current plan is to focus on development of the 3.0 branch/fork,
>> >> which is located here:
>> >>
>> >> http://bitbucket.org/yt_analysis/yt-3.0
>> >>
>> >> It is nearly functional for all major functions from the patch-based
>> >> perspective, but I am still working on projections and a handful of
>> >> other operations -- flushing data back to grids (for clump finding),
>> >> masking of cells (for extracted regions), and a few other medium-level
>> >> features.  Once it has reached feature completeness for patch-based, I
>> >> think it will be ready to have many of you start testing it, kicking
>> >> the tires, maybe even moving your development there.  I'll update this
>> >> group as I make progress, but I'd also be very appreciative if anyone
>> >> wants to start testing it with me and working on it.  I'm quite
>> >> excited about some aspects of it, and I've already been pleased with
>> >> simplifications in several areas.
>> >>
>> >> We should still probably leave open the possibility of a 2.5 release,
>> >> but with a much shorter changelog than 2.4 had!  :)
>> >>
>> >> I'll be around in IRC as well as on the mailing list if anyone has any
>> >> concerns, suggestions, or wants to get involved in this.
>> >>
>> >> -Matt
>> >>
>> >> On Thu, Jun 7, 2012 at 4:50 PM, Matthew Turk <matthewturk at gmail.com>
>> >> wrote:
>> >> > Hi all,
>> >> >
>> >> > Sam and I are working on issuing a PR about the volume rendering
>> >> > refactor.  This is the last major code change to go in place for 2.4,
>> >> > pending the quad_proj changes being accepted or rejected.  I'm still
>> >> > thinking about this refine_by issue, and that might also make it in.
>> >> > After the volume_refactor, it's just testing and documentation left:
>> >> >
>> >> >
>> >> >
>> >> > https://bitbucket.org/yt_analysis/yt/issues?status=new&status=open&milestone=2.4
>> >> >
>> >> > After 2.4 is released, I am going to shift my new development
>> >> > exclusively to the yt-3.0 branch, which is currently hosted here:
>> >> >
>> >> > http://bitbucket.org/yt_analysis/yt-3.0
>> >> >
>> >> > This will include geometry refactoring and many remaining
>> >> > de-Enzoifications of the code base.  3.0 is nearing a usable state,
>> >> > but will not be ready for production for a while.  However, I think
>> >> > that allowing the codebases to diverge too rapidly will lead to
>> >> > problems: a fracturing.  So once the 3.0 codebase can muster feature
>> >> > parity with 2.x, I am going to ask that all development on the 2.x
>> >> > branch cease, and all pull requests for features and new
>> >> > functionality
>> >> > be applied only to the 3.0 branch.  In advance of this, I would like
>> >> > to solicit help: we've identified a number of relatively
>> >> > straightforward changes to this code base that can and should be
>> >> > made.
>> >> >  For instance, the changing of field names.  I'd also like to
>> >> > consider
>> >> > moving to a more spread-out source code layout, like one class per
>> >> > file.  Changes like these make it difficult to do cross-branch merges
>> >> > and transplants, which is why I would like to save them for after
>> >> > feature parity has been reached and development on 2.x has halted.
>> >> >
>> >> > Previous threads:
>> >> >
>> >> >
>> >> >
>> >> > http://lists.spacepope.org/pipermail/yt-dev-spacepope.org/2012-February/001852.html
>> >> >
>> >> >
>> >> > http://lists.spacepope.org/pipermail/yt-dev-spacepope.org/2012-February/001868.html
>> >> >
>> >> >
>> >> > http://lists.spacepope.org/pipermail/yt-dev-spacepope.org/2012-February/001830.html
>> >> >
>> >> >
>> >> > http://lists.spacepope.org/pipermail/yt-dev-spacepope.org/2012-March/001941.html
>> >> >
>> >> >
>> >> > http://lists.spacepope.org/pipermail/yt-dev-spacepope.org/2012-March/001949.html
>> >> >
>> >> > If anyone has any strong objections to this strategy, please reply.
>> >> >
>> >> > Estimated Timeline:
>> >> >
>> >> >  * June 26, 2.4 release
>> >> >  * July 30, 3.0 reaches rough feature parity
>> >> >  * Fall sometime: 3.0 merged into main 'yt' branch
>> >> >  * Some time after that: 3.0 released
>> >> >
>> >> > Criteria for Feature Parity:
>> >> >
>> >> >  * All patch-based AMR codes support data region selection
>> >> >  * Parallelism is functional for all parallel-capable routines
>> >> >  * All patch-based AMR codes support projections, slices, cutting
>> >> > planes
>> >> >  * All patch-based AMR codes support volume rendering
>> >> >  * All patch-based AMR codes support halo finding
>> >> >  * Octree-based AMR codes (RAMSES and ART) support data region
>> >> > selection
>> >> >  * Octree-based AMR codes (RAMSES and ART) support slices,
>> >> > projections
>> >> > and cutting planes
>> >> >
>> >> > If any of you are interested in helping with this reengineering
>> >> > effort, I would be extremely interested in collaborating.  My goal
>> >> > for
>> >> > this effort is to focus on making yt a future-compatible code, and
>> >> > changing it from an organically grown system into one that we design
>> >> > based on the years of experience we all have now had with it.
>> >> >
>> >> > -Matt
>> >> _______________________________________________
>> >> yt-dev mailing list
>> >> yt-dev at lists.spacepope.org
>> >> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>> >
>> >
>> >
>> > _______________________________________________
>> > yt-dev mailing list
>> > yt-dev at lists.spacepope.org
>> > http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>> >
>> _______________________________________________
>> yt-dev mailing list
>> yt-dev at lists.spacepope.org
>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>
>
>
>
> --
> Samuel W. Skillman
> DOE Computational Science Graduate Fellow
> Center for Astrophysics and Space Astronomy
> University of Colorado at Boulder
> samuel.skillman[at]colorado.edu
>
> _______________________________________________
> yt-dev mailing list
> yt-dev at lists.spacepope.org
> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>