[yt-dev] demeshening sprint summary, plans going forward

[Nathan Goldbaum]

Hi all,

tl;dr: We are working on improving support and performance for particle
data in yt. We want feedback on our ideas and ask for people uninvolved in
this effort to do code reviews and participate in design discussions and
workflow discussions going forward

Last week we had a sprint here at NCSA around the "demeshening" idea a few
of us have been throwing around. The basic idea here is to incorporate the
code Meagan Lang has been working on over the past year into yt, allowing
improved performance for particle codes. One consequence of this work is
that there will no longer be a space-filling octree mesh for SPH data,
instead the data will be indexed by local mesh patches built on compressed
EWAH bitmap indices

As part of this sprint, we had lots of discussions about how this work
might proceed, and people also did some work on moving several aspects of
this effort forward:

I started work on an SPH pixelizer we can use for SlicePlot and
ProjectionPlot:

https://bitbucket.org/yt_analysis/yt/pull-requests/2382

Bili Dong started working on a ray data object that can be used to generate
absorption spectra from SPH data without depositing data onto an octree
mesh:

https://bitbucket.org/yt_analysis/yt/pull-requests/2386

And Matt Turk and Alex Lindsay worked on improvements for the field system:

https://bitbucket.org/yt_analysis/yt/pull-requests/2383
https://bitbucket.org/yt_analysis/yt/pull-requests/2385

Alex's pull request also received some love today and I'd like to request
review on it, as it will unblock further work on the pixelizers. In
particular, I'd like to build on top of Alex's pull request to add a new
kind of sampling_type, "local" which correspond to fully local fields. This
generalizes the current default situation (i.e. the field does not have a
ValidateSpatial validator). My ultimate goal here is to make it easier to
support SPH particle fields. For fields that are fully local, we don't need
to create different implementations for the 'cell' and 'particle'
samping_type, instead we can just use the same implementation for all field
types.

Once that is in, I think I'll be able to make it so projections and slices
work with the majority of yt derived fields using the new SPH pixelizers
I've added in PR 2382:

https://bitbucket.org/yt_analysis/yt/pull-requests/2382

Unfortunately PR 2382 won't be mergeable at that point because we also need
to come up with a solution for fields that require finite differences (this
is discussed a bit in one of Daniel Price's papers, starting around section
3.3.2: http://arxiv.org/pdf/1012.1885v1.pdf)

I think once we've agreed on the way SPH particle fields should work going
forward (I'm currently drafting a YTEP for this) then the demeshening work
can build on top of that in a piecemeal fashion.

If we can I'd like it if we try to upstream stuff early and often to make
this work easier to review and to improve testing by users and other
developers. I don't want to move development into another fork of yt or a
new named branch, since we had issues with that in the past. This approach
might add some work we wouldn't have otherwise done (i.e. we might need to
add compatibility layers that will eventually be torn down), but it will
help keep us honest and ensure we're getting reviews from people uninvolved
in this effort.

That said, if you'd like to try out the new code or even chip in, I'm happy
to pull changes from people's forks or to review pull requests against my
fork of yt.

I think that's it, sorry for the novel! I'd very much appreciate comments
or questions about this.

-Nathan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.spacepope.org/pipermail/yt-dev-spacepope.org/attachments/20160926/bf2e0066/attachment.htm>