[yt-users] in-situ visualization possible with yt?

Tue Nov 8 11:39:59 PST 2011

Hi Matthieu,

There's been quite a bit of interest in providing data to yt from
in-memory datasets.  You can try something like this:

http://hg.yt-project.org/yt/wiki/InterfacingWithYT

The idea would be that if you *have* the arrays, you should be able to
describe their mesh to yt very easily.  Unfortunately, the API is as
you can see clunky.  But I must warn you, the interface is in flux.
Fixing this up is a goal for the month of December, both documentation
and API-wise.  Would you be interested in helping us make this process
more streamlined?

-Matt

On Tue, Nov 8, 2011 at 2:31 PM, Matthieu Dorier
<matthieu.dorier at gmail.com> wrote:
> Thanks, I managed to make what I wanted with matplotlib.
>
> Matthieu
>
> 2011/11/8 Britton Smith <brittonsmith at gmail.com>
>>
>> Hi Matthieu,
>>
>> If you already have arrays that you are ready to plot, you are probably
>> better going straight to matplotlib from here.  You can find their
>> documentation here: http://matplotlib.sourceforge.net/
>> This is the package used for most of the plotting and visualization that
>> yt does.
>>
>> For example, you would probably want to use pcolormesh to plot up a 2D
>> numpy array:
>>
>> http://matplotlib.sourceforge.net/api/pyplot_api.html#matplotlib.pyplot.pcolormesh
>>
>> Britton
>>
>> On Tue, Nov 8, 2011 at 8:50 AM, Matthieu Dorier
>> <matthieu.dorier at gmail.com> wrote:
>>>
>>> Hello,
>>> I come back to this discussion, as I have instrumented my simulation to
>>> wrap the C++ arrays into NumPy arrays and call a python script in each
>>> process.
>>> This currently works and now I would like to actually do something with
>>> these NumPy arrays. I can't find any tutorial on the YT homepage on how to
>>> create an image from a NumPy array.
>>> (I'm not only new to YT, I'm also new to Python and learned the basic
>>> syntax while coding the wrapper).
>>> Let's imagine I have a 2D NumPy array of floating point values, and I
>>> want to plot it as an image, how would the python code look like?
>>> Thanks
>>> Matthieu
>>> 2011/10/5 Matthew Turk <matthewturk at gmail.com>
>>>>
>>>> Hi Matthieu,
>>>>
>>>> On Wed, Oct 5, 2011 at 11:45 AM, Matthieu Dorier
>>>> <matthieu.dorier at gmail.com> wrote:
>>>> > Hello Matthew,
>>>> >
>>>> > Thank you for your answer.
>>>> >
>>>> > Can YT work in parallel?
>>>>
>>>> Yup.
>>>>
>>>> > I intend to embed a python interpreter within my simulation in order
>>>> > to
>>>> > allow
>>>> > some simple data analysis to be performed without having to re-compile
>>>> > the
>>>> > application every time this analysis changes, so maybe it would be
>>>> > possible
>>>> > to call YT from an embedded python interpreter? I guess the problem is
>>>> > transferring raw data from the simulation to the interpreter without
>>>> > too
>>>> > much
>>>> > overhead...
>>>>
>>>> That's actually how Enzo works as well.  We then construct numpy array
>>>> wrappers around the already-allocated data, and then feed those to yt
>>>> in a dict.
>>>>
>>>> >
>>>> > My output is actually very simple, consisting in 2D arrays, each MPI
>>>> > rank
>>>> > being responsible for several chunks (equally sized, but not
>>>> > necessarily
>>>> > contiguous). I would like to avoid writing these 2D arrays into HDF5
>>>> > file
>>>> > and perform visualization on these files, but automatically make the
>>>> > processes
>>>> > synchronize to compute a single image from all these parts.
>>>>
>>>> That's what we do with Enzo, except with 3D.  There may be some
>>>> oddities to work out with 2D data.
>>>>
>>>> >
>>>> > Do you think it is worth investigating the use of YT for that or
>>>> > should I
>>>> > try other
>>>> > softwares?
>>>> > Thanks
>>>> >
>>>> > Matthieu
>>>>
>>>> I think you should give a look at yt -- but also guide your views by
>>>> reading how we do in situ with Enzo.  If you look in the Enzo source
>>>> code at InitializePythonInterface.C and ExposeDataHierarchy.C you can
>>>> see our current method, although I believe in the future with better
>>>> memory-pool allocations in Enzo we'll make this a bit more
>>>> streamlined.
>>>>
>>>> Please don't hesitate to write back with more questions!
>>>>
>>>> Best,
>>>>
>>>> Matt
>>>>
>>>> >
>>>> > 2011/10/5 Matthew Turk <matthewturk at gmail.com>
>>>> >>
>>>> >> Hi Matthieu,
>>>> >>
>>>> >> The short answer is that for some codes, yes, yt works as an
>>>> >> embedding
>>>> >> system.  The only code that this has been used extensively in is
>>>> >> Enzo,
>>>> >> and it will likely take a bit of work to get it to work.  We're eager
>>>> >> to test it out on other codes, however; the first and easiest
>>>> >> application would be a non-Enzo, patch-based AMR code.
>>>> >>
>>>> >> The current state is:
>>>> >>
>>>> >>  * In-memory between Enzo and yt works (Britton Smith can speak more
>>>> >> to
>>>> >> this)
>>>> >>  * Creation of arbitrary sources (using the Stream frontend)
>>>> >> currently
>>>> >> works, which can be used from within any Python code
>>>> >>  * No protocol exists (yet) for MPI inter-communicator data loading
>>>> >>
>>>> >> Our strategies for the future:
>>>> >>
>>>> >>  * We're working to create simple C libraries that can create the
>>>> >> necessary in-memory data objects to link against yt directly
>>>> >>  * The Stream frontend is usable and should be used by non-Enzo codes
>>>> >> as a prototyping mechanism
>>>> >>  * ParaView can call yt for serial calculations, and we are working
>>>> >> to
>>>> >> utilize its CoProcessing facilities to analyze and visualize in
>>>> >> parallel from running simulations
>>>> >>
>>>> >> So I think you should be able to add a code to in situ viz/analysis,
>>>> >> but it will likely take a bit of work at the current time to make
>>>> >> sure
>>>> >> everything correctly passes.  I'd be able to help you and send you
>>>> >> demo code if you would like.  Alternately, we intend to instrument yt
>>>> >> to use ParaView as a data loader, which will enable the CoProcessing
>>>> >> techniques from ParaView to be used as well.
>>>> >>
>>>> >> Let me know what you think,
>>>> >>
>>>> >> Matt
>>>> >>
>>>> >> On Tue, Oct 4, 2011 at 1:28 PM, Matthieu Dorier
>>>> >> <matthieu.dorier at gmail.com> wrote:
>>>> >> > Hello,
>>>> >> >
>>>> >> > I'm trying to make in-situ visualization available for a
>>>> >> > simulation,
>>>> >> > and I'm considering different softwares for that (VisIt, YT,...).
>>>> >> > I've never used YT, so my question is: is it possible to add
>>>> >> > some code in a simulation (written in C++) to have some kind of
>>>> >> > distributed rendering engine using YT, just like VisIt does with
>>>> >> > code-instrumentation?
>>>> >> > Thanks,
>>>> >> >
>>>> >> > Matthieu
>>>> >> >
>>>> >> > --
>>>> >> > Matthieu Dorier
>>>> >> > ENS Cachan, antenne de Bretagne
>>>> >> > Département informatique et télécommunication
>>>> >> > http://perso.eleves.bretagne.ens-cachan.fr/~mdori307/wiki/
>>>> >> >
>>>> >> > _______________________________________________
>>>> >> > yt-users mailing list
>>>> >> > yt-users at lists.spacepope.org
>>>> >> > http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>>>> >> >
>>>> >> >
>>>> >> _______________________________________________
>>>> >> yt-users mailing list
>>>> >> yt-users at lists.spacepope.org
>>>> >> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>>>> >
>>>> >
>>>> >
>>>> > --
>>>> > Matthieu Dorier
>>>> > ENS Cachan, antenne de Bretagne
>>>> > Département informatique et télécommunication
>>>> > http://perso.eleves.bretagne.ens-cachan.fr/~mdori307/wiki/
>>>> >
>>>> > _______________________________________________
>>>> > yt-users mailing list
>>>> > yt-users at lists.spacepope.org
>>>> > http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>>>> >
>>>> >
>>>> _______________________________________________
>>>> yt-users mailing list
>>>> yt-users at lists.spacepope.org
>>>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>>>
>>>
>>>
>>> --
>>> Matthieu Dorier
>>> ENS Cachan, antenne de Bretagne
>>> Département informatique et télécommunication
>>> http://perso.eleves.bretagne.ens-cachan.fr/~mdori307/wiki/
>>>
>>> _______________________________________________
>>> yt-users mailing list
>>> yt-users at lists.spacepope.org
>>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>>>
>>
>>
>> _______________________________________________
>> yt-users mailing list
>> yt-users at lists.spacepope.org
>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>>
>
>
>
> --
> Matthieu Dorier
> ENS Cachan, antenne de Bretagne
> Département informatique et télécommunication
> http://perso.eleves.bretagne.ens-cachan.fr/~mdori307/wiki/
>
> _______________________________________________
> yt-users mailing list
> yt-users at lists.spacepope.org
> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>
>