[yt-users] in-situ visualization possible with yt?

[Matthew Turk]

Hi Matthieu,

On Wed, Oct 5, 2011 at 11:45 AM, Matthieu Dorier
<matthieu.dorier at gmail.com> wrote:
> Hello Matthew,
>
> Thank you for your answer.
>
> Can YT work in parallel?

Yup.

> I intend to embed a python interpreter within my simulation in order to
> allow
> some simple data analysis to be performed without having to re-compile the
> application every time this analysis changes, so maybe it would be possible
> to call YT from an embedded python interpreter? I guess the problem is
> transferring raw data from the simulation to the interpreter without too
> much
> overhead...

That's actually how Enzo works as well.  We then construct numpy array
wrappers around the already-allocated data, and then feed those to yt
in a dict.

>
> My output is actually very simple, consisting in 2D arrays, each MPI rank
> being responsible for several chunks (equally sized, but not necessarily
> contiguous). I would like to avoid writing these 2D arrays into HDF5 file
> and perform visualization on these files, but automatically make the
> processes
> synchronize to compute a single image from all these parts.

That's what we do with Enzo, except with 3D.  There may be some
oddities to work out with 2D data.

>
> Do you think it is worth investigating the use of YT for that or should I
> try other
> softwares?
> Thanks
>
> Matthieu

I think you should give a look at yt -- but also guide your views by
reading how we do in situ with Enzo.  If you look in the Enzo source
code at InitializePythonInterface.C and ExposeDataHierarchy.C you can
see our current method, although I believe in the future with better
memory-pool allocations in Enzo we'll make this a bit more
streamlined.

Please don't hesitate to write back with more questions!

Best,

Matt

>
> 2011/10/5 Matthew Turk <matthewturk at gmail.com>
>>
>> Hi Matthieu,
>>
>> The short answer is that for some codes, yes, yt works as an embedding
>> system.  The only code that this has been used extensively in is Enzo,
>> and it will likely take a bit of work to get it to work.  We're eager
>> to test it out on other codes, however; the first and easiest
>> application would be a non-Enzo, patch-based AMR code.
>>
>> The current state is:
>>
>>  * In-memory between Enzo and yt works (Britton Smith can speak more to
>> this)
>>  * Creation of arbitrary sources (using the Stream frontend) currently
>> works, which can be used from within any Python code
>>  * No protocol exists (yet) for MPI inter-communicator data loading
>>
>> Our strategies for the future:
>>
>>  * We're working to create simple C libraries that can create the
>> necessary in-memory data objects to link against yt directly
>>  * The Stream frontend is usable and should be used by non-Enzo codes
>> as a prototyping mechanism
>>  * ParaView can call yt for serial calculations, and we are working to
>> utilize its CoProcessing facilities to analyze and visualize in
>> parallel from running simulations
>>
>> So I think you should be able to add a code to in situ viz/analysis,
>> but it will likely take a bit of work at the current time to make sure
>> everything correctly passes.  I'd be able to help you and send you
>> demo code if you would like.  Alternately, we intend to instrument yt
>> to use ParaView as a data loader, which will enable the CoProcessing
>> techniques from ParaView to be used as well.
>>
>> Let me know what you think,
>>
>> Matt
>>
>> On Tue, Oct 4, 2011 at 1:28 PM, Matthieu Dorier
>> <matthieu.dorier at gmail.com> wrote:
>> > Hello,
>> >
>> > I'm trying to make in-situ visualization available for a simulation,
>> > and I'm considering different softwares for that (VisIt, YT,...).
>> > I've never used YT, so my question is: is it possible to add
>> > some code in a simulation (written in C++) to have some kind of
>> > distributed rendering engine using YT, just like VisIt does with
>> > code-instrumentation?
>> > Thanks,
>> >
>> > Matthieu
>> >
>> > --
>> > Matthieu Dorier
>> > ENS Cachan, antenne de Bretagne
>> > Département informatique et télécommunication
>> > http://perso.eleves.bretagne.ens-cachan.fr/~mdori307/wiki/
>> >
>> > _______________________________________________
>> > yt-users mailing list
>> > yt-users at lists.spacepope.org
>> > http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>> >
>> >
>> _______________________________________________
>> yt-users mailing list
>> yt-users at lists.spacepope.org
>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>
>
>
> --
> Matthieu Dorier
> ENS Cachan, antenne de Bretagne
> Département informatique et télécommunication
> http://perso.eleves.bretagne.ens-cachan.fr/~mdori307/wiki/
>
> _______________________________________________
> yt-users mailing list
> yt-users at lists.spacepope.org
> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>
>