[yt-users] Performance when using derived fields

[Daniel Fenn]

Hi Nathan,

I've attached the html output of pyprof2html. I did it for two cases--a
large file on a fast disk, and a somewhat smaller file on a slower disk. In
both cases, I only used one processor. The output is not real meaningful to
me, since I'm not familiar with the code, and I'm not much of a python
programmer.

I'll work on doing this with one of the yt datasets, and let you know how
it goes.

Thanks for your help.

Dan

2014-11-27 0:08 GMT-05:00 Nathan Goldbaum <nathan12343 at gmail.com>:

>
>
> On Wed, Nov 26, 2014 at 8:09 PM, Daniel Fenn <dsfenn at gmail.com> wrote:
>
>> Hi everyone. Is it normal for slices to take substantially longer to be
>> created if they're made from a derived field rather than an intrinsic one?
>>
>> Specifically, I'm having an issue creating an axis-aligned slice using
>> the divergence of the velocity field. It's taking around 6.5 minutes just
>> to make the slice, whereas if I use temperature or density, it takes around
>> 10 seconds or so for the same dataset.
>>
>
> Velocity divergence is particularly expensive because it needs ghost
> zones, which must be filled in using cascading interpolation. Depending on
> the AMR structure in your data, this might be quite expensive.
>
> That said, 6.5 minutes for a slice is pretty bad.
>
> Is there any chance you can profile the execution of your script? If you
> run your script like so:
>
> $ python -m cProfile myscript.py
>
> you should get a summary printed to stdout of where the interpreter is
> spending most of its time. You can also visualize this in more depth if you
> dump the profile data to a file, and then use pyprof2html or snakeviz to
> load and display the data, e.g:
>
> $ python -m cProfile -o out.cprof myscript.py
> $ pyprof2html out.cprof
>
> cProfile comes with pyhon, but pyp4orf2html and snakeviz must be installed
> separately using pip.
>
> It would be great if you could file an issue about this.  Even better if
> you can reproduce it using one of the test datasets on yt-project.org/data
> and attach a script that we can run ourselves.
>
>
>>
>> I also notice that the amount of time it takes is not dependent on the
>> number of processors I'm using. I've used 1, 12, and 24 processors, with
>> identical results, even though I'm calling enable_parallelism(), and I can
>> see that all the processes are running.
>>
>
>> I read in the docs that slice operations aren't generally done in
>> parallel, but in this case it seems that maybe it would be beneficial. A
>> similar operation in VisIt completes much faster, so I'm wondering if I've
>> misconfigured something, or if there is something I can do to speed things
>> up.
>>
>> I'd appreciate any thoughts anyone has on the subject.
>>
>> Thanks,
>>
>> Dan
>>
>> _______________________________________________
>> yt-users mailing list
>> yt-users at lists.spacepope.org
>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>>
>>
>
> _______________________________________________
> yt-users mailing list
> yt-users at lists.spacepope.org
> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.spacepope.org/pipermail/yt-users-spacepope.org/attachments/20141127/9e9665ba/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: html.fastDisk.tar.gz
Type: application/x-gzip
Size: 802191 bytes
Desc: not available
URL: <http://lists.spacepope.org/pipermail/yt-users-spacepope.org/attachments/20141127/9e9665ba/attachment-0002.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: html.slowDisk.tar.gz
Type: application/x-gzip
Size: 801287 bytes
Desc: not available
URL: <http://lists.spacepope.org/pipermail/yt-users-spacepope.org/attachments/20141127/9e9665ba/attachment-0003.bin>