[yt-users] Bug Report: Memory leak when rendering projection plots in parallel

Gabriel Goodwin ggoodwin52 at gmail.com
Wed Jun 10 10:17:09 PDT 2015


Hello,

I believe I have found a memory leak when iterating through a DatasetSeries
in parallel to render projection plots of each data file in the series. I
work in a CFD research group and our code is built on boxlib, so I have
been using yt to generate images of my simulations. The DatasetSeries's I
work with are fairly large, on the order of 6000 to 7000 data sets per
series. To expedite the image rendering process, I have been using yt's
parallelism to iterate through the series with openmpi.

The ram usage, per mpi process, increases steadily throughout the duration
of the script's runtime. When running my attached script with 16 mpi
processes on a 16 core machine (1 process per core) the ram usage starts at
about 2.56 GB per process and gradually increases to over 25 GB per
process, at which time the script typically crashes due to insufficient ram
on the machine to continue running.

My simulations do involve data sets of increasing memory requirements
versus simulation time, but not to the scale mentioned above in the
corresponding ram increases. I determined that the crashing is due to a
memory leak by starting the script over after it crashed, using only the
data set files that had not yet been processed (to ensure that the memory
required to render the projection of each data set is the same as it was
before it crashed by starting over at the same time in the simulation as
when it crashed).

I also ran this script on my university's cluster, using several different
openmpi implementations, and always got the same result. Additionally, the
crashes always occur at approximately the same script runtime.

Attached to this email is a copy of a script I'm using. I run the script on
my 16 core mini-cluster using the command:

mpirun -np 16 --cpus-per-proc 1 br0.05_temp.py

I also attached a sample image of the projection plots I'm generating.

Any help with this bug is appreciated!

Thanks very much,
Gabe
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.spacepope.org/pipermail/yt-users-spacepope.org/attachments/20150610/aa0be2d8/attachment.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 000139400.png
Type: image/png
Size: 211853 bytes
Desc: not available
URL: <http://lists.spacepope.org/pipermail/yt-users-spacepope.org/attachments/20150610/aa0be2d8/attachment.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: br0.05_temp.py
Type: application/octet-stream
Size: 1736 bytes
Desc: not available
URL: <http://lists.spacepope.org/pipermail/yt-users-spacepope.org/attachments/20150610/aa0be2d8/attachment.obj>


More information about the yt-users mailing list