[yt-users] Parallelism in yt Applied to Large Datasets

Britton Smith brittonsmith at gmail.com
Wed Dec 6 13:12:26 PST 2017


Running out of RAM when iterating over datasets is also pretty common.
This might have something to do with how the iterator yields a loaded
dataset.  Either way, one thing that I've found helps is to explicitly
delete objects and clear out data.  Every yt data container has a
clear_data function that you can call to free up field arrays.  You might
add that as well as explicitly deleting the objects.  In addition, doing
your own garbage collection with gc.collect() seems to help quite a bit as
well.  Even then, I still often see RAM issues when doing this type of
analysis and so I will typically implement a script that can be restarted
from where it left off, usually by checking for the existence of output
files.

Britton

On Wed, Dec 6, 2017 at 12:21 PM, Nathan Goldbaum <nathan12343 at gmail.com>
wrote:

>
>
> On Wed, Dec 6, 2017 at 2:17 PM, Jason Galyardt <jason.galyardt at gmail.com>
> wrote:
>
>> Hi all,
>>
>> I am currently using dataset parallelization, like so:
>>
>> ts = yt.load("/path/mydata_hdf5_plt*")
>> results = {}
>> for sto, ds in ts.piter(storage=results):
>>    # processing on each dataset using:
>>    # cut regions
>>    # center of mass calculations
>>    # weighted averages
>>    # projections
>>
>
> It's possible that any one of these operations is causing your RAM to blow
> up. It might even be a bug in yt that's triggering a memory leak.
>
> Is there any chance you can narrow it down to a single operation that's
> causing the memory to blow up?
>
>
>>
>> if yt.is_root():
>>     # process aggregate results
>>
>> I'll try Britton's suggestion of specifying the number of processors to
>> use as a subset of the total available.
>>
>
> Yes that would probably help. Right now I suspect that you have more than
> one core on the same compute node simultaneously processing many output
> files. You probably only want one or a couple outputs per compute node.
>
>
>>
>> Also, it may or may not be relevant, but the cluster I'm using does not
>> support MPICH2 (installed by conda as a dependency for mpi4py) due to lack
>> of InfiniBand support. I have removed the mpi4py and mpich2 conda packages
>> and reinstalled with the OpenMPI implementation:
>>
>> conda remove mpi4py
>> conda remove mpich2
>> conda install -c mpi4py mpi4py
>>
>> To check that openmpi is now installed, you can do the following (an
>> asterisk will appear next to any installed packages):
>>
>> conda search -c mpi4py mpi4py
>> conda search -c mpi4py openmpi
>>
>> On my cluster, these commands show that I have the following packages
>> installed:
>>
>> mpi4py
>> *  2.0.0            py27_openmpi_2  mpi4py
>>
>> openmpi
>> *  1.10.2                        1  mpi4py
>>
>>
>> Is it possible that this is causing my RAM consumption problems?
>>
>
> Maybe but I kinda doubt it. If you're running yt on a cluster you probably
> don't want to use mpi4py from conda and instead just build mpi4py from
> source (with "pip install mpi4py") using the native MPI library for that
> cluster. From my experience mpi4py is pretty good about being able to
> detect the proper MPI library at compile time.
>
>
>>
>> Thanks for the help,
>> Jason
>>
>>
>>
>> On Wed, Dec 6, 2017 at 2:45 PM, <yt-users-request at lists.spacepope.org>
>> wrote:
>>
>>> Send yt-users mailing list submissions to
>>>         yt-users at lists.spacepope.org
>>>
>>> To subscribe or unsubscribe via the World Wide Web, visit
>>>         http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>>> or, via email, send a message with subject or body 'help' to
>>>         yt-users-request at lists.spacepope.org
>>>
>>> You can reach the person managing the list at
>>>         yt-users-owner at lists.spacepope.org
>>>
>>> When replying, please edit your Subject line so it is more specific
>>> than "Re: Contents of yt-users digest..."
>>>
>>> Today's Topics:
>>>
>>>    1. Parallelism in yt Applied to Large Datasets (Jason Galyardt)
>>>    2. Re: Parallelism in yt Applied to Large Datasets (Nathan Goldbaum)
>>>    3. Re: Parallelism in yt Applied to Large Datasets (Scott Feister)
>>>    4. Re: Parallelism in yt Applied to Large Datasets (Britton Smith)
>>>
>>>
>>> ---------- Forwarded message ----------
>>> From: Jason Galyardt <jason.galyardt at gmail.com>
>>> To: Discussion of the yt analysis package <yt-users at lists.spacepope.org>
>>> Cc:
>>> Bcc:
>>> Date: Wed, 6 Dec 2017 09:08:29 -0500
>>> Subject: [yt-users] Parallelism in yt Applied to Large Datasets
>>> Hi yt Folks,
>>>
>>> I've written a script that uses a yt DatasetSeries object to analyze a
>>> time series dataset generated by FLASH. It worked beautifully, until I
>>> tried to run it on a new cluster with significantly larger HDF5 files (4 GB
>>> to greater than 8 GB per file). Now, while running the script, the RAM
>>> usage just grows and grows until the OS kills the job.
>>>
>>> It seems to me that I need to use domain decomposition to process these
>>> large files. So, my question to the group is this: is it possible to use
>>> both domain decomposition *and* parallel time series processing in a single
>>> script? This would require that yt be able to subdivide the available MPI
>>> processors into a number of work groups, each work group handling a single
>>> input file.
>>>
>>> Cheers,
>>> Jason
>>>
>>> ------
>>> Jason Galyardt
>>> University of Georgia
>>>
>>>
>>>
>>> ---------- Forwarded message ----------
>>> From: Nathan Goldbaum <nathan12343 at gmail.com>
>>> To: Discussion of the yt analysis package <yt-users at lists.spacepope.org>
>>> Cc:
>>> Bcc:
>>> Date: Wed, 06 Dec 2017 14:25:01 +0000
>>> Subject: Re: [yt-users] Parallelism in yt Applied to Large Datasets
>>> That depends on what sort of analysis you are doing. Not all tasks in yt
>>> are parallel-aware.
>>>
>>> On Wed, Dec 6, 2017 at 8:08 AM Jason Galyardt <jason.galyardt at gmail.com>
>>> wrote:
>>>
>>>> Hi yt Folks,
>>>>
>>>> I've written a script that uses a yt DatasetSeries object to analyze a
>>>> time series dataset generated by FLASH. It worked beautifully, until I
>>>> tried to run it on a new cluster with significantly larger HDF5 files (4 GB
>>>> to greater than 8 GB per file). Now, while running the script, the RAM
>>>> usage just grows and grows until the OS kills the job.
>>>>
>>>> It seems to me that I need to use domain decomposition to process these
>>>> large files. So, my question to the group is this: is it possible to use
>>>> both domain decomposition *and* parallel time series processing in a single
>>>> script? This would require that yt be able to subdivide the available MPI
>>>> processors into a number of work groups, each work group handling a single
>>>> input file.
>>>>
>>>> Cheers,
>>>> Jason
>>>>
>>>> ------
>>>> Jason Galyardt
>>>> University of Georgia
>>>>
>>>> _______________________________________________
>>>> yt-users mailing list
>>>> yt-users at lists.spacepope.org
>>>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>>>>
>>>
>>>
>>> ---------- Forwarded message ----------
>>> From: Scott Feister <sfeister at gmail.com>
>>> To: Discussion of the yt analysis package <yt-users at lists.spacepope.org>
>>> Cc:
>>> Bcc:
>>> Date: Wed, 6 Dec 2017 11:30:18 -0800
>>> Subject: Re: [yt-users] Parallelism in yt Applied to Large Datasets
>>> Hi Jason,
>>>
>>> I don't know how to do both domain and time decomposition in yt, but I
>>> have been doing time-series analysis in yt of some fairly massive FLASH
>>> HDF5 outputs (~20 GB each) without a problem. If you'd like to share the
>>> script with me (you can send to feister at flash.uchicago.edu), I can take
>>> a look and see if I notice anything particularly wasting RAM. Maybe there's
>>> a simpler solution than resorting to domain decomposition!
>>>
>>> Best,
>>>
>>> Scott
>>>
>>>
>>> Scott Feister, Ph.D.
>>> Postdoctoral Researcher, Flash Center for Computational Science
>>> University of Chicago, Department of Astronomy and Astrophysics
>>>
>>> On Wed, Dec 6, 2017 at 6:25 AM, Nathan Goldbaum <nathan12343 at gmail.com>
>>> wrote:
>>>
>>>> That depends on what sort of analysis you are doing. Not all tasks in
>>>> yt are parallel-aware.
>>>>
>>>> On Wed, Dec 6, 2017 at 8:08 AM Jason Galyardt <jason.galyardt at gmail.com>
>>>> wrote:
>>>>
>>>>> Hi yt Folks,
>>>>>
>>>>> I've written a script that uses a yt DatasetSeries object to analyze a
>>>>> time series dataset generated by FLASH. It worked beautifully, until I
>>>>> tried to run it on a new cluster with significantly larger HDF5 files (4 GB
>>>>> to greater than 8 GB per file). Now, while running the script, the RAM
>>>>> usage just grows and grows until the OS kills the job.
>>>>>
>>>>> It seems to me that I need to use domain decomposition to process
>>>>> these large files. So, my question to the group is this: is it possible to
>>>>> use both domain decomposition *and* parallel time series processing in a
>>>>> single script? This would require that yt be able to subdivide the
>>>>> available MPI processors into a number of work groups, each work group
>>>>> handling a single input file.
>>>>>
>>>>> Cheers,
>>>>> Jason
>>>>>
>>>>> ------
>>>>> Jason Galyardt
>>>>> University of Georgia
>>>>>
>>>>> _______________________________________________
>>>>> yt-users mailing list
>>>>> yt-users at lists.spacepope.org
>>>>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>>>>>
>>>>
>>>> _______________________________________________
>>>> yt-users mailing list
>>>> yt-users at lists.spacepope.org
>>>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>>>>
>>>>
>>>
>>>
>>> ---------- Forwarded message ----------
>>> From: Britton Smith <brittonsmith at gmail.com>
>>> To: Discussion of the yt analysis package <yt-users at lists.spacepope.org>
>>> Cc:
>>> Bcc:
>>> Date: Wed, 6 Dec 2017 11:45:10 -0800
>>> Subject: Re: [yt-users] Parallelism in yt Applied to Large Datasets
>>> Hi Scott,
>>>
>>> yt can do the multi-level parallelism you're talking about, i.e.,
>>> parallelism over multiple datasets and in the operations on a single
>>> dataset.  I would start by looking here:
>>> http://yt-project.org/docs/dev/analyzing/parallel_computatio
>>> n.html#parallelization-over-multiple-objects-and-datasets
>>>
>>> Namely, have a look at the user of "piter" when looping over the
>>> DatasetSeries.  With that function, you can specify the number of jobs (the
>>> njobs keyword) to be a number less than the total number of processors you
>>> have available.  This will give you work groups with multiple processors
>>> for each dataset.  Then, as long as the operations you're trying to do have
>>> been parallelized, things will just work, i.e., that operation will employ
>>> all the cores of that work group.
>>>
>>> If you need to do some custom parallelization at the dataset level, I
>>> also suggest having a look at the parallel_objects command:
>>> http://yt-project.org/docs/dev/analyzing/parallel_computatio
>>> n.html#parallelizing-over-multiple-objects
>>>
>>> This has a similar structure to piter, only it is a more general looping
>>> construct that allows you to break the iterations of the loops into
>>> separate processors or workgroups.  parallel_objects is also nestable, so
>>> you can have nested loops that continually break things down further.
>>>
>>> I hope this helps.  Please, feel free to come back if you have more
>>> specific questions on parallelizing your analysis.
>>>
>>> Britton
>>>
>>>
>>>
>>> On Wed, Dec 6, 2017 at 11:30 AM, Scott Feister <sfeister at gmail.com>
>>> wrote:
>>>
>>>> Hi Jason,
>>>>
>>>> I don't know how to do both domain and time decomposition in yt, but I
>>>> have been doing time-series analysis in yt of some fairly massive FLASH
>>>> HDF5 outputs (~20 GB each) without a problem. If you'd like to share the
>>>> script with me (you can send to feister at flash.uchicago.edu), I can
>>>> take a look and see if I notice anything particularly wasting RAM. Maybe
>>>> there's a simpler solution than resorting to domain decomposition!
>>>>
>>>> Best,
>>>>
>>>> Scott
>>>>
>>>>
>>>> Scott Feister, Ph.D.
>>>> Postdoctoral Researcher, Flash Center for Computational Science
>>>> University of Chicago, Department of Astronomy and Astrophysics
>>>>
>>>> On Wed, Dec 6, 2017 at 6:25 AM, Nathan Goldbaum <nathan12343 at gmail.com>
>>>> wrote:
>>>>
>>>>> That depends on what sort of analysis you are doing. Not all tasks in
>>>>> yt are parallel-aware.
>>>>>
>>>>> On Wed, Dec 6, 2017 at 8:08 AM Jason Galyardt <
>>>>> jason.galyardt at gmail.com> wrote:
>>>>>
>>>>>> Hi yt Folks,
>>>>>>
>>>>>> I've written a script that uses a yt DatasetSeries object to analyze
>>>>>> a time series dataset generated by FLASH. It worked beautifully, until I
>>>>>> tried to run it on a new cluster with significantly larger HDF5 files (4 GB
>>>>>> to greater than 8 GB per file). Now, while running the script, the RAM
>>>>>> usage just grows and grows until the OS kills the job.
>>>>>>
>>>>>> It seems to me that I need to use domain decomposition to process
>>>>>> these large files. So, my question to the group is this: is it possible to
>>>>>> use both domain decomposition *and* parallel time series processing in a
>>>>>> single script? This would require that yt be able to subdivide the
>>>>>> available MPI processors into a number of work groups, each work group
>>>>>> handling a single input file.
>>>>>>
>>>>>> Cheers,
>>>>>> Jason
>>>>>>
>>>>>> ------
>>>>>> Jason Galyardt
>>>>>> University of Georgia
>>>>>>
>>>>>> _______________________________________________
>>>>>> yt-users mailing list
>>>>>> yt-users at lists.spacepope.org
>>>>>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> yt-users mailing list
>>>>> yt-users at lists.spacepope.org
>>>>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>>>>>
>>>>>
>>>>
>>>> _______________________________________________
>>>> yt-users mailing list
>>>> yt-users at lists.spacepope.org
>>>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>>>>
>>>>
>>>
>>> _______________________________________________
>>> yt-users mailing list
>>> yt-users at lists.spacepope.org
>>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>>>
>>>
>>
>> _______________________________________________
>> yt-users mailing list
>> yt-users at lists.spacepope.org
>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>>
>>
>
> _______________________________________________
> yt-users mailing list
> yt-users at lists.spacepope.org
> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.spacepope.org/pipermail/yt-users-spacepope.org/attachments/20171206/1e1e6e80/attachment-0002.html>


More information about the yt-users mailing list