[yt-dev] Reducing memory usage in time series
John Wise
jwise at physics.gatech.edu
Tue Feb 4 12:48:32 PST 2014
Hi Matt,
On 02/04/2014 01:44 PM, Matthew Turk wrote:
> Hi John,
>
> On Tue, Feb 4, 2014 at 1:21 PM, John Wise <jwise at physics.gatech.edu> wrote:
>> Hi all,
>>
>> I've been trying to run rockstar in a sizable Enzo simulation (150k grids)
>> with ~100 outputs, where it's running out of memory just loading the
>> hierarchies. One hierarchy instance consumes almost 1GB! I've found this
>> to be problem not specific to rockstar but time series objects.
>
> Hm. How are you iterating over the parameter files? With the time
> series we try to do a load/retain on demand system, where the
> parameter files and their hierarchies are only kept around as long as
> they need to be. Devin and Hilary looked at how this worked with
> Rockstar, and I thought they came up with that it was okay.
I create the time series with a list of pfs because I want to have them
sorted by time. I do this because I have both time-based and
redshift-based outputs. Also I've removed my call to the hierarchy
destructor in the static_output destructor because it wasn't having any
effect.
Then I iterate like the following...
ts = TimeSeriesData(pfs)
for pf in ts:
print "%20s (0): %d" % (pf, get_memory_usage())
pf.h
print "%20s (1): %d" % (pf, get_memory_usage())
del pf
print "%20s (2): %d" % ("", get_memory_usage())
which gives
output_0048 (0): 74
output_0048 (1): 971
(2): 972
output_0049 (0): 972
output_0049 (1): 1705
(2): 1705
output_0050 (0): 1705
output_0050 (1): 2434
(2): 2434
output_0051 (0): 2434
output_0051 (1): 3169
(2): 3170
Is there a better way to iterate to through the pfs? Removing the "del
pf" shows the same memory usage, fwiw.
However if I replace "del pf" with "del pf._instantiated_hierarchy"
using my destructor, I see nearly stable memory usage
output_0048 (0): 74
output_0048 (1): 971
(2): 963
output_0049 (0): 963
output_0049 (1): 978
(2): 978
output_0050 (0): 978
output_0050 (1): 978
(2): 978
output_0051 (0): 978
output_0051 (1): 982
(2): 983
I see stable behavior in rockstar if I add a "del
pf._instantiated_hierarchy" at the end of rh_read_particles().
>>
>> My solution is to explicitly delete the hierarchy's metadata and grids.
>> Since I haven't contributed to yt-3.0 yet, I wanted to run this by everyone
>> before submitting a PR.
>>
>> My question is about coding style, in that I see very few __del__()
>> functions now. In my working version, I've defined a __del__ function for
>> the grid_geometry_handler as
>>
>> def __del__(self):
>> del self.grid_dimensions
>> del self.grid_left_edge
>> del self.grid_right_edge
>> del self.grid_levels
>> del self.grid_particle_count
>> del self.grids
>>
>> When I delete pf._instantiated_hierarchy after each loop of a time series
>> iterator, I don't see any excessive memory usage anymore. It just reuses
>> the allocated memory from the previous iteration, which is totally fine by
>> me. However, when I include this in a __del__ function for a static_output,
>> I still see excessive memory usage, which is bizarre to me.
>
> Hmm.
>
> I'm of two minds on this. On the one hand, I am not really *opposed*
> to destructors, but I don't like that they are necessary. Because the
> hierarchy is weirdly self-referential to the static output, this
> sometimes causes problems and the garbage collector doesn't pick it
> up. However, when the parameter file is deallocated, it *should*
> deallocate all of the arrays. Whether it does or not may be related
> to the system allocator, and whether it reuses the memory is
> potentially also related to that. On the other hand, I'd rather fix
> the issue of having a separate index and static output object, and
> break the reference cycle between them.
I agree that having destructors are not the cleanest way to maintain
memory usage because it should be taken care of when the parameter file
is deallocated. I've tested this on a few systems (Mac OSX, my desktop
with the v3.11 kernel, and the local cluster with the v2.6.32 kernel),
and the system allocator behaves similarly.
> So I guess where I fall down on this is: I'd like to fix the
> underlying issue, which is something I have been off-and-on working
> on. But since you are measurably seeing improvement with this change,
> I'm okay with it going in. But hopefully it will become obsolete
> eventually. ;-)
I'm totally fine with this change being a temporary fix while the
underlying problem is being solved.
> Incidentally, I would still like to see how the time series is
> iterating, and how the references pass through the system.
>
> A related paper you might find interesting:
> http://www.dlr.de/sc/en/Portaldata/15/Resources/dokumente/PyHPC2013/submissions/pyhpc2013_submission_6.pdf
Looks like a good read :) Thanks for your input!
Cheers,
John
>>
>> Should I define a new routine in the grid_geometry_handler, something like
>> clear_hierarchy(), or keep the __del__ function? I ask because I want to
>> keep in line with the overall structure of yt-3.0. This could also be
>> included in the clear_all_data() call.
>>
>> What do people think the best approach would be?
--
John Wise
Assistant Professor of Physics
Center for Relativistic Astrophysics, Georgia Tech
http://cosmo.gatech.edu
More information about the yt-dev
mailing list