[yt-dev] Reducing memory usage in time series
John Wise
jwise at physics.gatech.edu
Tue Feb 4 10:21:30 PST 2014
Hi all,
I've been trying to run rockstar in a sizable Enzo simulation (150k
grids) with ~100 outputs, where it's running out of memory just loading
the hierarchies. One hierarchy instance consumes almost 1GB! I've
found this to be problem not specific to rockstar but time series objects.
My solution is to explicitly delete the hierarchy's metadata and grids.
Since I haven't contributed to yt-3.0 yet, I wanted to run this by
everyone before submitting a PR.
My question is about coding style, in that I see very few __del__()
functions now. In my working version, I've defined a __del__ function
for the grid_geometry_handler as
def __del__(self):
del self.grid_dimensions
del self.grid_left_edge
del self.grid_right_edge
del self.grid_levels
del self.grid_particle_count
del self.grids
When I delete pf._instantiated_hierarchy after each loop of a time
series iterator, I don't see any excessive memory usage anymore. It
just reuses the allocated memory from the previous iteration, which is
totally fine by me. However, when I include this in a __del__ function
for a static_output, I still see excessive memory usage, which is
bizarre to me.
Should I define a new routine in the grid_geometry_handler, something
like clear_hierarchy(), or keep the __del__ function? I ask because I
want to keep in line with the overall structure of yt-3.0. This could
also be included in the clear_all_data() call.
What do people think the best approach would be?
Thanks,
John
--
John Wise
Assistant Professor of Physics
Center for Relativistic Astrophysics, Georgia Tech
http://cosmo.gatech.edu
More information about the yt-dev
mailing list