[yt-dev] Reducing memory usage in time series

John Wise jwise at physics.gatech.edu
Tue Feb 4 10:21:30 PST 2014


Hi all,

I've been trying to run rockstar in a sizable Enzo simulation (150k 
grids) with ~100 outputs, where it's running out of memory just loading 
the hierarchies.  One hierarchy instance consumes almost 1GB!  I've 
found this to be problem not specific to rockstar but time series objects.

My solution is to explicitly delete the hierarchy's metadata and grids. 
  Since I haven't contributed to yt-3.0 yet, I wanted to run this by 
everyone before submitting a PR.

My question is about coding style, in that I see very few __del__() 
functions now.  In my working version, I've defined a __del__ function 
for the grid_geometry_handler as

     def __del__(self):
         del self.grid_dimensions
         del self.grid_left_edge
         del self.grid_right_edge
         del self.grid_levels
         del self.grid_particle_count
         del self.grids

When I delete pf._instantiated_hierarchy after each loop of a time 
series iterator, I don't see any excessive memory usage anymore.  It 
just reuses the allocated memory from the previous iteration, which is 
totally fine by me.  However, when I include this in a __del__ function 
for a static_output, I still see excessive memory usage, which is 
bizarre to me.

Should I define a new routine in the grid_geometry_handler, something 
like clear_hierarchy(), or keep the __del__ function?  I ask because I 
want to keep in line with the overall structure of yt-3.0.  This could 
also be included in the clear_all_data() call.

What do people think the best approach would be?

Thanks,
John

-- 
John Wise
Assistant Professor of Physics
Center for Relativistic Astrophysics, Georgia Tech
http://cosmo.gatech.edu



More information about the yt-dev mailing list