[yt-users] SFR analysis on 3200 cube data

Stephen Skory s at skory.us
Fri Jun 24 15:26:11 PDT 2011


Hi Geoffrey,

> I was thinking, when we load up a portion of an array say
> a=dd["creation_time"][0:99], are we only using the first 100 creation_time
> in memory or does it load up the whole array, then pick out the first 100?
>  If not we can just loop over several times (next one will be [100:199])
> until it gets to the end of the array.  Superslow, but it'll fit in
> memory...

You're pretty much correct in your suspicions that indexing the array
like [0:99] will load the whole array into memory and then cut it
down. However, I think that there might be something else going on
because what you ran should have worked. I think that even though you
are looping over individual grids, perhaps not all the objects are
being thrown away, and you're still running out of memory. Can you try
this script below where I've made a couple changes? I've added a call
to .clear_data() for each grid, and then a call to the garbage
collector for good measure. Let me know how it goes!

print "starting imports"
from yt.mods import *
from yt.analysis_modules.star_analysis.api import *
import gc
print "loaded modules"
pf = load("RD0017/RD0017")
print "loaded datafile"
dd = pf.h.all_data()
print "loaded all data"
sm = []
ct = []
for grid in pf.h.grids:
   print grid
   this_ct = grid['creation_time']
   this_sm = grid['ParticleMassMsun']
   select = (this_ct > 0)
   ext_ct = this_ct[na.where(select)]
   print na.size(ext_ct)
   ct.extend(ext_ct)
   sm.extend(this_sm[na.where(select)])
   grid.clear_data()
   gc.collect()


sfr = StarFormationRate(pf, star_mass=na.array(sm),
star_creation_time=na.array(ct),
volume=dd.volume('mpc'))

sfr.write_out(name="StarFormationRate.out")




-- 
Stephen Skory
s at skory.us
http://stephenskory.com/
510.621.3687 (google voice)



More information about the yt-users mailing list