[yt-users] Parallelizing a bunch of projections

[John ZuHone]

Hi all,

I know that parallelizing a single projection doesn’t really give you that much speed benefit. 

But what would happen if I tried to do something like this (please excuse the Python pseudocode):

fields = [big list of fields I want to project]

prj = ds.proj(“ones”, 2)
frb = prj.to_frb(…)

storage = {}
for sto, fd in parallel_objects(fields, storage=storage):
	buf = frb[fd]

	sto.result_id = fd
	sto.result = buf
	
I did something like this but it hung at the end, which makes me think that I’ve got some kind of blocking operation here that I’m not aware of.

Best,

John