[yt-users] Parallelizing a bunch of projections
Nathan Goldbaum
nathan12343 at gmail.com
Thu Dec 4 14:59:04 PST 2014
On Thu Dec 04 2014 at 2:54:52 PM John ZuHone <jzuhone at gmail.com> wrote:
> Hi all,
>
> I know that parallelizing a single projection doesn’t really give you that
> much speed benefit.
>
> But what would happen if I tried to do something like this (please excuse
> the Python pseudocode):
>
> fields = [big list of fields I want to project]
>
> prj = ds.proj(“ones”, 2)
> frb = prj.to_frb(…)
>
> storage = {}
> for sto, fd in parallel_objects(fields, storage=storage):
> buf = frb[fd]
>
> sto.result_id = fd
> sto.result = buf
>
What happens if you do something like:
storage = {}
for sto, fd in parallel_objects(fields, storage=storage):
prj = ds.proj(fd, 2)
frb = prj.to_frb(…)
buf = frb[fd]
sto.result_id = fd
sto.result = buf
?
> I did something like this but it hung at the end, which makes me think
> that I’ve got some kind of blocking operation here that I’m not aware of.
>
> Best,
>
> John
> _______________________________________________
> yt-users mailing list
> yt-users at lists.spacepope.org
> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.spacepope.org/pipermail/yt-users-spacepope.org/attachments/20141204/aba65299/attachment.html>
More information about the yt-users
mailing list