[yt-users] Parallelizing a bunch of projections

John ZuHone jzuhone at gmail.com
Thu Dec 4 15:04:37 PST 2014


Hi Nathan,

I actually was doing that before, and it did work. 

I was hoping that by creating just one projection and then querying the frb for fields that I would get a speed boost by not having to construct the projection object over and over again. Or is that not the case?

John

> On Dec 4, 2014, at 5:59 PM, Nathan Goldbaum <nathan12343 at gmail.com> wrote:
> 
> 
> 
> On Thu Dec 04 2014 at 2:54:52 PM John ZuHone <jzuhone at gmail.com <mailto:jzuhone at gmail.com>> wrote:
> Hi all,
> 
> I know that parallelizing a single projection doesn’t really give you that much speed benefit.
> 
> But what would happen if I tried to do something like this (please excuse the Python pseudocode):
> 
> fields = [big list of fields I want to project]
> 
> prj = ds.proj(“ones”, 2)
> frb = prj.to_frb(…)
> 
> storage = {}
> for sto, fd in parallel_objects(fields, storage=storage):
>         buf = frb[fd]
> 
>         sto.result_id = fd
>         sto.result = buf
> 
> What happens if you do something like:
> 
> storage = {}
> for sto, fd in parallel_objects(fields, storage=storage):
>         prj = ds.proj(fd, 2)
>         frb = prj.to_frb(…)
>         buf = frb[fd]
> 
>         sto.result_id = fd
>         sto.result = buf
>  
> ?
> 
> 
> I did something like this but it hung at the end, which makes me think that I’ve got some kind of blocking operation here that I’m not aware of.
> 
> Best,
> 
> John
> _______________________________________________
> yt-users mailing list
> yt-users at lists.spacepope.org <mailto:yt-users at lists.spacepope.org>
> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org <http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org>
> _______________________________________________
> yt-users mailing list
> yt-users at lists.spacepope.org <mailto:yt-users at lists.spacepope.org>
> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org <http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.spacepope.org/pipermail/yt-users-spacepope.org/attachments/20141204/3fbae598/attachment-0001.html>
-------------- next part --------------
_______________________________________________
yt-users mailing list
yt-users at lists.spacepope.org
http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org


More information about the yt-users mailing list