[yt-users] Parallelizing a bunch of projections

[Matthew Turk]

On Thu, Dec 4, 2014 at 4:59 PM, Nathan Goldbaum <nathan12343 at gmail.com> wrote:
>
>
> On Thu Dec 04 2014 at 2:54:52 PM John ZuHone <jzuhone at gmail.com> wrote:
>>
>> Hi all,
>>
>> I know that parallelizing a single projection doesn’t really give you that
>> much speed benefit.
>>
>> But what would happen if I tried to do something like this (please excuse
>> the Python pseudocode):
>>
>> fields = [big list of fields I want to project]
>>
>> prj = ds.proj(“ones”, 2)
>> frb = prj.to_frb(…)
>>
>> storage = {}
>> for sto, fd in parallel_objects(fields, storage=storage):
>>         buf = frb[fd]
>>
>>         sto.result_id = fd
>>         sto.result = buf
>
>
> What happens if you do something like:
>
> storage = {}
> for sto, fd in parallel_objects(fields, storage=storage):
>         prj = ds.proj(fd, 2)
>         frb = prj.to_frb(…)
>         buf = frb[fd]
>
>         sto.result_id = fd
>         sto.result = buf
>
> ?
>
>>
>> I did something like this but it hung at the end, which makes me think
>> that I’ve got some kind of blocking operation here that I’m not aware of.

There's also a barrier option to parallel_objects, which defaults to True.

>>
>> Best,
>>
>> John
>> _______________________________________________
>> yt-users mailing list
>> yt-users at lists.spacepope.org
>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>
>
> _______________________________________________
> yt-users mailing list
> yt-users at lists.spacepope.org
> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>
_______________________________________________
yt-users mailing list
yt-users at lists.spacepope.org
http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org