[yt-dev] yt-3.0 parallel_capable

Tue Apr 22 15:06:55 PDT 2014

Yup, I came to a similar conclusion:
https://bitbucket.org/yt_analysis/yt/issue/833/parallel_capable-getting-reset-to-false#comment-9785096

I agree that being careful with imports is probably the best solution.

On Tue, Apr 22, 2014 at 3:04 PM, Josh Moloney
<Joshua.Moloney at colorado.edu>wrote:

> I think I've figured out what's going on here, but I don't have much
> experience with decorators in python so I'm not 100% sure. My guess is that
> parallel_root_only(func) is evaluated as soon as the @parallel_root_only
> decorator is encountered in each module as it's imported. At this point
> enable_parallelism hasn't been called yet, so parallel_capable is still
> False (it's initial assigned value) and the original function gets returned
> to all the processors (instead of None to everything except root). This
> would explain why Britton got the expected parallel behavior when he
> switched it so parallel_capable was initially assigned True instead
> (although this might create problems if it isn't actually being run in
> parallel).
> I tried enclosing a call to enable_parallelism inside parallel_root_only
> right before the "if parallel_capable" and this makes it behave as expected
> on both mpi and single processor runs. Unfortunately it means
> enable_parallelism gets called multiple times on each processor, which
> again probably isn't ideal behavior.
> I think the best way to solve this is to make sure enable_parallelism gets
> called before other imports are done, but I'm not sure how feasible this
> is. Someone who knows more about how decorators work can probably make a
> better suggestion.
>        - Josh
>
>
> On Tue, Apr 22, 2014 at 2:14 PM, Britton Smith <brittonsmith at gmail.com>wrote:
>
>> Hi Nathan,
>>
>> I tried your suggestion, but it did not change anything.  I'll keep
>> looking around, but will file an issue for now.
>>
>> Britton
>>
>>
>> On Tue, Apr 22, 2014 at 8:57 PM, Nathan Goldbaum <nathan12343 at gmail.com>wrote:
>>
>>> Hi Britton,
>>>
>>> It looks like parallel_capable on line 43 of
>>> parallel_analysis_interface.py *wants* to be a global mutable variable that
>>> is used elsewhere in yt, but it is not. Perhaps at one point it was
>>> imported into yt.funcs?
>>>
>>> I think you could fix this by explicitly importing parallel_cabable from
>>> parallel_analysis_interface.py into startup_tasks.py.
>>>
>>> Hope that helps,
>>>
>>> Nathan
>>>
>>>
>>> On Tue, Apr 22, 2014 at 12:42 PM, Britton Smith <brittonsmith at gmail.com>wrote:
>>>
>>>> Hi again,
>>>>
>>>> I don't know how much this helps, but if I change line 43 of
>>>> parallel_analysis_interface.py from parallel_capable = False to
>>>> parallel_capable = True, things work as they are supposed to in parallel.
>>>>  Could there be an import somewhere that is resetting the value of
>>>> parallel_capable?
>>>>
>>>> Britton
>>>>
>>>>
>>>> On Tue, Apr 22, 2014 at 8:28 PM, Britton Smith <brittonsmith at gmail.com>wrote:
>>>>
>>>>> HI Nathan,
>>>>>
>>>>> Adding that call did not change the result.  To be clear, this is what
>>>>> I ran:
>>>>> http://paste.yt-project.org/show/4548/
>>>>>
>>>>> What is strange is that the logger was clearly set up for parallel as
>>>>> it correctly prints the different processor numbers.  Enable parallelism
>>>>> shows that parallel_capable is True, so somewhere along the line this is
>>>>> getting changed.
>>>>>
>>>>> Britton
>>>>>
>>>>>
>>>>> On Tue, Apr 22, 2014 at 8:23 PM, Nathan Goldbaum <
>>>>> nathan12343 at gmail.com> wrote:
>>>>>
>>>>>> Hi Britton,
>>>>>>
>>>>>> What happens when you call yt.enable_parallelism() at the top of the
>>>>>> script?
>>>>>>
>>>>>> There were some adjustments to the way parallelism works related to
>>>>>> YTEP-0019.  See this PR:
>>>>>> https://bitbucket.org/yt_analysis/yt/pull-request/729/ytep-0019/diff
>>>>>>
>>>>>> Do you have a short test script I can try running over here to
>>>>>> confirm the issue?
>>>>>>
>>>>>> -Nathan
>>>>>>
>>>>>>
>>>>>> On Tue, Apr 22, 2014 at 12:20 PM, Britton Smith <
>>>>>> brittonsmith at gmail.com> wrote:
>>>>>>
>>>>>>> Hi all,
>>>>>>>
>>>>>>> I don't recall when I first noticed this, but I'm finding that the
>>>>>>> parallel_root_only decorator doesn't seem to be working.  A simple test is
>>>>>>> to run a script that only loads a dataset.  The print_key_parameters
>>>>>>> function is inside of a parallel_root_only decorator, so all that should
>>>>>>> only be printed by the root processor, but when I run that script in
>>>>>>> parallel it is printed by all processors.  I am still looking into this,
>>>>>>> but so far I have found that printing the value of parallel_capable inside
>>>>>>> this decorator always gives False even when running in parallel.
>>>>>>>
>>>>>>> Can someone confirm this?  If so, does anyone know what's going on?
>>>>>>>
>>>>>>> Britton
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> yt-dev mailing list
>>>>>>> yt-dev at lists.spacepope.org
>>>>>>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>>>>>>>
>>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> yt-dev mailing list
>>>>>> yt-dev at lists.spacepope.org
>>>>>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>>>>>>
>>>>>>
>>>>>
>>>>
>>>> _______________________________________________
>>>> yt-dev mailing list
>>>> yt-dev at lists.spacepope.org
>>>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>>>>
>>>>
>>>
>>> _______________________________________________
>>> yt-dev mailing list
>>> yt-dev at lists.spacepope.org
>>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>>>
>>>
>>
>> _______________________________________________
>> yt-dev mailing list
>> yt-dev at lists.spacepope.org
>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>>
>>
>
> _______________________________________________
> yt-dev mailing list
> yt-dev at lists.spacepope.org
> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.spacepope.org/pipermail/yt-dev-spacepope.org/attachments/20140422/f5413b05/attachment.htm>