[yt-dev] Fwd: [mpi4py] Fwd: [Numpy-discussion] Improving Python+MPI import performance

[Britton Smith]

Hi Everyone,

I just tested this new import method with Matt's fix on Kraken against the
standard import for a variety of core counts.  In the attached figure, I
plot the mean time to import, with errorbars showing the minimum and
maximum time for all cores.  I think the plot speaks for itself.

Britton

On Sun, Jan 15, 2012 at 10:04 AM, Britton Smith <brittonsmith at gmail.com>wrote:

> Hi Matt,
>
> That addition is working for me on Kraken.  I'm running this test for a
> number of cores and will report soon.
>
> Britton
>
>
> On Sun, Jan 15, 2012 at 6:53 AM, Matthew Turk <matthewturk at gmail.com>wrote:
>
>> Hi again,
>>
>> I've written to the package maintainer, but if you insert this line:
>>
>> while fqname[-1] == ".": fqname = fqname[:-1]
>>
>> at the top of the function definition __import__module__ inside
>> MPI_Import.py, it should work for importing yt.mods.  This script
>> works great for me:
>>
>> from MPI_Import import mpi_import
>> with mpi_import():
>>    from yt.mods import *
>> mylog.info("Hello")
>>
>> I think it's ready to test on Kraken/Janus/etc.
>>
>> -Matt
>>
>> On Fri, Jan 13, 2012 at 2:31 PM, Matthew Turk <matthewturk at gmail.com>
>> wrote:
>> > Hi all,
>> >
>> > I was able to test it and find a problem with mutually-recursive
>> > imports in unittest.  I'll update if I can find a resolution.  For
>> > analysis_modules it should be okay, as if you access them via amods it
>> > should already know the location from which to import them.
>> >
>> > -Matt
>> >
>> > On Fri, Jan 13, 2012 at 2:29 PM, Stephen Skory <s at skory.us> wrote:
>> >> Hi Matt,
>> >>
>> >>> Britton or Stephen, this sounds like it's directly up your alley as
>> >>> you run on Kraken the most often.
>> >>
>> >> I can give, but wouldn't this help on all computers, or am I
>> >> misunderstanding things?
>> >>
>> >>> from yt.pmods import *
>> >>>
>> >>> Ideas?
>> >>
>> >> What about analysis modules that import stuff, too? Like halo finding.
>> >> I guess I have to read more about this, so that's more of a rhetorical
>> >> question.
>> >>
>> >> --
>> >> Stephen Skory
>> >> s at skory.us
>> >> http://stephenskory.com/
>> >> 510.621.3687 (google voice)
>> >> _______________________________________________
>> >> yt-dev mailing list
>> >> yt-dev at lists.spacepope.org
>> >> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>> _______________________________________________
>> yt-dev mailing list
>> yt-dev at lists.spacepope.org
>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.spacepope.org/pipermail/yt-dev-spacepope.org/attachments/20120115/8f32dc63/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: import_scaling.png
Type: image/png
Size: 37234 bytes
Desc: not available
URL: <http://lists.spacepope.org/pipermail/yt-dev-spacepope.org/attachments/20120115/8f32dc63/attachment-0001.png>