Matt is right about the perils of putting any mpi imports in a try block. Systems like Ranger will fail in a way that is not catchable by python when trying to import mpi4py and not running in parallel. I think the yt.pmods solution is the best for now. Since the imports problem really only gets serious for more than about 100 cores, I think it's ok to impose some additional requirements of understanding on the user if they're going to run jobs that large.<br>
Would it be possible to add some sort of helper function such that you could do something like the following in a script:<br>from yt.pmods import *<br>parallel_import("from my_analysis import *")<br><br>That be helpful.<br>
Britton<br><br><div class="gmail_quote">On Mon, Jan 16, 2012 at 1:20 PM, Matthew Turk <span dir="ltr"><<a href="mailto:matthewturk@gmail.com">matthewturk@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Hi all,<br>
<br>
Okay, seems like there is some confusion. My response was in<br>
reference to Britton's question, which I thought was "are there any<br>
sideeffects of [your fix for recursive imports] on the operation [of<br>
the import hack for MPI]?" There are not.<br>
<br>
My original statement, which Stephen disagrees with, is that we should<br>
require an explicit change on the part of the user before we (on their<br>
behalf) fundamentally modify the way the base functionality of<br>
'import' works for all Python modules. I have several motivations for<br>
this:<br>
<br>
* The import problem is generic for shared filesystems being accessed<br>
in parallel, but is only crippling at relatively large core counts on<br>
particularly large lustre systems, compared to what most users<br>
utilize. This is a -1 for global application of the MPI_Import fix.<br>
* The change of yt.mods to yt.pmods is not an invasive change,<br>
although I too do not like having different behavior for running in<br>
parallel. However, we do expect a number of things from users that<br>
run in parallel: an understanding of the resources they are to<br>
allocate, a set up of the queue script, and a recognition of which<br>
activities will parallelize and which will not. I still think it is<br>
not the best solution, but I believe it is non-invasive.<br>
* Every time we add on an additional non-sanitized import, we take a<br>
big performance hit. It is in our best interest for this to all occur<br>
at the outermost level.<br>
* Detecting whether we are running in parallel is not trivial.<br>
* On some machines, specifically SGI, if you run a script in parallel<br>
that contains a try/except block for importing MPI and it was not<br>
launched with MPI, it will die unceremoniously. We cannot rely on a<br>
try/except of importing MPI.<br>
<br>
All these things combined lead me to believe that I think we should<br>
not attempt to guess *for* the user<br>
<br>
My proposed change, of adding yt.pmods, would consist of a new file<br>
(yt/pmods.py) that contains the full contents of MPI_Import.py and<br>
that, at the end, performs this operation:<br>
<div class="im"><br>
with mpi_import():<br>
from yt.mods import *<br>
<br>
</div>What this would result in is a nearly self-contained script that<br>
returned to the user the contents of yt.mods; there'd be no<br>
duplication. An alternate solution, which I am not terribly keen on,<br>
would be to put manual context startup/shutdown inside yt.mods if<br>
startup_tasks.parallel_enabled is true. I feel like this would result<br>
in a lot of unnecessary side effects. However, with yt.pmods, while<br>
the yt imports would all be included, any additional imports would<br>
still need sanitation inside the users' script.<br>
<br>
The fallback would be to require users to use the with: statement themselves.<br>
<br>
-Matt<br>
<div class="HOEnZb"><div class="h5"><br>
On Mon, Jan 16, 2012 at 11:08 AM, j s oishi <<a href="mailto:jsoishi@gmail.com">jsoishi@gmail.com</a>> wrote:<br>
>> I am of the opinion we can do this with an alternate, parallel import that<br>
>> would be compatible with yt.mods. Something like yt.pmods. What do you<br>
>> think? What should usage of this be?<br>
><br>
> Not sure I understand this. How is yt.pmods compatible with yt.mods?<br>
> Do you mean both would have the same effect, but yt.pmods would use<br>
> the new mechanism for loading rather than the current standard one<br>
> which would be retained by yt.mods? If so, that sounds like an OK idea<br>
> to me, though if there is no side effect, it could lead to people<br>
> forgetting to sub in pmods, and then being stuck with bad, old<br>
> performance. Given that this will be most important where even a<br>
> single forgetful job could cost substantial allocation usage, maybe we<br>
> should think about making it automatic?<br>
><br>
> Or perhaps I am misunderstanding...<br>
> _______________________________________________<br>
> yt-dev mailing list<br>
> <a href="mailto:yt-dev@lists.spacepope.org">yt-dev@lists.spacepope.org</a><br>
> <a href="http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org" target="_blank">http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org</a><br>
_______________________________________________<br>
yt-dev mailing list<br>
<a href="mailto:yt-dev@lists.spacepope.org">yt-dev@lists.spacepope.org</a><br>
<a href="http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org" target="_blank">http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org</a><br>
</div></div></blockquote></div><br>