I think that's good enough for me, then. Should we wait until the fix you provided is accepted into the repo for this or just go ahead?<br><br>Britton<br><br><div class="gmail_quote">On Mon, Jan 16, 2012 at 1:33 PM, Matthew Turk <span dir="ltr"><<a href="mailto:matthewturk@gmail.com">matthewturk@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi Britton,<br>
<br>
As side effect of yt.pmods being imported is that you'll get the<br>
context manager, so you can then still do:<br>
<br>
from yt.pmods import *<br>
with mpi_import():<br>
from my_analysis import *<br>
<br>
-Matt<br>
<div class="HOEnZb"><div class="h5"><br>
On Mon, Jan 16, 2012 at 1:31 PM, Britton Smith <<a href="mailto:brittonsmith@gmail.com">brittonsmith@gmail.com</a>> wrote:<br>
> Matt is right about the perils of putting any mpi imports in a try block.<br>
> Systems like Ranger will fail in a way that is not catchable by python when<br>
> trying to import mpi4py and not running in parallel. I think the yt.pmods<br>
> solution is the best for now. Since the imports problem really only gets<br>
> serious for more than about 100 cores, I think it's ok to impose some<br>
> additional requirements of understanding on the user if they're going to run<br>
> jobs that large.<br>
> Would it be possible to add some sort of helper function such that you could<br>
> do something like the following in a script:<br>
> from yt.pmods import *<br>
> parallel_import("from my_analysis import *")<br>
><br>
> That be helpful.<br>
> Britton<br>
><br>
><br>
> On Mon, Jan 16, 2012 at 1:20 PM, Matthew Turk <<a href="mailto:matthewturk@gmail.com">matthewturk@gmail.com</a>> wrote:<br>
>><br>
>> Hi all,<br>
>><br>
>> Okay, seems like there is some confusion. My response was in<br>
>> reference to Britton's question, which I thought was "are there any<br>
>> sideeffects of [your fix for recursive imports] on the operation [of<br>
>> the import hack for MPI]?" There are not.<br>
>><br>
>> My original statement, which Stephen disagrees with, is that we should<br>
>> require an explicit change on the part of the user before we (on their<br>
>> behalf) fundamentally modify the way the base functionality of<br>
>> 'import' works for all Python modules. I have several motivations for<br>
>> this:<br>
>><br>
>> * The import problem is generic for shared filesystems being accessed<br>
>> in parallel, but is only crippling at relatively large core counts on<br>
>> particularly large lustre systems, compared to what most users<br>
>> utilize. This is a -1 for global application of the MPI_Import fix.<br>
>> * The change of yt.mods to yt.pmods is not an invasive change,<br>
>> although I too do not like having different behavior for running in<br>
>> parallel. However, we do expect a number of things from users that<br>
>> run in parallel: an understanding of the resources they are to<br>
>> allocate, a set up of the queue script, and a recognition of which<br>
>> activities will parallelize and which will not. I still think it is<br>
>> not the best solution, but I believe it is non-invasive.<br>
>> * Every time we add on an additional non-sanitized import, we take a<br>
>> big performance hit. It is in our best interest for this to all occur<br>
>> at the outermost level.<br>
>> * Detecting whether we are running in parallel is not trivial.<br>
>> * On some machines, specifically SGI, if you run a script in parallel<br>
>> that contains a try/except block for importing MPI and it was not<br>
>> launched with MPI, it will die unceremoniously. We cannot rely on a<br>
>> try/except of importing MPI.<br>
>><br>
>> All these things combined lead me to believe that I think we should<br>
>> not attempt to guess *for* the user<br>
>><br>
>> My proposed change, of adding yt.pmods, would consist of a new file<br>
>> (yt/pmods.py) that contains the full contents of MPI_Import.py and<br>
>> that, at the end, performs this operation:<br>
>><br>
>> with mpi_import():<br>
>> from yt.mods import *<br>
>><br>
>> What this would result in is a nearly self-contained script that<br>
>> returned to the user the contents of yt.mods; there'd be no<br>
>> duplication. An alternate solution, which I am not terribly keen on,<br>
>> would be to put manual context startup/shutdown inside yt.mods if<br>
>> startup_tasks.parallel_enabled is true. I feel like this would result<br>
>> in a lot of unnecessary side effects. However, with yt.pmods, while<br>
>> the yt imports would all be included, any additional imports would<br>
>> still need sanitation inside the users' script.<br>
>><br>
>> The fallback would be to require users to use the with: statement<br>
>> themselves.<br>
>><br>
>> -Matt<br>
>><br>
>> On Mon, Jan 16, 2012 at 11:08 AM, j s oishi <<a href="mailto:jsoishi@gmail.com">jsoishi@gmail.com</a>> wrote:<br>
>> >> I am of the opinion we can do this with an alternate, parallel import<br>
>> >> that<br>
>> >> would be compatible with yt.mods. Something like yt.pmods. What do you<br>
>> >> think? What should usage of this be?<br>
>> ><br>
>> > Not sure I understand this. How is yt.pmods compatible with yt.mods?<br>
>> > Do you mean both would have the same effect, but yt.pmods would use<br>
>> > the new mechanism for loading rather than the current standard one<br>
>> > which would be retained by yt.mods? If so, that sounds like an OK idea<br>
>> > to me, though if there is no side effect, it could lead to people<br>
>> > forgetting to sub in pmods, and then being stuck with bad, old<br>
>> > performance. Given that this will be most important where even a<br>
>> > single forgetful job could cost substantial allocation usage, maybe we<br>
>> > should think about making it automatic?<br>
>> ><br>
>> > Or perhaps I am misunderstanding...<br>
>> > _______________________________________________<br>
>> > yt-dev mailing list<br>
>> > <a href="mailto:yt-dev@lists.spacepope.org">yt-dev@lists.spacepope.org</a><br>
>> > <a href="http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org" target="_blank">http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org</a><br>
>> _______________________________________________<br>
>> yt-dev mailing list<br>
>> <a href="mailto:yt-dev@lists.spacepope.org">yt-dev@lists.spacepope.org</a><br>
>> <a href="http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org" target="_blank">http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org</a><br>
><br>
><br>
><br>
> _______________________________________________<br>
> yt-dev mailing list<br>
> <a href="mailto:yt-dev@lists.spacepope.org">yt-dev@lists.spacepope.org</a><br>
> <a href="http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org" target="_blank">http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org</a><br>
><br>
_______________________________________________<br>
yt-dev mailing list<br>
<a href="mailto:yt-dev@lists.spacepope.org">yt-dev@lists.spacepope.org</a><br>
<a href="http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org" target="_blank">http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org</a><br>
</div></div></blockquote></div><br>