[yt-users] Running yt in parallel on a subset of available processes

[Cameron Hummels]

There is also some discussion on how to do it in the developer
documentation:

http://yt-project.org/docs/dev/developing/developing.html#making-and-sharing-changes

Thanks for your hard work on this, and don't hesitate to ask for more help!

Cameron

On Thu, Dec 4, 2014 at 8:05 AM, Matthew Turk <matthewturk at gmail.com> wrote:

> Hi Dan,
>
> Yup, you can commit in that directory, and push to a fork.  Create a
> fork on bitbucket, then do your changes and commit, and issue this
> command:
>
> hg push https://bitbucket.org/yourusername/yt
>
> Then you can issue a pull request via their web interface.  You can
> also edit your hgrc to add an alias, but that can happen later.
>
> -Matt
>
> On Thu, Dec 4, 2014 at 8:08 AM, Daniel Fenn <dsfenn at gmail.com> wrote:
> > Hi Cameron,
> >
> > Yeah, no problem. I'm not exactly sure how to do that. Do I have to fork
> yt
> > first, or can I use my existing installation since it's already using
> > Mercurial? I'm familiar with subversion, since that's what we use in our
> > research group, but I'm a bit hazy on how the whole thing is set up with
> yt.
> > When I installed yt using the installation script, it created a yt-hg
> > directory. Is that a checked-out version of the code that I can commit?
> >
> > Thanks!
> >
> > Dan
> >
> > 2014-12-03 20:59 GMT-05:00 Cameron Hummels <chummels at gmail.com>:
> >
> >> Hey Daniel,
> >>
> >> This is great that you got this working, and it sounds like something
> from
> >> which others could benefit! Would you mind submitting this change to the
> >> main codebase in a pull request?
> >>
> >> Cameron
> >>
> >> On Wed, Dec 3, 2014 at 6:46 PM, Daniel Fenn <dsfenn at gmail.com> wrote:
> >>>
> >>> Hi Matt,
> >>>
> >>> I was able to get my MPI problem fixed by using subcommunicators. I
> >>> modified enable_parallelism() to take an optional argument that
> specifies
> >>> the MPI communicator yt should use. If none is specified, it defaults
> to
> >>> COMM_WORLD. Seems to be working pretty well so far.
> >>>
> >>> Thanks for the advice,
> >>>
> >>> Dan
> >>>
> >>> 2014-12-03 19:16 GMT-05:00 Matthew Turk <matthewturk at gmail.com>:
> >>>
> >>>> Hi Dan,
> >>>>
> >>>> I've done this before by modifying how the global communicator is
> >>>> created.  The way they work in yt is through pushing new communicators
> >>>> onto a stack, which can vary based on which processors you're on.  I
> >>>> don't think I ever wrote it up in a PR, but I think it could be done
> >>>> without too much work.
> >>>>
> >>>> On Wed, Dec 3, 2014 at 11:47 AM, Daniel Fenn <dsfenn at gmail.com>
> wrote:
> >>>> > Is there a way to restrict yt to only use a subset of the available
> >>>> > processes when it's running in parallel? I have a set of ipython
> >>>> > engines
> >>>> > running, and I'm trying to launch a yt script on a subset of them,
> but
> >>>> > it
> >>>> > just hangs. It works fine if I launch the same script on all the
> >>>> > processes.
> >>>> >
> >>>> > Printing out MPI.COMM_WORLD.Get_size() from the script gives the
> total
> >>>> > number of engines, not the size of the subset I'm trying to use.
> >>>> >
> >>>> > I haven't done a lot of parallel programming, but I would guess that
> >>>> > the
> >>>> > running processes are waiting on one that's not running a yt script,
> >>>> > since
> >>>> > they can see the total number of processes available, but not the
> size
> >>>> > of
> >>>> > the subset of processes that are actually running a yt script.
> >>>> >
> >>>> > Does anyone have any thoughts on how to do this?
> >>>> >
> >>>> > Thanks,
> >>>> >
> >>>> > Dan
> >>>> >
> >>>> > _______________________________________________
> >>>> > yt-users mailing list
> >>>> > yt-users at lists.spacepope.org
> >>>> > http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
> >>>> >
> >>>> _______________________________________________
> >>>> yt-users mailing list
> >>>> yt-users at lists.spacepope.org
> >>>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
> >>>
> >>>
> >>>
> >>> _______________________________________________
> >>> yt-users mailing list
> >>> yt-users at lists.spacepope.org
> >>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
> >>>
> >>
> >>
> >>
> >> --
> >> Cameron Hummels
> >> Postdoctoral Researcher
> >> Steward Observatory
> >> University of Arizona
> >> http://chummels.org
> >>
> >> _______________________________________________
> >> yt-users mailing list
> >> yt-users at lists.spacepope.org
> >> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
> >>
> >
> >
> > _______________________________________________
> > yt-users mailing list
> > yt-users at lists.spacepope.org
> > http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
> >
> _______________________________________________
> yt-users mailing list
> yt-users at lists.spacepope.org
> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>



-- 
Cameron Hummels
Postdoctoral Researcher
Steward Observatory
University of Arizona
http://chummels.org
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.spacepope.org/pipermail/yt-users-spacepope.org/attachments/20141204/46ad0619/attachment.htm>