[yt-users] Running yt in parallel on a subset of available processes

Nathan Goldbaum nathan12343 at gmail.com
Thu Dec 4 16:30:11 PST 2014


On Thu Dec 04 2014 at 4:23:06 PM Daniel Fenn <dsfenn at gmail.com> wrote:

> Ok, thanks for the help--that ended up being pretty easy. Something else
> I'm confused about is why the same files appear in two different
> directories. There's the hg-yt directory in YT_DIR/src,
>

The yt-hg directory is a clone of the yt mercurial repository. It gets
created by the install script.


> and then there's YT_DIR/yt, which seems to be the exact same thing. What's
> the difference between them?
>

I don't have this directory in my install script build.  Is there a chance
you created this directory at some point?


> Another question--will the fork that I created be updated with future
> changes to the code, or do I have to manually update it, and, if so, how?
>

You will need to manually keep it up to date.  You can do this by clicking
the "sync now" link on the overview page for your fork on the bitbucket web
interface, or by manually pushing changes to your fork using "hg push".
You can configure mercurial so that your default push location is your
fork, see "hg help paths".

That said, I don't think it's important to keep your fork up to date. I
generally have my paths setup so I pull directly from the main
yt_analysis/yt repository and then push to my fork only when I want to
issue pull requests.  If your fork is out of date, you when you push the
changes for your pull request, you will also push the changes that will
bring it up to date.


>
> Thanks for your help,
>
> Dan
>
> 2014-12-04 11:23 GMT-05:00 Cameron Hummels <chummels at gmail.com>:
>
> There is also some discussion on how to do it in the developer
>> documentation:
>>
>>
>> http://yt-project.org/docs/dev/developing/developing.html#making-and-sharing-changes
>>
>> Thanks for your hard work on this, and don't hesitate to ask for more
>> help!
>>
>> Cameron
>>
>> On Thu, Dec 4, 2014 at 8:05 AM, Matthew Turk <matthewturk at gmail.com>
>> wrote:
>>
>>> Hi Dan,
>>>
>>> Yup, you can commit in that directory, and push to a fork.  Create a
>>> fork on bitbucket, then do your changes and commit, and issue this
>>> command:
>>>
>>> hg push https://bitbucket.org/yourusername/yt
>>>
>>> Then you can issue a pull request via their web interface.  You can
>>> also edit your hgrc to add an alias, but that can happen later.
>>>
>>> -Matt
>>>
>>> On Thu, Dec 4, 2014 at 8:08 AM, Daniel Fenn <dsfenn at gmail.com> wrote:
>>> > Hi Cameron,
>>> >
>>> > Yeah, no problem. I'm not exactly sure how to do that. Do I have to
>>> fork yt
>>> > first, or can I use my existing installation since it's already using
>>> > Mercurial? I'm familiar with subversion, since that's what we use in
>>> our
>>> > research group, but I'm a bit hazy on how the whole thing is set up
>>> with yt.
>>> > When I installed yt using the installation script, it created a yt-hg
>>> > directory. Is that a checked-out version of the code that I can commit?
>>> >
>>> > Thanks!
>>> >
>>> > Dan
>>> >
>>> > 2014-12-03 20:59 GMT-05:00 Cameron Hummels <chummels at gmail.com>:
>>> >
>>> >> Hey Daniel,
>>> >>
>>> >> This is great that you got this working, and it sounds like something
>>> from
>>> >> which others could benefit! Would you mind submitting this change to
>>> the
>>> >> main codebase in a pull request?
>>> >>
>>> >> Cameron
>>> >>
>>> >> On Wed, Dec 3, 2014 at 6:46 PM, Daniel Fenn <dsfenn at gmail.com> wrote:
>>> >>>
>>> >>> Hi Matt,
>>> >>>
>>> >>> I was able to get my MPI problem fixed by using subcommunicators. I
>>> >>> modified enable_parallelism() to take an optional argument that
>>> specifies
>>> >>> the MPI communicator yt should use. If none is specified, it
>>> defaults to
>>> >>> COMM_WORLD. Seems to be working pretty well so far.
>>> >>>
>>> >>> Thanks for the advice,
>>> >>>
>>> >>> Dan
>>> >>>
>>> >>> 2014-12-03 19:16 GMT-05:00 Matthew Turk <matthewturk at gmail.com>:
>>> >>>
>>> >>>> Hi Dan,
>>> >>>>
>>> >>>> I've done this before by modifying how the global communicator is
>>> >>>> created.  The way they work in yt is through pushing new
>>> communicators
>>> >>>> onto a stack, which can vary based on which processors you're on.  I
>>> >>>> don't think I ever wrote it up in a PR, but I think it could be done
>>> >>>> without too much work.
>>> >>>>
>>> >>>> On Wed, Dec 3, 2014 at 11:47 AM, Daniel Fenn <dsfenn at gmail.com>
>>> wrote:
>>> >>>> > Is there a way to restrict yt to only use a subset of the
>>> available
>>> >>>> > processes when it's running in parallel? I have a set of ipython
>>> >>>> > engines
>>> >>>> > running, and I'm trying to launch a yt script on a subset of
>>> them, but
>>> >>>> > it
>>> >>>> > just hangs. It works fine if I launch the same script on all the
>>> >>>> > processes.
>>> >>>> >
>>> >>>> > Printing out MPI.COMM_WORLD.Get_size() from the script gives the
>>> total
>>> >>>> > number of engines, not the size of the subset I'm trying to use.
>>> >>>> >
>>> >>>> > I haven't done a lot of parallel programming, but I would guess
>>> that
>>> >>>> > the
>>> >>>> > running processes are waiting on one that's not running a yt
>>> script,
>>> >>>> > since
>>> >>>> > they can see the total number of processes available, but not the
>>> size
>>> >>>> > of
>>> >>>> > the subset of processes that are actually running a yt script.
>>> >>>> >
>>> >>>> > Does anyone have any thoughts on how to do this?
>>> >>>> >
>>> >>>> > Thanks,
>>> >>>> >
>>> >>>> > Dan
>>> >>>> >
>>> >>>> > _______________________________________________
>>> >>>> > yt-users mailing list
>>> >>>> > yt-users at lists.spacepope.org
>>> >>>> > http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>>> >>>> >
>>> >>>> _______________________________________________
>>> >>>> yt-users mailing list
>>> >>>> yt-users at lists.spacepope.org
>>> >>>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>>> >>>
>>> >>>
>>> >>>
>>> >>> _______________________________________________
>>> >>> yt-users mailing list
>>> >>> yt-users at lists.spacepope.org
>>> >>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>>> >>>
>>> >>
>>> >>
>>> >>
>>> >> --
>>> >> Cameron Hummels
>>> >> Postdoctoral Researcher
>>> >> Steward Observatory
>>> >> University of Arizona
>>> >> http://chummels.org
>>> >>
>>> >> _______________________________________________
>>> >> yt-users mailing list
>>> >> yt-users at lists.spacepope.org
>>> >> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>>> >>
>>> >
>>> >
>>> > _______________________________________________
>>> > yt-users mailing list
>>> > yt-users at lists.spacepope.org
>>> > http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>>> >
>>> _______________________________________________
>>> yt-users mailing list
>>> yt-users at lists.spacepope.org
>>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>>>
>>
>>
>>
>> --
>> Cameron Hummels
>> Postdoctoral Researcher
>> Steward Observatory
>> University of Arizona
>> http://chummels.org
>>
>> _______________________________________________
>> yt-users mailing list
>> yt-users at lists.spacepope.org
>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>>
>> _______________________________________________
> yt-users mailing list
> yt-users at lists.spacepope.org
> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.spacepope.org/pipermail/yt-users-spacepope.org/attachments/20141205/7a44f619/attachment.htm>


More information about the yt-users mailing list