[yt-users] Running yt in parallel on a subset of available processes

Nathan Goldbaum nathan12343 at gmail.com
Thu Dec 4 17:30:11 PST 2014


On Thu Dec 04 2014 at 5:08:18 PM Daniel Fenn <dsfenn at gmail.com> wrote:

> There's a good chance I created that directory at some point. That cuts
> down on my confusion a good deal. One final question--I'm trying to test
> out some of the recent changes that were made--specifically pull requests
> #1328 and #1331. I thought I would just have to do hg update in the hg-yt
> directory.
>
> The update worked fine, and it now says I'm at version 3.1-dev, but when I
> check the log, I don't see any mention of those pull requests. Did I miss
> them (very possible), or have they not been added to the main repository? I
> notice that they both still say "open". If that's the case, is there a way
> that I can download the changed files?
>

To test a PR, you're going to need to pull it in locally from the issuer's
fork of yt, then update to the tip of their PR.  I'm looking at our
documentation, and I don't think we cover this specific workflow, so this
is something that should probably be added.

Here what I do.  Go to the pull request page on bitbucket, and click over
to the commits tab.  I.e. this page for PR 1328:

https://bitbucket.org/yt_analysis/yt/pull-request/1328/delay-running-callbacks-until-the-plot/commits

That lists all of the changesets that are part of the pull request.  You're
going to need to copy/paste the changeset hash for the tip of the pull
request, in this case it's c62d7d0.

Next, in your local clone of yt, do:

$ hg pull -r c62d7d0 https://bitbucket.org/ngoldbaum/yt

Note that I'm pulling from the PR issuer's repository.  Pull requests are
not part of the main repository until they have been merged.

Next, just in case any compiled code changes (i.e. a cython .pyx file), run:

$ python setup.py develop

It's good to get into the habit of doing this, since you will run into
weird behavior if any compiled files changed and they didn't get recompiled.

Now the changeset in the tip of the pull request is the "active" change.
You can test the pull request.

To do back to "mainline" yt, you can issue the following command:

hg update <tip_hash>

Where <tip_hash> is the changeset hash corresponding to the current tip of
the "yt" branch on the main yt bitbucket repository. I usually look at the
following page to figure out what that is:

https://bitbucket.org/yt_analysis/yt/commits/branch/yt

You can also automate this workflow using the remotebranches extension for
mercurial:

http://mercurial.selenic.com/wiki/RemoteBranchesExtension

If at any time you get confused about where you are on the changeset graph,
take a look at the output of "hg log -G".  You can also use a gui like
tortoisehg or sourcetree, if you find that more comfortable.

Best,

Nathan


>
> Thanks for taking the time to answer all my questions.
>
> Dan
>
> 2014-12-04 19:30 GMT-05:00 Nathan Goldbaum <nathan12343 at gmail.com>:
>
>
>>
>> On Thu Dec 04 2014 at 4:23:06 PM Daniel Fenn <dsfenn at gmail.com> wrote:
>>
>>> Ok, thanks for the help--that ended up being pretty easy. Something else
>>> I'm confused about is why the same files appear in two different
>>> directories. There's the hg-yt directory in YT_DIR/src,
>>>
>>
>> The yt-hg directory is a clone of the yt mercurial repository. It gets
>> created by the install script.
>>
>>
>>> and then there's YT_DIR/yt, which seems to be the exact same thing.
>>> What's the difference between them?
>>>
>>
>> I don't have this directory in my install script build.  Is there a
>> chance you created this directory at some point?
>>
>>
>>> Another question--will the fork that I created be updated with future
>>> changes to the code, or do I have to manually update it, and, if so, how?
>>>
>>
>> You will need to manually keep it up to date.  You can do this by
>> clicking the "sync now" link on the overview page for your fork on the
>> bitbucket web interface, or by manually pushing changes to your fork using
>> "hg push".  You can configure mercurial so that your default push location
>> is your fork, see "hg help paths".
>>
>> That said, I don't think it's important to keep your fork up to date. I
>> generally have my paths setup so I pull directly from the main
>> yt_analysis/yt repository and then push to my fork only when I want to
>> issue pull requests.  If your fork is out of date, you when you push the
>> changes for your pull request, you will also push the changes that will
>> bring it up to date.
>>
>>
>>>
>>> Thanks for your help,
>>>
>>> Dan
>>>
>>> 2014-12-04 11:23 GMT-05:00 Cameron Hummels <chummels at gmail.com>:
>>>
>>> There is also some discussion on how to do it in the developer
>>>> documentation:
>>>>
>>>>
>>>> http://yt-project.org/docs/dev/developing/developing.html#making-and-sharing-changes
>>>>
>>>> Thanks for your hard work on this, and don't hesitate to ask for more
>>>> help!
>>>>
>>>> Cameron
>>>>
>>>> On Thu, Dec 4, 2014 at 8:05 AM, Matthew Turk <matthewturk at gmail.com>
>>>> wrote:
>>>>
>>>>> Hi Dan,
>>>>>
>>>>> Yup, you can commit in that directory, and push to a fork.  Create a
>>>>> fork on bitbucket, then do your changes and commit, and issue this
>>>>> command:
>>>>>
>>>>> hg push https://bitbucket.org/yourusername/yt
>>>>>
>>>>> Then you can issue a pull request via their web interface.  You can
>>>>> also edit your hgrc to add an alias, but that can happen later.
>>>>>
>>>>> -Matt
>>>>>
>>>>> On Thu, Dec 4, 2014 at 8:08 AM, Daniel Fenn <dsfenn at gmail.com> wrote:
>>>>> > Hi Cameron,
>>>>> >
>>>>> > Yeah, no problem. I'm not exactly sure how to do that. Do I have to
>>>>> fork yt
>>>>> > first, or can I use my existing installation since it's already using
>>>>> > Mercurial? I'm familiar with subversion, since that's what we use in
>>>>> our
>>>>> > research group, but I'm a bit hazy on how the whole thing is set up
>>>>> with yt.
>>>>> > When I installed yt using the installation script, it created a yt-hg
>>>>> > directory. Is that a checked-out version of the code that I can
>>>>> commit?
>>>>> >
>>>>> > Thanks!
>>>>> >
>>>>> > Dan
>>>>> >
>>>>> > 2014-12-03 20:59 GMT-05:00 Cameron Hummels <chummels at gmail.com>:
>>>>> >
>>>>> >> Hey Daniel,
>>>>> >>
>>>>> >> This is great that you got this working, and it sounds like
>>>>> something from
>>>>> >> which others could benefit! Would you mind submitting this change
>>>>> to the
>>>>> >> main codebase in a pull request?
>>>>> >>
>>>>> >> Cameron
>>>>> >>
>>>>> >> On Wed, Dec 3, 2014 at 6:46 PM, Daniel Fenn <dsfenn at gmail.com>
>>>>> wrote:
>>>>> >>>
>>>>> >>> Hi Matt,
>>>>> >>>
>>>>> >>> I was able to get my MPI problem fixed by using subcommunicators. I
>>>>> >>> modified enable_parallelism() to take an optional argument that
>>>>> specifies
>>>>> >>> the MPI communicator yt should use. If none is specified, it
>>>>> defaults to
>>>>> >>> COMM_WORLD. Seems to be working pretty well so far.
>>>>> >>>
>>>>> >>> Thanks for the advice,
>>>>> >>>
>>>>> >>> Dan
>>>>> >>>
>>>>> >>> 2014-12-03 19:16 GMT-05:00 Matthew Turk <matthewturk at gmail.com>:
>>>>> >>>
>>>>> >>>> Hi Dan,
>>>>> >>>>
>>>>> >>>> I've done this before by modifying how the global communicator is
>>>>> >>>> created.  The way they work in yt is through pushing new
>>>>> communicators
>>>>> >>>> onto a stack, which can vary based on which processors you're
>>>>> on.  I
>>>>> >>>> don't think I ever wrote it up in a PR, but I think it could be
>>>>> done
>>>>> >>>> without too much work.
>>>>> >>>>
>>>>> >>>> On Wed, Dec 3, 2014 at 11:47 AM, Daniel Fenn <dsfenn at gmail.com>
>>>>> wrote:
>>>>> >>>> > Is there a way to restrict yt to only use a subset of the
>>>>> available
>>>>> >>>> > processes when it's running in parallel? I have a set of ipython
>>>>> >>>> > engines
>>>>> >>>> > running, and I'm trying to launch a yt script on a subset of
>>>>> them, but
>>>>> >>>> > it
>>>>> >>>> > just hangs. It works fine if I launch the same script on all the
>>>>> >>>> > processes.
>>>>> >>>> >
>>>>> >>>> > Printing out MPI.COMM_WORLD.Get_size() from the script gives
>>>>> the total
>>>>> >>>> > number of engines, not the size of the subset I'm trying to use.
>>>>> >>>> >
>>>>> >>>> > I haven't done a lot of parallel programming, but I would guess
>>>>> that
>>>>> >>>> > the
>>>>> >>>> > running processes are waiting on one that's not running a yt
>>>>> script,
>>>>> >>>> > since
>>>>> >>>> > they can see the total number of processes available, but not
>>>>> the size
>>>>> >>>> > of
>>>>> >>>> > the subset of processes that are actually running a yt script.
>>>>> >>>> >
>>>>> >>>> > Does anyone have any thoughts on how to do this?
>>>>> >>>> >
>>>>> >>>> > Thanks,
>>>>> >>>> >
>>>>> >>>> > Dan
>>>>> >>>> >
>>>>> >>>> > _______________________________________________
>>>>> >>>> > yt-users mailing list
>>>>> >>>> > yt-users at lists.spacepope.org
>>>>> >>>> > http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>>>>> >>>> >
>>>>> >>>> _______________________________________________
>>>>> >>>> yt-users mailing list
>>>>> >>>> yt-users at lists.spacepope.org
>>>>> >>>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>>>>> >>>
>>>>> >>>
>>>>> >>>
>>>>> >>> _______________________________________________
>>>>> >>> yt-users mailing list
>>>>> >>> yt-users at lists.spacepope.org
>>>>> >>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>>>>> >>>
>>>>> >>
>>>>> >>
>>>>> >>
>>>>> >> --
>>>>> >> Cameron Hummels
>>>>> >> Postdoctoral Researcher
>>>>> >> Steward Observatory
>>>>> >> University of Arizona
>>>>> >> http://chummels.org
>>>>> >>
>>>>> >> _______________________________________________
>>>>> >> yt-users mailing list
>>>>> >> yt-users at lists.spacepope.org
>>>>> >> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>>>>> >>
>>>>> >
>>>>> >
>>>>> > _______________________________________________
>>>>> > yt-users mailing list
>>>>> > yt-users at lists.spacepope.org
>>>>> > http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>>>>> >
>>>>> _______________________________________________
>>>>> yt-users mailing list
>>>>> yt-users at lists.spacepope.org
>>>>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Cameron Hummels
>>>> Postdoctoral Researcher
>>>> Steward Observatory
>>>> University of Arizona
>>>> http://chummels.org
>>>>
>>>> _______________________________________________
>>>> yt-users mailing list
>>>> yt-users at lists.spacepope.org
>>>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>>>>
>>>> _______________________________________________
>>> yt-users mailing list
>>> yt-users at lists.spacepope.org
>>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>>>
>>
>> _______________________________________________
>> yt-users mailing list
>> yt-users at lists.spacepope.org
>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>>
>> _______________________________________________
> yt-users mailing list
> yt-users at lists.spacepope.org
> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.spacepope.org/pipermail/yt-users-spacepope.org/attachments/20141205/4b647bc1/attachment-0001.htm>
-------------- next part --------------
_______________________________________________
yt-users mailing list
yt-users at lists.spacepope.org
http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org


More information about the yt-users mailing list