[yt-users] Running yt in parallel on a subset of available processes

Daniel Fenn dsfenn at gmail.com
Thu Dec 4 17:06:41 PST 2014


There's a good chance I created that directory at some point. That cuts
down on my confusion a good deal. One final question--I'm trying to test
out some of the recent changes that were made--specifically pull requests
#1328 and #1331. I thought I would just have to do hg update in the hg-yt
directory.

The update worked fine, and it now says I'm at version 3.1-dev, but when I
check the log, I don't see any mention of those pull requests. Did I miss
them (very possible), or have they not been added to the main repository? I
notice that they both still say "open". If that's the case, is there a way
that I can download the changed files?

Thanks for taking the time to answer all my questions.

Dan

2014-12-04 19:30 GMT-05:00 Nathan Goldbaum <nathan12343 at gmail.com>:

>
>
> On Thu Dec 04 2014 at 4:23:06 PM Daniel Fenn <dsfenn at gmail.com> wrote:
>
>> Ok, thanks for the help--that ended up being pretty easy. Something else
>> I'm confused about is why the same files appear in two different
>> directories. There's the hg-yt directory in YT_DIR/src,
>>
>
> The yt-hg directory is a clone of the yt mercurial repository. It gets
> created by the install script.
>
>
>> and then there's YT_DIR/yt, which seems to be the exact same thing.
>> What's the difference between them?
>>
>
> I don't have this directory in my install script build.  Is there a chance
> you created this directory at some point?
>
>
>> Another question--will the fork that I created be updated with future
>> changes to the code, or do I have to manually update it, and, if so, how?
>>
>
> You will need to manually keep it up to date.  You can do this by clicking
> the "sync now" link on the overview page for your fork on the bitbucket web
> interface, or by manually pushing changes to your fork using "hg push".
> You can configure mercurial so that your default push location is your
> fork, see "hg help paths".
>
> That said, I don't think it's important to keep your fork up to date. I
> generally have my paths setup so I pull directly from the main
> yt_analysis/yt repository and then push to my fork only when I want to
> issue pull requests.  If your fork is out of date, you when you push the
> changes for your pull request, you will also push the changes that will
> bring it up to date.
>
>
>>
>> Thanks for your help,
>>
>> Dan
>>
>> 2014-12-04 11:23 GMT-05:00 Cameron Hummels <chummels at gmail.com>:
>>
>> There is also some discussion on how to do it in the developer
>>> documentation:
>>>
>>>
>>> http://yt-project.org/docs/dev/developing/developing.html#making-and-sharing-changes
>>>
>>> Thanks for your hard work on this, and don't hesitate to ask for more
>>> help!
>>>
>>> Cameron
>>>
>>> On Thu, Dec 4, 2014 at 8:05 AM, Matthew Turk <matthewturk at gmail.com>
>>> wrote:
>>>
>>>> Hi Dan,
>>>>
>>>> Yup, you can commit in that directory, and push to a fork.  Create a
>>>> fork on bitbucket, then do your changes and commit, and issue this
>>>> command:
>>>>
>>>> hg push https://bitbucket.org/yourusername/yt
>>>>
>>>> Then you can issue a pull request via their web interface.  You can
>>>> also edit your hgrc to add an alias, but that can happen later.
>>>>
>>>> -Matt
>>>>
>>>> On Thu, Dec 4, 2014 at 8:08 AM, Daniel Fenn <dsfenn at gmail.com> wrote:
>>>> > Hi Cameron,
>>>> >
>>>> > Yeah, no problem. I'm not exactly sure how to do that. Do I have to
>>>> fork yt
>>>> > first, or can I use my existing installation since it's already using
>>>> > Mercurial? I'm familiar with subversion, since that's what we use in
>>>> our
>>>> > research group, but I'm a bit hazy on how the whole thing is set up
>>>> with yt.
>>>> > When I installed yt using the installation script, it created a yt-hg
>>>> > directory. Is that a checked-out version of the code that I can
>>>> commit?
>>>> >
>>>> > Thanks!
>>>> >
>>>> > Dan
>>>> >
>>>> > 2014-12-03 20:59 GMT-05:00 Cameron Hummels <chummels at gmail.com>:
>>>> >
>>>> >> Hey Daniel,
>>>> >>
>>>> >> This is great that you got this working, and it sounds like
>>>> something from
>>>> >> which others could benefit! Would you mind submitting this change to
>>>> the
>>>> >> main codebase in a pull request?
>>>> >>
>>>> >> Cameron
>>>> >>
>>>> >> On Wed, Dec 3, 2014 at 6:46 PM, Daniel Fenn <dsfenn at gmail.com>
>>>> wrote:
>>>> >>>
>>>> >>> Hi Matt,
>>>> >>>
>>>> >>> I was able to get my MPI problem fixed by using subcommunicators. I
>>>> >>> modified enable_parallelism() to take an optional argument that
>>>> specifies
>>>> >>> the MPI communicator yt should use. If none is specified, it
>>>> defaults to
>>>> >>> COMM_WORLD. Seems to be working pretty well so far.
>>>> >>>
>>>> >>> Thanks for the advice,
>>>> >>>
>>>> >>> Dan
>>>> >>>
>>>> >>> 2014-12-03 19:16 GMT-05:00 Matthew Turk <matthewturk at gmail.com>:
>>>> >>>
>>>> >>>> Hi Dan,
>>>> >>>>
>>>> >>>> I've done this before by modifying how the global communicator is
>>>> >>>> created.  The way they work in yt is through pushing new
>>>> communicators
>>>> >>>> onto a stack, which can vary based on which processors you're on.
>>>> I
>>>> >>>> don't think I ever wrote it up in a PR, but I think it could be
>>>> done
>>>> >>>> without too much work.
>>>> >>>>
>>>> >>>> On Wed, Dec 3, 2014 at 11:47 AM, Daniel Fenn <dsfenn at gmail.com>
>>>> wrote:
>>>> >>>> > Is there a way to restrict yt to only use a subset of the
>>>> available
>>>> >>>> > processes when it's running in parallel? I have a set of ipython
>>>> >>>> > engines
>>>> >>>> > running, and I'm trying to launch a yt script on a subset of
>>>> them, but
>>>> >>>> > it
>>>> >>>> > just hangs. It works fine if I launch the same script on all the
>>>> >>>> > processes.
>>>> >>>> >
>>>> >>>> > Printing out MPI.COMM_WORLD.Get_size() from the script gives the
>>>> total
>>>> >>>> > number of engines, not the size of the subset I'm trying to use.
>>>> >>>> >
>>>> >>>> > I haven't done a lot of parallel programming, but I would guess
>>>> that
>>>> >>>> > the
>>>> >>>> > running processes are waiting on one that's not running a yt
>>>> script,
>>>> >>>> > since
>>>> >>>> > they can see the total number of processes available, but not
>>>> the size
>>>> >>>> > of
>>>> >>>> > the subset of processes that are actually running a yt script.
>>>> >>>> >
>>>> >>>> > Does anyone have any thoughts on how to do this?
>>>> >>>> >
>>>> >>>> > Thanks,
>>>> >>>> >
>>>> >>>> > Dan
>>>> >>>> >
>>>> >>>> > _______________________________________________
>>>> >>>> > yt-users mailing list
>>>> >>>> > yt-users at lists.spacepope.org
>>>> >>>> > http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>>>> >>>> >
>>>> >>>> _______________________________________________
>>>> >>>> yt-users mailing list
>>>> >>>> yt-users at lists.spacepope.org
>>>> >>>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>>>> >>>
>>>> >>>
>>>> >>>
>>>> >>> _______________________________________________
>>>> >>> yt-users mailing list
>>>> >>> yt-users at lists.spacepope.org
>>>> >>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>>>> >>>
>>>> >>
>>>> >>
>>>> >>
>>>> >> --
>>>> >> Cameron Hummels
>>>> >> Postdoctoral Researcher
>>>> >> Steward Observatory
>>>> >> University of Arizona
>>>> >> http://chummels.org
>>>> >>
>>>> >> _______________________________________________
>>>> >> yt-users mailing list
>>>> >> yt-users at lists.spacepope.org
>>>> >> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>>>> >>
>>>> >
>>>> >
>>>> > _______________________________________________
>>>> > yt-users mailing list
>>>> > yt-users at lists.spacepope.org
>>>> > http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>>>> >
>>>> _______________________________________________
>>>> yt-users mailing list
>>>> yt-users at lists.spacepope.org
>>>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>>>>
>>>
>>>
>>>
>>> --
>>> Cameron Hummels
>>> Postdoctoral Researcher
>>> Steward Observatory
>>> University of Arizona
>>> http://chummels.org
>>>
>>> _______________________________________________
>>> yt-users mailing list
>>> yt-users at lists.spacepope.org
>>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>>>
>>> _______________________________________________
>> yt-users mailing list
>> yt-users at lists.spacepope.org
>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>>
>
> _______________________________________________
> yt-users mailing list
> yt-users at lists.spacepope.org
> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.spacepope.org/pipermail/yt-users-spacepope.org/attachments/20141204/1bb4535c/attachment.htm>
-------------- next part --------------
_______________________________________________
yt-users mailing list
yt-users at lists.spacepope.org
http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org


More information about the yt-users mailing list