[yt-users] Running yt in parallel on a subset of available processes

Thu Dec 4 16:21:00 PST 2014

Ok, thanks for the help--that ended up being pretty easy. Something else
I'm confused about is why the same files appear in two different
directories. There's the hg-yt directory in YT_DIR/src, and then there's
YT_DIR/yt, which seems to be the exact same thing. What's the difference
between them?

Another question--will the fork that I created be updated with future
changes to the code, or do I have to manually update it, and, if so, how?

Thanks for your help,

Dan

2014-12-04 11:23 GMT-05:00 Cameron Hummels <chummels at gmail.com>:

> There is also some discussion on how to do it in the developer
> documentation:
>
>
> http://yt-project.org/docs/dev/developing/developing.html#making-and-sharing-changes
>
> Thanks for your hard work on this, and don't hesitate to ask for more help!
>
> Cameron
>
> On Thu, Dec 4, 2014 at 8:05 AM, Matthew Turk <matthewturk at gmail.com>
> wrote:
>
>> Hi Dan,
>>
>> Yup, you can commit in that directory, and push to a fork.  Create a
>> fork on bitbucket, then do your changes and commit, and issue this
>> command:
>>
>> hg push https://bitbucket.org/yourusername/yt
>>
>> Then you can issue a pull request via their web interface.  You can
>> also edit your hgrc to add an alias, but that can happen later.
>>
>> -Matt
>>
>> On Thu, Dec 4, 2014 at 8:08 AM, Daniel Fenn <dsfenn at gmail.com> wrote:
>> > Hi Cameron,
>> >
>> > Yeah, no problem. I'm not exactly sure how to do that. Do I have to
>> fork yt
>> > first, or can I use my existing installation since it's already using
>> > Mercurial? I'm familiar with subversion, since that's what we use in our
>> > research group, but I'm a bit hazy on how the whole thing is set up
>> with yt.
>> > When I installed yt using the installation script, it created a yt-hg
>> > directory. Is that a checked-out version of the code that I can commit?
>> >
>> > Thanks!
>> >
>> > Dan
>> >
>> > 2014-12-03 20:59 GMT-05:00 Cameron Hummels <chummels at gmail.com>:
>> >
>> >> Hey Daniel,
>> >>
>> >> This is great that you got this working, and it sounds like something
>> from
>> >> which others could benefit! Would you mind submitting this change to
>> the
>> >> main codebase in a pull request?
>> >>
>> >> Cameron
>> >>
>> >> On Wed, Dec 3, 2014 at 6:46 PM, Daniel Fenn <dsfenn at gmail.com> wrote:
>> >>>
>> >>> Hi Matt,
>> >>>
>> >>> I was able to get my MPI problem fixed by using subcommunicators. I
>> >>> modified enable_parallelism() to take an optional argument that
>> specifies
>> >>> the MPI communicator yt should use. If none is specified, it defaults
>> to
>> >>> COMM_WORLD. Seems to be working pretty well so far.
>> >>>
>> >>> Thanks for the advice,
>> >>>
>> >>> Dan
>> >>>
>> >>> 2014-12-03 19:16 GMT-05:00 Matthew Turk <matthewturk at gmail.com>:
>> >>>
>> >>>> Hi Dan,
>> >>>>
>> >>>> I've done this before by modifying how the global communicator is
>> >>>> created.  The way they work in yt is through pushing new
>> communicators
>> >>>> onto a stack, which can vary based on which processors you're on.  I
>> >>>> don't think I ever wrote it up in a PR, but I think it could be done
>> >>>> without too much work.
>> >>>>
>> >>>> On Wed, Dec 3, 2014 at 11:47 AM, Daniel Fenn <dsfenn at gmail.com>
>> wrote:
>> >>>> > Is there a way to restrict yt to only use a subset of the available
>> >>>> > processes when it's running in parallel? I have a set of ipython
>> >>>> > engines
>> >>>> > running, and I'm trying to launch a yt script on a subset of them,
>> but
>> >>>> > it
>> >>>> > just hangs. It works fine if I launch the same script on all the
>> >>>> > processes.
>> >>>> >
>> >>>> > Printing out MPI.COMM_WORLD.Get_size() from the script gives the
>> total
>> >>>> > number of engines, not the size of the subset I'm trying to use.
>> >>>> >
>> >>>> > I haven't done a lot of parallel programming, but I would guess
>> that
>> >>>> > the
>> >>>> > running processes are waiting on one that's not running a yt
>> script,
>> >>>> > since
>> >>>> > they can see the total number of processes available, but not the
>> size
>> >>>> > of
>> >>>> > the subset of processes that are actually running a yt script.
>> >>>> >
>> >>>> > Does anyone have any thoughts on how to do this?
>> >>>> >
>> >>>> > Thanks,
>> >>>> >
>> >>>> > Dan
>> >>>> >
>> >>>> > _______________________________________________
>> >>>> > yt-users mailing list
>> >>>> > yt-users at lists.spacepope.org
>> >>>> > http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>> >>>> >
>> >>>> _______________________________________________
>> >>>> yt-users mailing list
>> >>>> yt-users at lists.spacepope.org
>> >>>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>> >>>
>> >>>
>> >>>
>> >>> _______________________________________________
>> >>> yt-users mailing list
>> >>> yt-users at lists.spacepope.org
>> >>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>> >>>
>> >>
>> >>
>> >>
>> >> --
>> >> Cameron Hummels
>> >> Postdoctoral Researcher
>> >> Steward Observatory
>> >> University of Arizona
>> >> http://chummels.org
>> >>
>> >> _______________________________________________
>> >> yt-users mailing list
>> >> yt-users at lists.spacepope.org
>> >> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>> >>
>> >
>> >
>> > _______________________________________________
>> > yt-users mailing list
>> > yt-users at lists.spacepope.org
>> > http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>> >
>> _______________________________________________
>> yt-users mailing list
>> yt-users at lists.spacepope.org
>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>>
>
>
>
> --
> Cameron Hummels
> Postdoctoral Researcher
> Steward Observatory
> University of Arizona
> http://chummels.org
>
> _______________________________________________
> yt-users mailing list
> yt-users at lists.spacepope.org
> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.spacepope.org/pipermail/yt-users-spacepope.org/attachments/20141205/e7c16e29/attachment.htm>