[yt-users] ipython on a cluster

[Nathan Goldbaum]

Hey Rick,

Can we keep this discussion on the mailing list?  Just in case others in
the future have similar problems.

How did you install yt?  If you used the install script, did you remember
to activate the yt environment before launching ipcluster?  Is IPython
installed into yt's environment?

-Nathan

On Tue, Jan 20, 2015 at 3:31 PM, Rick Sarmento <rsarment at asu.edu> wrote:

> Hum… I used the ipcluster command on the vis node I’m using
>
> And I get through all your init steps:
>
> In [1]:
>
> from IPython.parallel import Client
> rc = Client()
> In [2]:
>
> rc.ids
> Out[2]:
> [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15]
> In [3]:
>
> %autopx
> %autopx enabled
>
> In [4]:
>
> from mpi4py import MPI
>
> MPI.COMM_WORLD.Get_rank()
>
> Out[0:1]: 8
> Out[1:1]: 1
> Out[2:1]: 14
> Out[3:1]: 7
> Out[4:1]: 9
> Out[5:1]: 15
> Out[6:1]: 6
> Out[7:1]: 12
> Out[8:1]: 4
> Out[9:1]: 0
> Out[10:1]: 5
> Out[11:1]: 10
> Out[12:1]: 2
> Out[13:1]: 11
> Out[14:1]: 3
> Out[15:1]: 13
> In [5]:
>
>
> But then it can’t find yt!!
>
> import yt
> yt.enable_parallelism()
> import numpy as np
> import os
> import shutil
> import h5py
> import gc
> import glob
>
>
> AttributeError                            Traceback (most recent call last)<ipython-input-3-915313e529f2> in <module>()----> 1 import yt
>
> ...
>
>
> Weird tho … cuz if I kill the cluster and try again, import YT loads… *Can
> you think of anything obvious?* It’s like my path to yt goes away once I
> start the cluster … and I’m getting some default version (old) on stampede.
>
> Thanks Nathan .. I’m watching the tutorial now …
>
> *Cheers!*
>
> *Rick Sarmento*
> SESE Astronomy/Astrophysics Grad Student
> rsarment at asu.edu
>
>
>
>
>
>
>
> On Jan 17, 2015, at 12:54 PM, Rick Sarmento <rsarment at asu.edu> wrote:
>
> Thanks for the info Nathan…
>
> I’m trying to use STAMPEDE out at Texas (http://vis.tacc.xsede.org/), and
> they have a way to kick-off access to their 16 core visualization nodes via
> the web… But that probably doesn’t help given the info below. I’ll try to
> work with support to see how I can get it to work…
>
> If I start the cluster using your instructions (via a ssh session), below,
> I’d have to login — but then I can’t get remote access to the notebook
> interface because of the firewalls, etc. Although there’s like a way to
> setup an ssh tunnel, I haven’t figured it out yet.
>
> Thx!
>
> *Cheers!*
>
> *Rick Sarmento*
> SESE Astronomy/Astrophysics Grad Student
> rsarment at asu.edu
>
> Hi Rick,
>
> You need to launch the notebook in parallel using an MPI parallel IPython
> cluster.  In addition, you will need to configure the notebook to hook into
> IPython's built-in parallelism.
>
> I've uploaded a notebook that uses parallel IPython here:
>
> http://nbviewer.ipython.org/gist/ngoldbaum/32ca07cf4f5b3dd06add
>
> Note that all the yt operations aren't really important, but the first few
> cells where I check to make sure IPython's parallelism works and that MPI
> parllelism is working are quite important.
>
> Note also that you will need to launch the IPython cluster separately from
> the notebook server using the "ipcluster" command.
>
> If you want to learn more about parallel IPython, I'd encourage you to
> take a look at Min Ranger-Kelley's tutorial from scipy 2014:
> http://pyvideo.org/video/2738/interactive-parallel-computing-with-ipython-part
>
> All that said, it does add a significant amount of semantic overhead to
> use the IPython notebook in parallel.  It's generally much more
> straightforward to work with yt in parallel using regular python scripts.
>
> Hope that helps,
>
> Nathan
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.spacepope.org/pipermail/yt-users-spacepope.org/attachments/20150120/186b9b39/attachment.htm>
-------------- next part --------------
_______________________________________________
yt-users mailing list
yt-users at lists.spacepope.org
http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org