[yt-users] installing YT without internet connection

Luigi Iapichino luigi at uni-heidelberg.de
Mon Feb 3 06:50:43 PST 2014


Dear Philipp, dear yt users,

I attach to this email the modified version of my install script, used to 
install yt 2.6 on a machine without internet access. The procedure for the 
installation is also attached as a separate txt file, but I copy it below, for 
sake of simplicity, in case of later discussion.

As I wrote a few hours ago, this is not straightforward, but it works for me, 
and I haven't come to any better solution. If anyone else has better 
suggestions, I would be glad to hear from them.

Cheers,

  Luigi

---

Here I describe the installation of yt 2.6. For other versions, please check 
the version numbers of the software that I mention below.

Let's start with a bit of syntax:

<remote-machine>: the remote machine without Internet access (a high-
performance supercomputer, most of the times);
<remote-username>: your username on that machine;
<remote-directory>: a directory containing the yt source, in the remote 
machine (create it, if not already existing);
<local-machine>: a PC with Internet access (your local PC, for example)
<local-directory>: on your local machine, this is an initially empty directory 
where you will mount the content of <remote-directory>.

Moreover, every command given below will be preceded by <remote-machine> or 
<local-machine>, indicating on which machine it should be executed.

First of all, in order to install yt, or to update it by using Mercurial on a 
machine with no Internet access, you have to remotely mount the source 
directory in your local PC. 
So, in order to remotely mount <remote-directory> on <local-machine>, the 
command is:

(<local-machine>) sshfs -o workaround=rename <remote-username>@<remote-
machine>:<remote-directory> <local-directory> 

After this command, on <local-directory> you will find the content of <remote-
directory>. After working with it, you can unmount with 

(<local-machine>) fusermount -u <local-directory>

Beware: both sshfs and fusermount have to be installed on <local-machine>, and 
in order to use them you should be inserted in some special user group (I 
don't remember which one exactly, but if you ask your local administrator it 
will be fixed for you).

Now, the yt part: I attach here a modified install script. The most outstanding 
changes with respect to the standard script are: 

- two paths for DEST_DIR, one for <local-machine> and one for <remote-
machine>. You must switch between the two of them, because the script will be 
executed partly on <local-machine>, and partly on <remote-machine>.

- INST_HG=0 : you don't need Mercurial on <remote-machine>, but you have it 
already on <local-machine>. Beware: it's somehow funny, but although you don't 
need  hg on <remote-machine>, it should be installed there. If not, please 
substite the lines 803-810 with the following ones:

    # We assume that hg can be found in the path.
#    if type -P hg &>/dev/null
#    then
        export HG_EXEC=hg
#    else
#        echo "Cannot find mercurial.  Please set INST_HG=1."
#        do_exit
#    fi

which simply disable a check that would raise an error.

- Comment the installation of pip (l. 844-845: it does not work on SuperMUC, 
with this installation procedure)

- Installation of distribute: before starting the script, uncomment the lines 
841-842. After the install script reaches that point, comment them again. 

- l. 882 to 885: I moved there the installation of some required software, not 
included in the initial download step. 

Here is finally the installation procedure:

1) (<local-machine>) Perform the remote mount, as described above.

2) (<local-machine>) Create a directory called yt-... and copy there the 
modified installation script. Make the script executable. 

3) (<local-machine>) Execute the script, up to the point where the sources are 
downloaded. The file distribute-0.6.32.tar.gz is not taken together with the 
other ones, It must be added to src manually (important!). Moreover, you need 
two additional source files copied in src: pyparsing-1.5.6.tar.gz and python-
dateutil-1.5.tar.gz (look for them on the Web). 

4) (<remote-machine>) Execute again the install script, which now compiles the 
tools until Python and then fails. 

5) (<local-machine>) For going on, the script executes the hg-related commands 
for the yt source. After this part it crashes, and I go again to <remote-
machine>. 

6) (<remote-machine>) It will try to update yt from here, failing. Stop again 
and do it from <local-machine>, then go back to (<remote-machine>). 

7) (<remote-machine>) Go on with the script, which (hopefully!) now proceeds 
to the end without further problems.

---

On Monday 03 February 2014 09:27:31 Luigi Iapichino wrote:
> Hi Philipp,
> 
> I did exactly what you are looking for, on SuperMUC in Garching. I will send
> my modified version of the install script, with instructions for the
> installation, in the course of the day. It's a rather cumbersome procedure
> so, if anyone else has met and solved the same problem, please join the
> discussion, so that we can compare and improve together.
> 
> Cheers,
> 
>   Luigi
> 
> On Monday 03 February 2014 08:47:12 Philipp Girichidis wrote:
> > Dear YT users,
> > 
> > I have a problem during the installation of yt. On the supercomputer,
> > where
> > all our simulations are stored, we do not have internet connection because
> > of 'security reasons'. So, I tried to download the data on another
> > machine,
> > then copy the files to this supercomputer and tried to install everything
> > from the packages. But there are quite a few other points in the install
> > script, where the setup process needs access to the internet. I also tried
> > to set up a virtual machine with a linux system similar to the one in the
> > computing center, but no success. Did anyone run into a similar problem
> > and
> > got a solution or has a guess how to first download everything and then
> > installing yt without further internet connection?
> > Best regards,
> > Philipp

-- 

---------------------------------------------------------------

Luigi Iapichino
Universität Heidelberg, Zentrum für Astronomie
Institut für Theoretische Astrophysik
Philosophenweg 12, D-69120 Heidelberg, Germany
Tel: +49 6221 548983, Fax: +49 6221 544221
e-mail: luigi at uni-heidelberg.de
URL: http://www.ita.uni-heidelberg.de/~luigi/
-------------- next part --------------
A non-text attachment was scrubbed...
Name: install_script-remote.sh
Type: application/x-shellscript
Size: 39687 bytes
Desc: not available
URL: <http://lists.spacepope.org/pipermail/yt-users-spacepope.org/attachments/20140203/bcb4527c/attachment.bin>
-------------- next part --------------
Here I describe the installation of yt 2.6. For other versions, please check the version numbers of the software that I mention below.

Let's start with a bit of syntax:

<remote-machine>: the remote machine without Internet access (a high-performance supercomputer, most of the times);
<remote-username>: your username on that machine;
<remote-directory>: a directory containing the yt source, in the remote machine (create it, if not already existing);
<local-machine>: a PC with Internet access (your local PC, for example)
<local-directory>: on your local machine, this is an initially empty directory where you will mount the content of <remote-directory>.

Moreover, every command given below will be preceded by <remote-machine> or <local-machine>, indicating on which machine it should be executed.

First of all, in order to install yt, or to update it by using Mercurial on a machine with no Internet access, you have to remotely mount the source directory in your local PC. 
So, in order to remotely mount <remote-directory> on <local-machine>, the command is:

(<local-machine>) sshfs -o workaround=rename <remote-username>@<remote-machine>:<remote-directory> <local-directory> 

After this command, on <local-directory> you will find the content of <remote-directory>. After working with it, you can unmount with 

(<local-machine>) fusermount -u <local-directory>

Beware: both sshfs and fusermount have to be installed on <local-machine>, and in order to use them you should be inserted in some special user group (I don't remember which one exactly, but if you ask your local administrator it will be fixed for you).

Now, the yt part: I attach here a modified install script. The most outstanding changes with respect to the standard script are: 

- two paths for DEST_DIR, one for <local-machine> and one for <remote-machine>. You must switch between the two of them, because the script will be executed partly on <local-machine>, and partly on <remote-machine>.

- INST_HG=0 : you don't need Mercurial on <remote-machine>, but you have it already on <local-machine>. Beware: it's somehow funny, but although you don't need  hg on <remote-machine>, it should be installed there. If not, please substite the lines 803-810 with the following ones:

    # We assume that hg can be found in the path.
#    if type -P hg &>/dev/null
#    then
        export HG_EXEC=hg
#    else
#        echo "Cannot find mercurial.  Please set INST_HG=1."
#        do_exit
#    fi

which simply disable a check that would raise an error.

- Comment the installation of pip (l. 844-845: it does not work on SuperMUC, with this installation procedure)

- Installation of distribute: before starting the script, uncomment the lines 841-842. After the install script reaches that point, comment them again. 

- l. 882 to 885: I moved there the installation of some required software, not included in the initial download step. 

Here is finally the installation procedure:

1) (<local-machine>) Perform the remote mount, as described above.

2) (<local-machine>) Create a directory called yt-... and copy there the modified installation script. Make the script executable. 

3) (<local-machine>) Execute the script, up to the point where the sources are downloaded. The file distribute-0.6.32.tar.gz is not taken together with the other ones, It must be added to src manually (important!). Moreover, you need two additional source files copied in src: pyparsing-1.5.6.tar.gz and python-dateutil-1.5.tar.gz (look for them on the Web). 

4) (<remote-machine>) Execute again the install script, which now compiles the tools until Python and then fails. 

5) (<local-machine>) For going on, the script executes the hg-related commands for the yt source. After this part it crashes, and I go again to <remote-machine>. 

6) (<remote-machine>) It will try to update yt from here, failing. Stop again and do it from <local-machine>, then go back to (<remote-machine>. 

7) (<remote-machine>) Go on with the script, which (hopefully!) now proceeds to the end without further problems.


More information about the yt-users mailing list