manoel horta ribeiro

A Concise Guide for a Nice Jupyter Setup in a Remote Linux Server

By the end of this guide you’ll have a nice set-up in a remote server with:

  • convenient ssh settings;
  • stable conda settings;
  • a great jupyter setup;

My sys-admin skills are by now means exemplary, but this got me a pretty neat setup all around…

Step 1: ssh settings

First things first. Lets set up a way of easily logging-in the remote serve through the command line!

Step 1a: Create your ssh key

This can be done by running:

ssh-keygen -t rsa -b 4096 -C "your_email@example.com" -f $HOME/.ssh/rsa.pub

Then you need to add your key to all remote machines needed:

ssh-copy-id -i ~/.ssh/rsa.pub REMOTE

Once the key has been installed, test it with:

ssh -i ~/.ssh/rsa.pub REMOTE

The purpose of this step is so that you don’t need to constantly log in into the machine! With the key, you never need to write your password. Notice that someone needs to have created an account for you in the remote machine. To make this run extra smooth, we still need one more thing…

Step 1b: Edit your .ssh/config

You can bypass passwords by making entries for each remote machine in your config file. A standard entry looks like this:

host dcc  #%% Gateway
    HostName login.dcc.ufmg.br
    User username
    IdentityFile ~/.ssh/rsa.pub
    ForwardAgent yes
    AddKeysToAgent yes

With this, you won’t need to provide your key when trying to log in:

ssh -i ~/.ssh/rsa.pub REMOTE

you can simply do:

ssh REMOTE

But also, you can use a easier alias instead of remote. Here the alias declared was dcc. This means you can do:

ssh dcc

And you’ll be in the machine! If your machine is accessible from anywhere, these steps will suffice. However, sometimes you gotta hop-through-machines to do log-in into a intra-network. To do so we’ll need to go even further!

Step 1c: Add proxy-jumps (if needed!)

Sometimes you need to add multiple “ssh”-jumps. To do so then you need to edit your .ssh/config file. When I studied at UFMG, I had to do a 3-way hop. I had to log-in first in login.dcc.ufmg.br; then in cerberus.speed; and then in one of the machines, hydra{number}. To make these hops happen naturally once you need to add:

host dcc  #%% Gateway
    HostName login.dcc.ufmg.br
    User username
    IdentityFile ~/.ssh/rsa.pub
    ForwardAgent yes
    AddKeysToAgent yes
     
host cerberus  #%% Gateway
    HostName cerberus.speed
    User username
    ProxyJump dcc
    IdentityFile ~/.ssh/rsa.pub
    ForwardAgent yes
    AddKeysToAgent yes

host hydra3
    User username
    ProxyJump cerberus
    IdentityFile ~/.ssh/rsa.pub
    ForwardAgent yes
    AddKeysToAgent yes

Step 2: Installing conda

Anaconda is a package manager which allows you to create virtual environments. It is very convenient to use it in remote machines because it lets you install things locally, without sudo. Also, it lets you have several different virtual environments, which can have different python versions, etc.

To install conda you must download a bash script from their website. I suggest you follow the instructions they provide directly. Remember to log in and out so that conda is loaded in your path after the installation.

There are several things you can do with conda. The most basic one is to create environments and move through them.

To create an environment you can use:

conda create -n envname python=3.7

To activate it you can use

source activate envname

Once you activate an environment you can use pip inside that environment to install packages. Sometimes you can also use conda to install packages, specially if its something non-pythonic. It is quite handy if you don’t have sudo rights.

Step 3: Installing and setting jupyter

To install jupyter, go to your base conda environment (the one logged in by default) and run:

pip install jupyter

I suggest that you install jupyter in your base env. Then after that, you need to manually add each env to jupyter. To do so, enter the environment, install ipykernel and then add it to jupyter

source activate envname
conda install -c anaconda ipykernel
python -m ipykernel install --user --name=envname

Then go back to the base environment:

conda activate

Now we will set up some jupyter stuff! We begin by generating a password by running:

python3 -c "from notebook.auth import passwd; print(passwd())"

Which will output something like:

sha1:0686104888f6:79c7e12f64925ceb0e859aa23934e41798ba7da8

Now we’ll add our password in jupyter. Run:

jupyter notebook --generate-config

Which will output something like:

Writing default config to: ~/.jupyter/jupyter_notebook_config.py

Then add edit file, adding the following lines to the top:

c.NotebookApp.ip = '*'
c.NotebookApp.open_browser = False
c.NotebookApp.password = 'sha1:0686104888f6:79c7e12f64925ceb0e859aa23934e41798ba7da8'
c.NotebookApp.port = 1234

This will make jupyter run on port 1234 (you may need to change this!), and will ask for your password! Now create a new screen and run jupyter

screen -S jupyter
(wait for new screen)
jupyter notebook
(press ctrl A + ctrl D to detach) 

Now log off! Back to your machine! You can access your jupyter notebook:

ssh -fNT -L 1234:localhost:1234 REMOTE;

Where host is the server alias that you edited in your config! After you do this, you’ll be able to simply go to localhost:1234 and find your server notebook running there!

Written on July 16, 2020