From Wikimedia Commons

Connecting remote instances via SSH

Intro

PyTorch Scholarship Challenge from Facebook via Udacity was an amazing course to learn how neural nets are built and work. For the final project, the neural net was trained to recognize some flowers. After training was finished, VGG19 model was generated and since it had many features, the file size was quite noticeable - 492.8 MB.

Problem

The problem popped out when I tried to copy it to a local computer. Speed was painfully slow (~75Kb/s). It took ~1h 30m to download it! Since I didn’t expect this but needed the file, I had to pay for 1,5 hour of expensive GPU instance just to copy it, not for actually using the GPU.

Later on, I’ve checked the situation again, and it appeared that the problem was with the routing to this particular instance. I mean not with my internet connection :)

Solution

I’d like to give credits to fantastic support from Exoscale company, where GPU instances are hosted. They have replied to my help request very quickly with the testing command I had to execute on my local computer:

mtr -4rzw IP_ADDRESS

Here IP_ADDRESS was the actual IP address of my GPU (remote) instance. mtr is a very powerful utility to diagnose your network connections. You can read more about it, including typical problems, here. You’ll probably need to install it first and for macOS, use brew. If you don’t have brew installed here you will find very detailed instruction on how to do it. If you have it, then:

brew install mtr

After successful installation, execute the command and you’ll need sudo privileges for it:

sudo mtr -4rzw IP_ADDRESS

The output:

Start: 2018–12–30T21:11:29+0100
HOST: MBP15.local          Loss% Snt Last Avg Best Wrst StDev
1.  AS??? IP_ADDRESS        0.0% 10  1.0  1.3  0.8  2.1 0.4
2.  AS6855 IP_ADDRESS       0.0% 10  2.6  2.5  1.9  3.2 0.3
3.  AS??? ???              100.0 10  0.0  0.0  0.0  0.0 0.0
4.  AS??? ???              100.0 10  0.0  0.0  0.0  0.0 0.0
5.  AS3320 IP_ADDRESS       0.0% 10  4.5  4.4  4.1  4.6 0.2
6.  AS3320 IP_ADDRESS       0.0% 10 22.8 23.2 21.1 24.1 0.9
7.  AS3320 IP_ADDRESS       0.0% 10 22.6 23.4 21.6 24.4 0.9
8.  AS3320 IP_ADDRESS       0.0% 10 20.8 22.5 20.6 26.4 2.3
9.  AS13030 IP_ADDRESS      0.0% 10 26.3 26.4 25.9 28.1 0.6
10. AS13030 IP_ADDRESS     20.0% 10 38.3 38.4 38.3 38.6 0.2
11. AS??? IP_ADDRESS       10.0% 10 35.5 35.5 35.3 35.7 0.1
12. AS61098 GPU_INSTANCE   10.0% 10 35.8 36.4 35.4 42.4 2.3

There are clearly some problems (look at 100% losses at hop 3 and 4) and to put a long story short, it was between some big ISPs who couldn’t agree on how to transfer the traffic between networks. So there was not much neither Exoscale’s tech support nor I could do about it at that time :( There had to be a workaround.

The idea was to copy the model to another very cheap Exoscale instance first, then shutdown GPU instance and finally copy the model file from the cheap instance to a local computer. So the task was to connect from Remote Host 1 (RH1) instance to Remote Host 2 (RH2) via SSH.

GPU instance aka RH2 ← → CPU instance aka RH1 ← → Local

This is how you generate public-private key pair to connect RH1 to RH2. After logging from Local into to RH1 in the terminal window:

On the RH1

1. Generate a key pair without the password
   user@rh1:~$ ssh-keygen -t rsa -N ‘’ -f key_file_name.rsa
2. Display the content of the PUBLIC key file
   user@rh1:~$ cat ~/key_file_name.rsa.pub
3. Select all displayed characters and press Ctrl+C to copy the PUBLIC key.

On the RH2

1. Change permission to the folder
   user@rh2:~$ chmod 700 ~/.ssh
2. Change permission to the file, containing authorized keys
   user@rh2:~$ chmod 600 ~/.ssh/authorized_keys
3. Open the file for editing. It may already contain some other PUBLIC keys generated earlier.
   user@rh2:~$ nano ~/.ssh/authorized_keys
4. Paste the copied PUBLIC key.
5. Press Ctrl + V to paste the PRIVATE key
6. Press Ctrl + X, y, Enter to save and exit with the same file name
7. Exit from RH2 instance.
   user@rh2:~$ exit

That’s it! Now you can connect from RH1 to RH2 and copy files over SSH using scp command.

To copy a file from RH1 to RH2 (command below is one line, not two!):
scp -i keyfilename.rsa /home/user/file.name user@ip_address_of_rh2:/home/user/file.name

To copy a file from RH2 to RH1 (command below is one line, not two!):
scp -i keyfilename.rsa user@ip_address_of_rh2:/home/user/file.name /home/user/file.name

Conclusion

This article serves as a small reminder of steps and commands to diagnose connection problems and also to establish a secure connection between two remote instances if it would be required in the future. Hopefully, it may help someone else.

Thank you for reading. Please, hit the clap button if it’s ok for you. If you have any questions, please, feel free to comment.

About Denis Rasulev: 28 years working in IT industry, mostly in Sales and Business Development. From 2015 focusing efforts on various projects in Data Science field.