Nectar

Module 7

Launching and Connecting

Introduction

In this module we are going to learn how to launch an instance and how to connect to it. We are going to discuss two ways to connect: (a) with the command line and (b) to a remote desktop, with the benefits of having a graphical interface. We will briefly take a look at how to run graphical applications via the command line (instead of the remote desktop), and we will also set up a web server to publish a very simple website. Finally, we will set up the different cloud storage types, so we can access the storage remotely and directly from our instance.

Videos

The following videos go through most of the content in this module and offer a less in-depth description of the subject than the documentation does.

https://www.youtube.com/watch?v=Y9JVdyRPlcg

https://www.youtube.com/watch?v=cVX3e_bYee8

https://www.youtube.com/watch?v=eDDVUj1qyC0

https://www.youtube.com/watch?v=H3a9-QNQsZ4

https://www.youtube.com/watch?v=XrlHNvQMTN4

https://www.youtube.com/watch?v=7V8XB6_qR98

General notes

What is an “instance”?

In Module 5 we have already discussed in more detail what an instance is, but let’s do a little recap at this point. In the NeCTAR Research Cloud, an instance is a virtual machine. Instances running inside the Research Cloud are just like real-life computers, except that they run remotely, “in the Cloud”. Here and when learning about them on the web, you will often find them referred to as “instances”.

Instance vs “local computer”

In this document we will often talk about the “local computer” as opposed to the “instance”. The local computer is the one you are probably sitting in front of right now: It is your home or office computer that you use to connect to the instance. The instance is your virtual machine that is running in the NeCTAR cloud.

You may be familiar with connecting to remote computers already. You may have connected to a remote desktop, or used a command line. The same tools and techniques apply when connecting to running instances. Your instance has a public IP address and can be reached and controlled with any remote access tools you wish to use, after you have set it up for remote access. For example, you can use an SSH terminal, or alternatively connect to your remote desktop.

In this module we will create and launch an instance (a virtual machine, short: VM) and go through the two options on how you can connect to your instance (terminal and remote desktop).

Use of Terminals

Throughout the exercises, we will use a Terminal to type commands. To prepare, make sure you know how to open your Terminal by following instructions below for your Operating System.

On Windows, you will use two terminals:

On Mac and Linux, there are built-in terminals available. You can access them as follows:

Conventions

The notation throughout the training documents can be interpreted as follows:

Words in italics are used for names and terminology, e.g. name of a software, or name of a computing concept. It may also just emphasise a word in the traditional way. Quotations are also written in italics and are put in between quotatioin marks.

Words in bold are used to highlight words which identify important concepts of a paragraph, to make it easier for users to skim through the text to find a paragraph which explains a certain idea, concept or technology.

image

Additional information which is optional to read is displayed in info boxes like this one.

image

Important information is displayed in boxes like this one.

image

Definition of terms are displayed in boxes of this style.

image

Possibly specific prerequisites for reading a particular section are contained in this type of box at the beginning of a section.

The use of command line tools is part of this course. In a Terminal, you will be directed to type in commands. Commands are formatted as follows:

command-name argument1 argument2 ... argumentn

Throughout the exercises, when you see a word in pointed brackets, like <this-word>, it means that you have to replace everything inside the brackets, and including the brackets, with whatever is described within the brackets.

For example, if you are prompted to run a command named the-command which takes two arguments:

the-command -f <yourfile>

Then you have to replace the second argument, <yourfile>, with the file name that is being referenced in the exercise. For example

the-command -f thefile.txt

When editing a file, the contents of it will be displayed in a different font and with background colour as follows:

The content of the file
The next line in this file

Output on the command line terminal is printed in boxes formatted as follows:

NectarInstance:~ Jennifer$ whoami
Jennifer

Create a ssh key

First, we will need to create a ssh key pair for connecting to our instances. An ssh key pair is required to encrypt the data which is transferred to communicate with the instance. The pair consists of two keys: A private key, and a public key. The public key is called “public” because that’s the one which is placed on the server and which you may upload online without worrying about security issues. It is useless without its counterpart, the private key. The private key is the secret key which you keep on your local computer so that you may connect to the instance. You should keep the private key very safe and never share it with anyone. Take care not to lose it, otherwise you won’t be able to connect to your machine. The connection to the machine only works with both keys.

image The image above shows two computers connected through their ports, with an ssh server and client encrypting the connection. Applications can use this secure ssh connection to exchange data.

We will first have to generate this key pair before we can proceed with launching an instance.

The ssh keys will have a name. You should choose a name that you can associate with your instance, for example “Nectar_Key”. As a general rule of thumb, file names should consist of alphanumeric characters and underscores ( _ ) only. You should avoid white spaces. They makes working with files on Unix systems a bit clumsy.

Note: The process differs between Unix/Mac Systems and Windows. Follow the steps described below for your platform, and then continue to the next section.

Mac and Linux

SSH key pairs can be generated with the ssh-keygen command.

If you choose Nectar_Key as your key name, you will have two files after the generation of keys: The private key, called Nectar_Key and the public key, called Nectar_Key.pub.

  1. Open your Unix or Mac Terminal. You will start off in your “home” directory. If you already had the terminal open before, make sure you are in the home directory, by simply typing
    cd
    Now, check if you already have a directory called .ssh, by typing
    ls -a
    A list of files in your home directory will be printed. If you don’t see the name .ssh in this list, create the directory (if you see the directory, skip this command):
    mkdir .ssh
    and change to this directory:
    cd .ssh
    Now generate the key with
    ssh-keygen -t rsa -f Nectar_Key
    This generates a pair of keys in the directory .ssh, which is the default location for ssh keys.
    You will be asked to specify a passphrase. Choose a secure passphrase! But make sure that you store your password somewhere safe - you will not be able to connect to your instance if you forget the password!

  2. You will have generated (1) a key you keep private, named Nectar_Key and (2) a public key, named Nectar_Key.pub. If you type
    ls
    Then you should see both files listed.

  3. Open the public key file with your favourite text editor. If you don’t have a favourite editor, here are some suggestions:
    On a Mac, type
    open -e Nectar_Key.pub
    which should open the key in TextEdit.
    On a Linux gnome environment (e.g. Unity), type
    gedit Nectar_Key
    or if you are using KDE, type
    kate Nectar_Key
    If you are using another desktop environment we assume you are familiar with your favourite text editor.

  4. Copy all the text which is contained in the file Nectar_Key.pub into the clipboard. With CTRL+A you can mark all text, and with CTRL+C copy it (on a Mac it is Command (⌘) instead of CTRL). You will need the contents of the public key on the keyboard in one of the next steps.

Windows

Open the PuTTYgen program by clicking on PuTTYgen.exe which you downloaded in the beginning of this tutorial.

image

  1. For Type of key to generate, select SSH-2 RSA.

  2. Click on Generate.

  3. Move your mouse in the area below the progress bar. When the progress bar is full, PuTTYgen generates your key pair.

image

  1. Type a passphrase in the Key passphrase field. Choose a secure passphrase! Type the same passphrase in the Confirm passphrase field.

  2. Right-click in the text field “Public key for pasting into OpenSSH authorized_keys file” and choose Select All. Right-click again in the same text field and choose Copy. This copies the content onto the clipboard, we will need to paste it somewhere else in one of the next steps.

  3. Click the Save private key button to save the private key. It is important that you save the key, because you will need the file later. So save it in a folder where you can find it again. In this exercise, we will name the file Nectar_Key.ppk.

  4. Save your public key as well by clicking the Save public key button. As a file name, type Nectar_Key.pub.txt.

Launching an instance

We will now create an instance and choose the resources we require. For this tutorial, we don’t need a lot of resources. You may require more for your research. For more details about how to request an allocation, see Module 5.

Security note

Before we start, a brief note about security: It is important that we make our instance secure from the very moment we create it. In order to enable the right security settings, we will select the instance to be part of one or several “security groups”.

image

A Security Group opens up access to the instance from outside. Essentially, a security group is a collection of firewall rules, which open up different Ports to access the machine from outside.

Module 5 has already introduced the concept of Ports. Now we need to free up some Ports in order to be able to communicate with our instance. For example, we will first connect to our instance with a SSH shell from our terminal (Reminder: SSH encrypts our communication with the server). SSH uses Port number 22, so this has to be freed up.

image

Note: All the pre-defined security groups mentioned in the instructions are available in Project Trials only. To learn how to create your own security group (or re-create the trial groups), refer to the Appendix of this module.

The following steps will explain how to create an instance and how to make sure it has been assigned to the security groups we need to connect to it.

Step 1: Logging on

First, we will have to log on to the Dashboard, which is our interface for managing our virtual machines. The Dashboard can be accessed with any browser, e.g. Firefox, Safari or Windows Internet Explorer. Open your browser of choice (we suggest Firefox), and go to:

https://dashboard.rc.nectar.org.au/

image

Click “Log in” on the first window. Then, select your organisation from the drop-down menu (you may also type in the first characters of your organisation name in order to find it quicker). You will be redirected to a page where you can type in your username and password. This should in most cases be the username and password you usually use to access your account at your University or Research Organisation.

Click OK for all terms and conditions if logging in the first time (you will not have to do this every time).

Step 2: Open the Launch Dialogue

Go to Dashboard > Instances. A list of your running instances will be listed. If this is the first time you are doing this, the list will be empty. Now click on Launch instance on the top right of the overview.

image

Step 3: Selecting resources

image

Choose the name of your instance and the “flavor”. Different flavors allocate different amounts of memory and disk space for you. Select the flavors in the drop-down box and information about the resources are displayed on the right side of the window. You may select a few flavors to see the differences between the flavors. For this tutorial, choose the flavorm1.small”, which offers 30GB of ephemeral storage (we will need this in a later section.

You could launch several instances of the same flavor, but for now we will launch only 1, so leave “Instance Count” at 1. Leave “Instance Boot Source” on “Boot from Image”. Choose the most recent version of “NeCTAR Ubuntu” from the “Image name” list.

image

NOTE: At the time of writing the image “NeCTAR Ubuntu 14.10 (Utopic) amd64” was used. If you choose a version much newer than this, there may be some differences to the descriptions given in this tutorial. In general, a newer version should work fine with the exercises below, you may only see different text (phrasing) in the terminals command line output.

Don’t click “Launch” just yet… we have to set the security settings first!

Step 4: Ensuring security

Now, we will choose the security settings. Go to the tab “Access & Security”. We can choose which Security Groups the instance will belong to (see above for explanation about security groups). We will connect to the instance with ssh. Ssh uses port number 22, so this has to be freed up. Therefore, tick the “ssh” box. We will also publish a little website in a later exercise, which requires http, so tick the “http” box also. To test whether your instance is reachable, we will also need “icmp”.
Note: We can also change the security groups of the instance later on, but it’s convenient to do this now.

Next, we will specify the key for connecting to this instance (that’s the one we just generated).

image

  1. You should have the contents of your public key still in your clipboard, unless you have copied and pasted something else since we did this above. If you have, copy the contents of Nectar_Key.pub again.

  2. Click on the plus (“+”) to add a new key.

  3. Specify the Key Pair Name. This can be any name, but for simplicity choose Nectar_Key also.

  4. Paste the contents into the large text box labelled Public Key (use CTRL+V, or right click in the box and “Paste”).

  5. Now, click “Import key pair”. This key will now be available for future use. Next time you launch an instance you can select this key from the drop-down list directly, without having to copy and paste it again. The window closes and you are back to the window to configure your instance.

Step 5 (optional): Choose your Availability zone

image If you need access to specific facilities (e.g. Volume storage), you can manually choose the location of your instance in the “Availability Zone” tab. The Availability Zone is where your instance will be running physically. Unless you know that you need access to specific services, it is recommended to leave this in the default setting, because then the location with the most available resources is chosen.

Step 5: Launching the instance

You can now launch the instance by clicking on the “Launch” button. You will get back to the overview of your instances. You will see the status of your instance in the overview. You need to wait a few minutes.

The Instance will normally pass from Status “Build” to “Active”. You will see an IP address for your instance (4 numbers separated by a dot). Write it down, or keep this window open so you can see it, as we will need it in the next step to connect to your instance.

Congratulations!!

Your instance is now reachable on this IP address. Your applications can connect to the Ports that you have allowed earlier in the Security Groups.

Connect to the instance

To connect to your instance, you will need its IP address. You should have just written this down, or you can look it up on the Dashboard (Compute > Instances).

In the following text, your IP address is described as NNN.NNN.NNN.NNN, which you will have to replace with your own instance’s IP address.

To get a secure connection, we need to log on to the instance via ssh.

Connect via ssh

Step 1: Test connectivity

First, let’s do a little test to see if your instance is reachable. On Linux/Mac, you can use the command line shell that you have just opened.
On Windows, open your Windows command line.

Use the terminal to try to “ping” your instance (checking that you can reach it):

ping NNN.NNN.NNN.NNN

replacing the N’s with the IP address of your NeCTAR instance you noted above.

The ping command just sends a simple signal to your instance and checks whether it responds. If the ping does not work, then most likely neither will any of the following steps. The ping has worked if you get a response every few seconds, e.g.

PING 144.6.227.170     
(144.6.227.170) 56(84) bytes of data.    
64 bytes from 144.6.227.170: icmp_seq=1 ttl=49 time=354 ms    
64 bytes from 144.6.227.170: icmp_seq=2 ttl=49 time=358 ms

Step 2: Logging in

We will now log in to our instance using ssh with the private key (the username might be different depending on the image you launched).

Follow the instructions below for the operating system of your local computer.

Note: If the ping has worked but the SSH connection does not work after following instructions below, follow the troubleshooting steps in the Appendix.

Linux/Mac

We will need a command line shell (a “Terminal”) in this exercise. On Linux and Mac, you have a built-in terminal. Open it now (if you are unsure how, refer to the notes in the introduction).

Go to your command line and type

ssh -i Nectar_Key ubuntu@NNN.NNN.NNN.NNN

using your own IP address instead of NNN.NNN.NNN.NNN. The command line will ask you to establish the authenticity of the server. Type “yes”.

You will then be asked for your passphrase (that’s the one you chose when generating the SSH Key).

Note: If you named your key other than Nectar_Key, you should replace this with your own private key file name here. If you saved the private key in another folder, you will have to specify the full path to it (e.g. ssh -i /home/john/yourkeyfolder/Nectar_Key NNN.NNN.NNN.NNN).

On a Mac:

You may have to specify the full path to the key. If the above does not work, try instead:

ssh -i $HOME/.ssh/Nectar_Key ubuntu@NNN.NNN.NNN.NNN

image

Our instance has a default username that we need to specify when we log in. In this tutorial, we chose a Ubuntu Image when launching the instance. At the time of writing, the user name for Ubuntu is “ubuntu”. Other NeCTAR images may expect different usernames.

Try using root instead of ubuntu (“root@NNN.NNN.NNN.NNN”) for other Operating Systems. For Fedora, the user name may be ec2-user. The right user name for the image you chose is documented at the NeCTAR Image_Catalog.

Well done!

Now, you should be connected to your instance and your ssh terminal command line is ready to use.

Windows

On a windows system, we will use the PuTTY terminal to connect to the instance. We first have to set up our PuTTY session to include the details of our instance, including the keys we are using.

Double-click on the file “PuTTY.exe” which you downloaded when you created the SSH key.

In your PuTTY Window, type your instance’s IP address into the “Host” field (you should have taken a note about the IP address earlier, if not you can find it on the Dashboard: Go to Compute > Instances).

image

Then, select the “Data” category in the navigation window. Here, we will have to specify the username. Choose “ubuntu” as your username. Leave the other fields on default settings.

image

image

Our instance has a default username that we need to specify when we log in. In this tutorial, we chose a Ubuntu Image when launching the instance. At the time of writing, the user name for Ubuntu is “ubuntu”. Other NeCTAR images may expect different usernames.

Try using root instead of ubuntu for other Operating Systems. For Fedora, the user name may be ec2-user. The right user name for the image you chose is documented at the NeCTAR Image_Catalog.

Next, click on the “SSH > Auth” sub-category. There is a field labelled “Private key file for authentication”. Click on the “Browse” button:

image

In the browse window that comes up, select the private SSH key file that you saved on your computer earlier (we named it Nectar_Key)

image

Note: If you have generated your ssh key on a Linux system and now need to convert it to Windows, see how to do this in the Appendix.

Instead of converting keys, you can also simply create a separate set of keys to use with PuTTY. In this case, you have to add the new key as an authorized key on the instance. See the Appendix for instructions on how to do this.

Now, return to the “Session” screen in the left navigation window. This is the screen we started at. We will save our settings (the “session”), so that in future we don’t need to type in all the details again. In the “Saved Sessions” section, choose any name for this session so that you can remember it, for example “MyNeCTARInstance”. Then, click on the “Save” button.

image

Now, connect to the instance by clicking the “Open” button. The first time that you connect, you will be asked to verify the identity of the remote server. This is expected the first time you connect to a new server, so you can select “Yes” to continue. You will be prompted for a passphrase. Type in the passphrase that you used to create the SSH key earlier on.

image

Every time you connect to the instance via Putty, you will have to type in your passphrase. This may get a little cumbersome, so note that there is a tool called Pageant that you can get on the same page where you downloaded PuTTY. Pageant is an SSH authentication agent. It holds your private keys in memory, already decoded, so that you can use them often without needing to type the passphrase every time. It also eliminates the need to explicitly specify the relevant key to each user account, if you use more than one account to log into a server.

We will not use pageant in this tutorial, but you may be interested to read up on how to use it. You can find instruction in the Appendix.

Well done!

Now, a black Terminal window should open and your command line is ready to use.

Configuration and control

Welcome to your new virtual machine!

First of all, we want to make sure our system is updated. On Debian and its derivatives, like Ubuntu, you can update your packages with the commands:

sudo apt-get update

sudo apt-get upgrade

The first command refreshes the package information, and the second upgrades your system.

image

There are several ways to upgrade your system. The command above (sudo apt-get upgrade) simply updates the existing packages. You can also update more intelligently. Here’s a description of the various upgrade commands (extracts from the apt-get manual):

  • apt-get upgrade: upgrade is used to install the newest versions of all packages currently installed on the system [..]. Packages currently installed with new versions available are retrieved and upgraded; under no circumstances are currently installed packages removed, or packages not already installed retrieved and installed. New versions of currently installed packages that cannot be upgraded without changing the install status of another package will be left at their current version.

  • apt-get dist-upgrade: dist-upgrade in addition to performing the function of upgrade, also intelligently handles changing dependencies with new versions of packages; apt-get has a “smart” conflict resolution system, and it will attempt to upgrade the most important packages at the expense of less important ones if necessary. So, dist-upgrade command may remove some packages.

  • apt-get full-upgrade: full-upgrade performs the function of upgrade but may also remove installed packages if that is required in order to resolve a package conflict.

You may also want to change the password for the user ubuntu:

passwd

Note: You will not be prompted for your ubuntu user password when you connect through ssh. This is because in the default configuration, the ssh key is regarded sufficient for authentication purposes. Therefore, each user on the virtual machine should have their own ssh key.

More importantly, you should change the password for root access (administrator rights), especially if you are planning to add more users to the virtual machine:

sudo passwd root

and type in the new password twice.

image

If you do not change the root password, and later add other users to the VM, any user will be able to log in as root and perform administrator tasks (using the command su -).

When you create new user accounts, and you do not want them to have root access, make sure you do not include them in the user group sudo. Any user which is member of this group can execute administration tasks.

Having problems connecting? The Appendix includes a section on troubleshooting.

Using a command line text file editor

Sometimes, we will have to edit files on the instance via the command line. Because this is a command line, we cannot use programs like Word or iNote to edit a file. Instead, we have to use a command line editor.

There is a large number of command line editors available. A simple and easy-to-use editor is nano. Vim is also popular but a bit more difficult to use for beginners.

If you want to edit a file, type

nano <path-to-your-filename>

The terminal changes into a simple editor. You can edit the file and then exit the editor with Ctrl+X. A prompt will come up asking if you want to save the file, type ‘Y’ to confirm this.

We will create a file to test this. Type

nano MyFile.txt

and hit enter to confirm the command. Write a little story or anything you are just thinking of into the file. Then, hit Ctrl+X to exit, and confirm with “Y” that you want to save the file. Now, type

ls

and hit enter to confirm the command. You will see the list of files that are there. The list should contain a file “MyFile.txt”. You may print the contents of the text file onto the terminal by typing:

cat MyFile.txt

and you should be able to read your message in the terminal. Exit again with Ctrl+X.

Restarting your instance

Sometimes, you may want to restart your instance. For example, to make sure everything works correctly after you installed some packages. You can do this in the Dashboard.

Step 1. Reboot on the Dashboard.

Go to the Dashboard and click on Instances on the navigation panel on the left to see your instance. On the right side of your instance, you can open up a drop-down box.

image

click on “Soft Reboot Instance” (a soft reboot is what we know as a reboot. A hard reboot forces the reboot in case the soft reboot does not react).

Step 2. Re-connect ssh client.

Your ssh session, or the PuTTY session if you are on Windows, will have disconnected now, so you have to re-connect. On Windows PuTTY, you should have your login session still saved so you just need to load these settings up. On Mac and Linux, simply re-run the ssh command (Tip: you can browse the command line history with the “up/down” arrow keys in your ssh terminal. Your ssh command should be in there so you don’t have to type it again).

Tip for Unix/Mac

You can make your life a bit easier by saving your configuration for the ssh connection. This will avoid always having to type your IP address. There is two options to do this: a) specifying all details or b) creating an alias for your IP address. Then, instead of typing in the full ssh command every time, you will just do the following:

Option 1: ssh MyNectarInstance

Option 2: ssh -i Nectar_Key ubuntu@MyNectarInstance

Option 1

You need to edit a file on your local computer (not on the instance!).

Open a new terminal on your computer. Then, open up the file ~/.ssh/config using your favourite editor, for example:

nano ~/.ssh/config

and add the following contents:

Host MyNectarInstance   
HostName NNN.NNN.NNN.NNN   
User ubuntu   
IdentityFile ~/.ssh/Nectar_Key

Again, replacing the N’s with your IP address.
You can choose any other alias name instead of MyNectarInstance. Save the file and close it.

From now on, you can connect with the command:

ssh MyNectarInstance

or, if you chose another name, replace MyNectarInstance with it.

Option 2

You may also just alias your IP address in the system’s host file.

On Unix systems and on Mac 10.2 and newer, the file to edit is usually the file /etc/hosts. On Mac 9 and earlier you can find it in the Preferences or Systems folder.

Open this file in your favourite editor, for example:

sudo nano /etc/hosts

and add the following line:

NNN.NNN.NNN.NNN   MyNectarInstance

replacing the N’s by the IP address of your instance. You may also choose another name instead of MyNectarInstance. Next time you ssh into the instance, you can connect with:

ssh -i <path-to-your-keyfile> ubuntu@MyNectarInstance

Note this is the same command we used earlier, but you can type MyNectarInstance instead of your IP address. You will be asked to accept the authenticity of the host again. Answer with “yes”.

Connect to a remote desktop

Using the command line may not feel very intuitive, especially for beginners. Aside from that, sometimes we need to run software that has a graphical interface.

As an alternative to connecting with a ssh terminal, we can also connect to the desktop of our instance and get a graphical interface as we know it from our own computers. A remote desktop login allows users to connect to their desktop while being physically away from their computer.

You can also connect to your own home computer’s desktop while you are away, if you set up your home computer accordingly with remote access. This may be a bit difficult to do from your organisation or from your home, as you need access to the computer via the internet from outside, which may require the permission of firewalls used on the premises, and you need a fixed IP address.

However on your NeCTAR instance, it is easy to set up a remote desktop, because the instance is already accessible from outside.

image

Some remote desktop solutions allow only one user to see the desktop at the same time. Other software also allows desktop sharing in which several users can see a desktop simultaneously. Such a shared desktop can be used as a platform for real-time collaboration. This mostly requires a separate setup, which is not part of this tutorial.

To connect to a remote desktop, the remote machine (your instance) has to have a remote desktop server installed and running. On your home or office computer, you have to install a remote desktop client (sometimes called “viewer”) which connects to the remote desktop server on your instance. The client then displays the desktop of your remote machine. It is like a “window” into your instances desktop; The desktop environment itself is running on the instance.

Terminals on remote desktop vs SSH terminal

We have previously connected to the instance with via a ssh terminal. When we get a window into the desktop of our instance, we can open a terminal on the desktop as well.

The image below shows both (1) a ssh terminal (left) and (2) a desktop with an open terminal (right):

image

So what is the difference between the two terminals? When you type commands in either of them, it will have the same effect. They are both terminals that you can use to control your instance. However, the difference is that

  1. The ssh terminal is set up through a secure ssh communication line with your instance. If we use a ssh terminal, only text is being transferred (the commands you type and the responses you get). This is very fast, but you cannot run graphical applications, because you can only communicate with the instance via text.
    Note: it actually is possible to run graphical applications via the terminal as well, with some disadvantages. This is discussed later in this module, but this is very slow and usually not recommended.

  2. The terminal on the remote desktop is connected securely as well (at least the way we will set it up in this tutorial). But instead, the communication includes graphical data and events (e.g. your mouse clicks, and the windows displayed). This is slower than using the terminal only, but you may find it is still fast enough to work with. You can run graphical applications and use the OS menu to find programs and change settings, which may be easier for many of us.

Remote desktop protocols

When you are connected to a remote desktop, everything you do to interact with the system, for example clicking the mouse, moving a window or typing on your keyboard, is communicated to the remote machine (in this case, your instance). For this kind of communication, computers use “protocols” which are designed for this purpose.

The type of protocol determines properties of the remote desktop, for example how secure it is, how fast it works and what kind of desktop environments it supports. In this tutorial we will use two different protocols to connect to our remote desktop: NX and VNC.

image

Some wide-spread protocols include:

  • RDP (Remote Desktop Protocol)
    • Originally developed by Microsoft to connect to Windows machines.
    • The protocol is not very secure, therefore setting up “tunneling” through a secure connection is required.
  • NX
    • NX compresses the data to minimize the amount of information transmitted. NX also caches data to make the session as responsive as possible. For example the first time a menu is opened it may take a few seconds, but on each subsequent opening the menu will appear almost instantly. Therefore, NX allows fast speed even with slow internet connections.
    • NX uses the SSH protocol to send its data and is therefore secure.
  • VNC (Virtual Network Computing)
    • The VNC Protocol (called the RFB protocol) is very simple. It is based on graphic primitives from server to client (“Put a rectangle of pixel data at the specified X,Y position”) and event messages from client to server. This makes VNC a bit slow.
    • By default, this is no secure protocol. Therefore, you need to “tunnel” the connection via a secure SSH connection to make it secure. This may seem a bit complicated to achieve for beginners. Alternatively, some clients like UltraVNC) support the use of an encryption plugin which encrypts the entire VNC session.
    • Because VNC is quite popular, the client software options are pretty good for Unix and Windows platforms.
    • VNC allows multiple users to see (share) the same desktop.
    • To transfer files, printing or audio requires separate services. The VNC only handles displays and inputs, such as interaction with keyboard and mouse.
    • Users may access desktops with VNC through their Web browser, without needing to install separate viewer applications. VNC displays a desktop within a browser window by running a Java applet, and this facility should work on any system that has a JVM installed.
    • VNC does not support the newer “3D desktops” (e.g. the newer GNOME/Unity or KDE desktops environments. The desktop environments have to be run in classical 2D mode for it to work (e.g. GNOME Flashback).
    • To learn more about VNC, here is a good article

Free remote desktop client and server software includes:

  • Vnc4server
    • the main VNC server software for Linux.
  • UltraVNC, RealVNC, TightVNC
    • popular clients for VNC sessions
  • X2Go
    • An open source product which is pretty fast.
    • It uses the NX protocol and is operates through a secure SSH connection
    • Supports remote audio, printing, clipboard, file sharing etc.
    • Can share a user’s session remotely, but it has to be set up separately.
    • It is fairly easy to set up
    • There is even a readily available browser plugin to make remotely controlling another PC possible from a web browser. But: Port forwarding and Dynamic DNS are recommended if you’re using this over the Web.
    • Due to the use of older NX libraries, it has problems with the newer “3D Desktops”, e.g. Gnome/Unity and newer KDE desktop environments. Only the older environments are supported.
    • No Android / iOS client available, unless using the web interface
  • NX Server, free edition
    • Easy to set up
    • Not very fast
    • Only free to 3 users
    • Can’t share a user’s session remotely.
  • FreeNX Server
    • A Server for the NX protocol, similar to NX Server, but free.
    • No real good documentation, not easy to install
    • Can’t share a user’s session remotely.
  • NoMachine)
    • A client to connect to an NX server (e.g. FreeNX).
    • Not free for commercial use (only private)
    • Simultaneous connections from only 2 users.
  • NeatX (outdated)
    • Google has released NeatX, a complete independent implementation of the NX protocol.
    • Neatx was developed as part of an internal project which has now finished. It had no releases and is not being actively developed.
    • Can’t share a user’s session remotely.
  • xrdp
    • uses the RDP protocol (which is not very secure)
    • Servers are also available for Unix and Mac.

Commercial products may be better and easier to set up than the free software listed above, but they come at a cost (at least for commercial use). Popular software includes:

  • TeamViewer
    • supports Windows, OS X, Linux, Android, and iOS
    • free for personal use
    • setup very easy
    • pretty expensive for commercial / business use
  • Splashtop
    • supports Windows, OS X, Linux, Android, and iOS
    • allows you to stream audio and video across computers with minimal latency
    • very easy to use and set up
  • RealVNC
    • The “Official” VNC software
    • Offers its client and server apps for Windows, OS X, Linux, Android, iOS, Chrome
    • free for basic individual private use, but added features and support come at an extra cost.

Wikipedia offers a good comparison of remote desktop software.

The options which are easiest to set up and provide a fairly good user experience are X2GO (using the NX protocol) and VNC. In the following exercises, we will see how we can set up both of these options using freely available software.

VNC vs X2Go

X2Go uses the NX protocol, and VNC the RFB protocol. This makes for different properties and advantages of each approach.

The VNC protocol is pixel based. This can make it a bit slow. One advantage is that it is widely used, and several available remote desktop products use this network protocol. This means that you may use any VNC viewer (client) to connect to a system that offers remote access with the VNC protocol.

The NX protocol (used by X2Go) has a better understanding of the underlying graphic layout than VNC (which is pixel-based). Therefore, it is often more efficient (faster).

In summary:

VNC:

NX (X2Go):

Install a desktop environment

First of all, for either of the options (VNC or X2Go), we need to install a desktop environment on our instance (Remember: The desktop actually runs on the instance, you only get an interactive window to it on your local computer. So we need to install a desktop on your instance).

image

A desktop environment is what we use to graphically interact with a system (it provides the desktop, windows that open up and symbols you can click on to start a program). Windows and Mac OS have only one desktop environment, but in most Unix-based systems you can choose from several environments. Popular choices are Unity, Gnome and KDE. There are also popular simple and light-weight environments called LXDE, XFCE, Mate and more.

You may install several desktop environments, but you only need to choose one to start with. The following desktop environments are being discussed in this section:

If you are unsure which desktop environment to choose, we recommend you start with LXDE.

image

Unfortunately, there is no remote desktop solution that supports all desktop environments. The desktop environments which are currently not supported for remote connections are the ones using 3D desktop acceleration. This affects e.g. Unity and Gnome3. All traditional remote desktop approaches (VNC, NX etc.) fail to work with this. Some versions of Gnome can be run in “flashback” Mode (with the 3D acceleration switched off), but that can be a bit tricky to set up and does not necessarily work. We will therefore discuss only the desktop environments which do work.

image

Note: This tutorial is assuming you are using Ubuntu 14.10 (UTopic), however later versions may also work with the same instructions. When in doubt, links to external documentation are given at the relevant places.

LXDE

This light-weight environment is easy to install and works well for remote connections.

image

Install the LXDE environment on your instance by typing into your ssh terminal:

sudo apt-get install lxde

XFCE

Install the XFCE environment by typing into your ssh terminal

sudo apt-get install xfce4

image

GNOME

Gnome will have to be run in Flashback mode to disable the 3D acceleration which is not supported in VNC or X2Go (see for example this official documentation and this one on ubuntuusers.de). X2Go is only compatible with Gnome versions earlier than 3.12 (see x2Go documentation), and ideally earlier than 3.08. The Ubuntu version we use in this tutorial has Gnome 3.12.2, so you cannot use Gnome with X2Go with the Ubuntu version used in this tutorial. However, you may still get Gnome working with VNC by following the instructions below.

image

You can find out your Gnome version (after you installed it with following instructions) by typing in your ssh command line:
gnome-shell --version
If you are using a version older than 3.12, you cannot use Gnome with X2Go (you may run into trouble with versions older than 3.08 too). However, you may still get it working with VNC.

To install gnome in “flashback mode”, follow these instructions:

  1. Install the gnome environment by typing into your ssh terminal:
    sudo apt-get install gnome-core
    Alternatively, if you want to have some apps like Gimp, LibreOffice etc. pre-installed, instead of the command above, type:
    sudo apt-get install gnome
    (this will take longer as more packages are downloaded).
    Confirm the installation of the package with ‘Y’.
    Hit [Enter] to confirm the message that comes up.
    Then, you have to select the default display manager. It does not really matter which one you choose. The correct display manager will be started by your remote client anyway. LightDM or GDM are display managers for Gnome. If unsure, just select LightDM.

  2. To switch on flashback mode , we have to install a package and disable the newer scrollbars. Type:
    sudo apt-get install gnome-session-flashback

  3. [This step is not necessary on the Ubuntu version used in this tutorial]
    sudo apt-get purge overlay-scrollbar gsettings set org.gnome.desktop.interface ubuntu-overlay-scrollbars false

  4. [This step is not necessary on the Ubuntu version in this tutorial]
    You might also need the following package on older Ubuntu versions (if in doubt, install it anyway, it should already be installed and then it does nothing)
    sudo apt-get install xfonts-base

KDE

Note: Only versions earlier than KDE 5 are supported in X2Go. With the ubuntu version used in this course, you may use KDE.

image

You can find out your version of KDE (after you installed it with following instructions) with:
kde4-config -v

Install the desktop environment by typing into your ssh terminal:

sudo apt-get install kde-plasma-desktop

Alternatively, if you want to have some more apps pre-installed (this will take longer and take up more space), instead of the command above, type:

sudo apt-get install kde-full

Confirm the installation of the package with ‘Y’. This may take a while as a lot of packages need to be downloaded.

Hit [Enter] to confirm the message about installing KDE. Then, you have to select the default display manager. It does not really matter which one you choose if there are several options. The correct display manager will be started by your remote destkop client anyway. KDM is the display manager for KDE, so if unsure, just select this one.

image

Other desktop environments

There are also other desktop environments, for example MATE, which we will not discuss further in this tutorial.

Connect with X2Go

We will first install the X2Go Server on your instance, and afterwards the X2Go client on your local computer. Then you can access the virtual machine’s desktop with the X2Go client.

Server installation

image

In this tutorial we are using Ubuntu Version 14.10 on our instance and the following instructions are going to assume this. If you are using an earlier version of Ubuntu, or another Linux distribution, you may need to do other steps. This will be commented upon in the following instructions at the relevant places. See the official X2Go documentation for details on the steps to perform.

  1. Add the X2Go repository and refresh the software package system by typing into your ssh terminal:
    sudo add-apt-repository ppa:x2go/stable
    and confirm this with [Enter]. Then, type:
    sudo apt-get update

  2. Install the X2Go server. Type into your ssh terminal:
    sudo apt-get install x2goserver x2goserver-xsession
    Confirm the question on whether to use up the said disk space with ‘Y

  3. We now have to install extra bindings for our desktop environment which add extra features for X2Go (e.g. file sharing). Follow the instructions for the environment you want to use (if it is not listed, see official documentation)
    • LXFE: In your ssh terminal, type
      sudo aptitude install lxde
      sudo apt-get install x2golxdebindings
      Note: If the installer package complains about unmet dependencies and suggests to remove lxde, say yes to this. Afterwards, re-install lxde again with the command

    • GNOME: GNOME only works for versions earlier than gnome 3.10, better earlier than gnome 3.08, as pointed out earlier). In your ssh terminal, type:
      apt-get install x2gognomebindings

    • KDE: In your ssh terminal type:
      sudo apt-get install plasma-widget-x2go

  4. The X2Go server should now be running. You can check this with typing into your ssh terminal:
    service --status-all | grep x2go
    The result should show a line like this:
    [ + ]  x2goserver

    If the service is running. If it isn’t, you can start the server by typing:
    service x2goserver start

If in doubt, or anything is not working, try restarting your instance before you try anything else (we have already restarted the instance here).

Client installation

You will now install the X2Go client on your local computer. Depending on which OS you are using, instructions will differ. Go to the instructions below for your OS. If you have a platform not listed here, you can find additional documentation on how to install the client on the official X2Go website. A good instruction can also be found here.

Connecting the X2Go client

We will now start the X2Go client application that you installed in the last step and connect it to the instance.

Step 1: Start the client

On Linux, you simply type into your local computer’s terminal

x2goclient

On Windows and Mac, start the application as you do with any other program (find the x2goclient application). E.g. on Mac, bring up the spotlight (Cmd + Space) and search for “x2goclient”.

Step 2: Start a new session

The session settings should come up right away if you are using the client for the first time. If you are not, click on “New session”, which is the top left button that looks like a blank sheet of paper.

image

Hint for Mac users: the X2Go windows may be sized too large for your screen initially, so you cannot see the usual “OK” and “Cancel” buttons in the dialogue windows. To resolve this, you simply have to select the window by clicking on the top window bar and holding the left mouse button to move the window a little bit. When you release the mouse button, the window should be automatically resized to fit your screen.

The session preferences window should look similar to this:

image

  1. Specify your session name in the first field (In the screenshot, we called it “NeCTAR Tutorial session”).

  2. In the “host” field, type in your IP address

  3. In the “Login” field, specify the username (in this tutorial, it is “ubuntu”).

  4. Select your private SSH key in the field “Use RSA/DSA key for ssh connection”. You may click on the little folder symbol behind the field to browse for your private ssh key file.
    Note for Mac/Linux: earlier in this tutorial, we have put the key in the .ssh folder in your home directory. If you cannot see this directory, you have to create a shortcut to it. Open a terminal on your local computer, and type in your local computer’s terminal:
    ln -s .ssh ssh
    You should now have a folder called “ssh” in your home directory which you can see from the “select file” dialogue.

  5. As a “Session type”, select the Desktop environment you installed earlier (e.g. LXDE, Gnome, KDE). You may also have installed several desktop environments, in this case select which one you wish to start.

  6. Click “OK” to get to the X2Go Start Window. If asked whether to trust the host, click “Yes”.

Step 3: Connect to the desktop

We will now connect to our remote desktop. You should now be seeing the X2Go Start Window:

image

On the right side of the Window, you can see a shortcut to your session. Note that you can quickly change the desktop environment and screen resolution in the box. Don’t do this now however, let’s first try to connect with the default. You can change this next time you log in.

Click on the box with your session name, and the client is going to try to connect to your instance’s remote desktop. You will have to type in your ssh key passphrase.

This may take a while, especially if it is the first time you connect. Don’t get impatient. This is a good time to grab a cup of coffee.

A window will then come up which shows your desktop. Below is an image of an example when XFCE was used:

image

You can increase the screen size and the resolution of your desktop should adapt to it. Resizing the window can sometimes be a bit slow, therefore you may try different screen resolutions in the login box (which displays your session name) so you don’t have to resize the window every time.

To close your remote desktop, you should log off. This is in most desktop environments a power button somewhere on the desktop, or it is accessible with a right click on the desktop. Choose “logout” as an option, not “restart” or “shutdown”. If in doubt, just close the window in which the remote desktop is displayed (this is the “dirty” option, but it will also work).

When you log back in, you don’t have to specify a password for the ubuntu user, as we are using your ssh key instead.

image

Important Note if you are using KDE:

KDE does not adapt the screen size that we set in the client. Therefore, we have to disable the “KScreen KDED module”. We will do this right away.

  1. On your remote KDE desktop, go to “system settings”. You can right-click on the desktop and then select “Run command…” from the drop down.

  2. In the text input field that comes up, type “System Settings” and the drop-down should show the System Settings which you can click on.

  3. Go to System Settings > Startup and Shutdown > Service Manager, and uncheck “KScreen 2”.

  4. Log out (Right click on Desktop > Leave > Log out) and log back in again (You don’t have to specify a password for the ubuntu user when you log back in, as we are using your ssh key instead). You should now see your full desktop. Your screen size should adapt to what you have set in the session settings of your X2Go client (default is 800x600).

File sharing with X2Go

One nice thing about X2Go is that you can do file sharing between your local computer and your instance easily on the desktop.

Let’s now enable file sharing so we can drag and drop files between our computer to the instance’s desktop.

Step 1. Enable file sharing

We have to enable file sharing for our instance’s user “ubuntu”. For this, you have to add the user to the group “fuse” on your instance. Go to your ssh terminal (or open a terminal on your remote desktop) and type:

sudo adduser ubuntu fuse

image

Important if are using a Mac:

You also have to enable remote login to support file sharing.

To enable file sharing, you also have to change your system preferences. Go to the Apple (on top left of your screen) > System preferences, and then select “Sharing”:

image

Activate the checkbox for “Remote login”:

image

Should you have problems connecting in the following steps, try restarting your Mac computer for the changes to take effect.

Now, we have to configure the X2Go client session preferences to enable file sharing.

First, create a folder on the desktop (or elsewhere if you like) which you are going to use as our “shared folder”. Name it for example “x2goSharedFolder”.

Think of this folder as a place where you can put files which you can then access from within your instance also. You can Drag & Drop files into this folder, and it will be easy to access them from your instances remote desktop. Conversely, you can Drag & Drop files from within your remote desktop into this folder, and they will appear on your local computer as well.

Close your remote desktop by logging off (or just shutting the window in which your remote desktop is displayed).

Then, go back to the X2Go Start Window in the X2Go client (If you aren’t there already, click ‘cancel’ button in the login field). You should now see this X2Go Start Window:

image

Via the little arrow on the right bottom of your session box, select “Session preferences” as shown in the screenshot above. Here you can change preferences of your session. Go to the Tab Shared folders, and click on the little folder symbol to select a folder of your choice which you want to use for sharing (earlier we have named this “x2goSharedFolder” on our desktop):

image

When you have selected the folder, don’t forget to click on the “Add” button next to the path to your file. Next to the new element added to the box, enable the “Automount” check box. Finally, click “OK” to close the window.

image

Step 2. Logging in.

Now you can open your remote desktop by clicking on your session box and logging in again. You don’t have to specify a password for the ubuntu user, as we are using the ssh key instead. You will typically find the shared folder of your computer as a link on the remote desktop (see comments below for your desktop environment).

Step 3. Remaining configurations for individual desktops.

Some desktop environments require some extra settings to make file sharing work. Refer to the documentation below for your desktop environment.

If you are using LXDE:

No extra settings required. Your shared folder should be on the desktop now.

If you are using KDE:

image Your shared folder will be visible on your remote machine by adding the “X2Go plasmoid applet” to the desktop. To add the widget, right-click on the desktop and select “Add widgets…”.

Note: You will need to have installed the plasma widget, which we did in a previous step with the command *sudo apt-get install plasma-widget-x2go.

In the search window that comes up, type “x2go” and the x2goplasmoid should come up in the list. Double-click on it to add it to the desktop. image

You can now access your shared folder on your local computer via the link on the plasmoid. After you open the folder (double click), you should then see something like this on your desktop: image

image

You can change the size of the widget by dragging it if it looks graphically wrong. You need to hover over the right-hand menu of the widget and click with your mouse on the little box in the top: image

You can then drag it out to the right side:

image

If you are using XFCE:

File sharing has to be set up manually. The shared folder can most likely be found in the folder /tmp/.x2go-ubuntu/media/disk. Check what the name of your folder may be. Type into your ssh terminal:

ls /tmp/.x2go-ubuntu/media/disk

you should recognize your folder name, which is named after the path where it is located on your computer (if you named your folder x2GoSharedFolder, the name should end with this). Then, type

ls /tmp/.x2go-ubuntu/media/disk/<your-folder-name>

to see all the files you have in your shared folder (if your folder is empty, no files will be listed, of course). Right-click on the desktop and select “Open Terminal Here”. Then, type

ln -s /tmp/.x2go-ubuntu/media/disk/<your-folder-name> MySharedFolder

replacing <your-folder-name>> with the name you found above. Be careful with spelling (it’s case sensitive) — if you don’t get the path to the folder right, it will not work. You can also change the name MySharedFolder in the command to something else. When you click on the desktop again, the link to the shared folder should appear.

image

image

You can verify where your shared folder has been made available by looking into the file “/var/log/syslog” on your instance. You should see a line that contains the following:*“… successfully mounted @127.0.0.1:30003/ to /tmp/.x2go-ubuntu/media/disk/*

Step 4: Exchange files

Your shared folder can now be accessed both from your instance and your local computer.

First, on your local computer, create a text file in your shared folder (x2GoSharedFolder), write some text in it, and save it to the shared folder.

For example on a Mac:

  1. Open TextEdit (Cmd+Space, type “textedit” and open the TextEdit program)

  2. Write a message into the window

  3. Click “File > Save” and select the shared folder as a place to save the file to. Name it for example “x2goTest”.

image

Now, go to your remote desktop. Open your shared folder (you should have found where it is in the last step). For example, when you are using the LXDE desktop environment, your shared folder will be on your desktop.

image

Now if you open the shared folder, you should see the text file you have just created. You may open it to see what’s in there:

image

Depending on which text editor you used to create your file, it may not look the same on the remote machine. This is because it uses a Linux Operating system, and the default editor on your instance may not open the file with the program which supports this file format. For example, in the screenshot above, you can see that the RTF file we created on a Mac contains formatting instructions as well (you can see it is an RTF file because it is called x2goTest.rtf). Because Linux opens it in a “plain text editor”, you can see the formatting instructions as well.
This should not worry you at this point, because we are just testing how to share files with your remote desktop, but this is a good place to notice some differences between operating systems and text files.

Now, we are going to do it the other way round: We are going to create a text file on our remote desktop, copy it to the shared folder, and see that we can access it in our shared folder on our local computer as well.

On your remote desktop, right click on your desktop, and select “Create new > empty file” (This may be called different in your desktop environment, but it should not be too hard to find how to create a new file. If there is no such command on the desktop, open your file explorer, go to your Desktop in the navigation pane on the left, and try to do it there).
Name the file “TestOnVM.txt”.

image

Open the file by double-clicking on it. A text editor will open. Write a message into the file, save it (File > Save), then close it.

image

Now, drag the file from the desktop to your shared folder. A message will appear that it is being copied, and then you should see it in your shared folder:

image

Now, you can open your shared folder on your local machine (the x2goSharedFolder on your desktop).

It will have the new file in it! You may double-click on it to open it and find the message you typed on your remote desktop inside.

image

Troubleshooting

If you ever run into trouble with the desktop environment, try first to log out and log back in again. If that does not work, you may have to restart the X2Go server. You can do this in the ssh command line of the instance (you may also open a terminal on your remote desktop to do this):

sudo service x2goserver restart
or sudo /etc/init.d/x2goserver restart

Resources

Connect with VNC

7.3.4. Connecting with VNC

We will now connect to our remote desktop using VNC. We will first install the VNC Server on the instance, and then use a VNC Client (also called “viewer”) on our local computer to connect to the remote desktop.

Install the VNC Server on your instance

In this tutorial, we will use the VNC Server called vnc4server on our instance. For an overview of other servers, refer to the Ubuntu help page.

Step 1. Install VNC server

To install the server, type into your ssh terminal:

sudo apt-get install vnc4server

Confirm installation of the package with [Enter].

Step 2. Configure the VNC server

We will now have to edit a file in order to start up the right desktop environment.

image

The following instructions have been tested for Ubuntu 14.04 and 14.10. If you have another Ubuntu version on your instance, you may have to look online if the following instructions don’t work. Some more information is given at this link.

First, we have to start up the VNC server once, in order to create a password and generate the configuration file. In your ssh terminal, type:

vncserver

and you will be prompted to choose a password. This is a password which is used for the VNC server only — it is not the same as your ssh key (although you may of course choose the same passphrase)!

After you finished typing and confirming your password, we will shut down the vnc server again and change some settings. To shut it down, type:

vncserver -kill :1

Then, open the settings file:

nano .vnc/xstartup

In the editor that comes up, make the following changes:

Uncomment the following lines, by putting a ‘#’ in front of them:

# x-terminal-emulator -geometry 80x24+10+10 -ls -title "$VNCDESKTOP Desktop" &    
# x-window-manager &

Additionally, make the following changes to the file, depending on your desktop environment:

Step 3. Start the server

To start the vnc server, you need to configure the resolution which your screen should have (You can change this to another resolution in the command below). To start with, choose a low resolution which will definitely fit into the screen on your local computer with some margin:

vncserver -geometry 1024x768 -depth 24

The result of the command will display a line similar to the following.

New '*hostname*:#* (*username*)' desktop is *hostname*:**#**

where hostname and username are replaced by your instances values, and a number is displayed instead of the hash #. Take note of this number following the colon (:). This will be part of the port number you will need in the SSH tunnel setup, which we will do in the next step. This will also be the number you will need to use to eventually terminate the server when you are finished using it (if you want to shut it down at all).

If you want to shut down the VNC server, type into your ssh terminal (use the number you have taken note of instead of the #):

vncserver -kill :#

For example, if your port number before was 1, type

vncserver -kill :1

image

Whenever you want to change your desktop environment (e.g. from LXDE to KDE), you have to shut down (“kill”) the vnc server with these commands. Then, you have to edit your .vnc/startup file as discussed in the last step, following the instructions for your desktop environment. Then, restart the VNC server again, as you have done earlier.

Connect to the remote desktop

If you tried to use a VNC client right now, it would not be accessible from outside, because our instance is protected by a firewall. When we launched our instance (as we did in this section), we have only allowed secure ssh connections on Port 22 (and insecure connections to web services on Port 80). VNC servers usually provide their services on Port 5900 (or 5901, 5902..). This Port is still being blocked by our firewall!

One way to get access to our VNC server would be to open up this port in our firewall and allow an insecure connection (remember: VNC does not have in-build encryption). This is not desirable — we want a secure connection, of course.

The safe way to do this is to “tunnel” all our VNC communications through SSH, which then encrypts all communication. The following steps will explain how this is done.

Step 1: Set up ssh tunneling

Be sure you have the number (#) you were given in the above vncserver command. We will use this number to determine the Destination Port. Use your number and add it to 5900. For example, if your number is 1, then 5901 will be you destination port.

Follow the steps below for your local computer’s operating system:

Linux / Mac

In a terminal on your local computer, create a tunnel from your local machine (Port 5900) to your instance (Port 5901) with this command:

ssh -i <path-to-your-keyfile> -L 5900:localhost:5901 -N -f -l ubuntu NNN.NNN.NNN.NNN

where you should replace the NNN.NNN.NNN.NNN with yours instance’s IP address, and <path-to-your-keyfile> is the path to where you have saved your ssh key.

Hint for Linux users: If you copy your private key to the /home//.ssh folder, as we have done in this tutorial, you won't have to specify the key every time and can skip the -i option.

image

The -f -N options in the above command cause the tunnel to run in the background: once you have typed in the password, the command line prompt returns to your terminal. If you want to close your ssh connection, you have to kill it. In your terminal on your local computer, type:

ps aux | grep ssh

In the printed result, find your ssh tunneling command and note the number on the left, which is displayed after your user name. One line of output for example looks like this:

ubuntu   52577   0.0   0.2   2512756   17076 …

In this example, 52577 is the process number

kill <the-ssh-tunnel-process-number>

If you don’t use the -N -f options, you will be logged into an ssh terminal to your instance, and can use it as usual to type commands on your instance. If you type logout or close the terminal, the ssh tunnel will close as well.

Windows

  1. Launch PuTTY and load your previously saved configuration by selecting the session name and clicking Load. (If you have Putty open, close it and then re-open it to get to this window).
    image

  2. Select SSH > Tunnels in the left pane. Enter these values:
    Source port: 5900
    Destination: localhost:5901
    Then, click the Add button. Your tunnel will now appear in the list. image

  3. You can save the tunneling along with your session by selecting Session in the left pane. Click on your session name in the Saved Sessions pane, then click Save (or create a new name instead by typing in a new name, e.g. called “TunneledNectarSession”).

  4. Now you can click Open to start the SSH terminal. The tunnel will now be active as long as the PuTTY terminal window stays open. You can use this SSH terminal as a shell into the instance as well (as we have been doing the whole time), while simultaneously having your remote desktop open.

  5. The next time you run PuTTY, you must open these saved settings by selecting the name in the Saved Sessions pane and clicking Load, then clicking Open. Or, you can simply double-click the session name to load and run the saved session.

See also the section about SSH Tunnelling in the Appendix for more information.

Step 2: Install the VNC Client on your computer

We can connect to our desktop using a VNC Client (also often referred to as “VNC Viewer”). You may download any VNC viewer you want.

Here, we’ll suggest the following, depending on which OS your home Computer runs:

Ubuntu

You can use the pre-installed Remmina client.

Alternatively, install vinagre with the command
sudo apt-get install vinagre.

You can use the free RealVNC Viewer. Continue with the instructions below (Mac/Windows/Linux alternative client: RealVNC) if you would like to use RealVNC.

We recommend you use the default Remmina client to start with.

Windows

You can use the free RealVNC Viewer. Continue with the instructions below (Mac/Windows/Linux alternative client: RealVNC) if you would like to use RealVNC.

Alternatively, you can also use UltraVNC or TightVNC ( Note for TightVNC: Choose the “custom” installation mode and then select to only install the client, you may have to unselect the server).

Choose the client you want and install it. We suggest you use RealVNC for this tutorial.

Mac OS X

OS X has a VNC client built in. It’s been installed in every version of OS X since 10.4. It’s called Screen Sharing and it’s located in your CoreServices folder, a difficult to find folder which is part of OS X.

The easiest way to find screen sharing via the “spotlight search”: Type Cmd+Space to bring it up. Type “screen sharing” and the app should be found.

Or you can also find the application in this path:

/System/Library/CoreServices/Screen Sharing.app

or, as of OS X 10.10 Yosemite the containing folder is now located in

/System/Library/CoreServices/Applications

You can access it through the terminal (Applications Folder > Utilities > Terminal.app). In the terminal, type:

open /System/Library/CoreServices

or on OS X greater than 10.10:

open /System/Library/CoreServices/Applications

That will open a Finder window where you will see Screen Sharing and you can open it or create an alias if you want.

You can use the free RealVNC Viewer. Continue with the instructions below (Mac/Windows/Linux alternative client: RealVNC).

Mac/Windows/Linux alternative client: RealVNC

Download the client (the viewer) from the RealVNC website. When downloading, it may ask for your E-Mail, but you can specify some random details, the viewer will still work. When installing, choose to only install the client (the viewer):

image

When you connect (we will do this in a moment), it asks you whether you accept an insecure connection. Say yes (it will in fact be secure via our tunnel).

RealVNC by default compresses the colours to save bandwidth. To see your screen in full colour, you need to go to Options in the initial window, then disable The checkbox for *Adapt to network speed, and push up the slider to Best quality.

Step 3: Connect to the VNC server

Open your VNC client (“viewer”) of choice. This will generally be a very simple interface with just a few fields to fill. The following instructions are generic for all clients. There are also some more detailed instructions given below for some examples.

Now connect, and you should get a password prompt, in which you type the password you just chose for the VNC Server (not your ssh passphrase, unless you chose the same!). Then, you should get to the remote desktop.

image

Do not use any ssh or security options that clients may offer, because we have just done the ssh tunnel separately ourselves. Your connection will be secure. If you have followed the instructions above, your connection is secure, even if the vnc viewer says it is not (The VNC viewer does actually not realise it is running through the ssh tunnel).

image

Example on the image above: Connecting with TightVNC on a Windows machine. The Putty Terminal is displayed on the left.

Mac Example client: Screen sharing

To connect with the in-built Screen Sharing app, open it (Apple+Space and type “screen sharing and open it up). Into the text field, type *localhost:5900 *and then click “connect”:

image

You will be prompted for your password. That’s the password you selected when you set up the VNC Server in the previous step.

image

Now a window should open to the remote desktop.

Example client: RealVNC

Start the RealVNC client (on a Mac, you can find it via the spotlight search: Cmd+Space and then type “realVNC”. It will open a folder in which you find the executable).

To launch a connection, type “localhost:5900” into the text field. Leave the encryption setting on “Let VNC Server choose” and click “connect”.

image

When it asks you whether you accept an insecure connection, say yes (it will in fact be secure via our ssh tunnel).

image

You will now have to type in your password. That’s the password you chose when setting up the VNC server on your instance in the last step.

image

RealVNC by default compresses the colours to save bandwidth. To see your screen in full colour, you need to go to the realVNC “Options”. You can either do this in the initial login window, or when your remote desktop is open, mouse over the top of the window and select “options”.

image

In the options, disable The checkbox for Adapt to network speed, and push up the slider to Best quality (better quality will be slower but mostly you may want it anyway). Also enable the checkbox Scale to window size if you have issues with the window being too large for your desktop.

General troubleshooting

Your colours look a bit off?

Some clients (e.g. RealVNC and Remmina) use fewer colours and higher compression in order to reduce latency of your remote desktop. You have to change this settings in the client - each client is different so you might have to find out where you can change this.

In Remmina, you can create a saved session (by clicking on “new”) instead of using the default Quick Connect. You can change the settings for the colour quality there.

RealVNC by default compresses the colours to save bandwidth. To see your screen in full colour, you need to go to Options in the initial login window, then disable the checkbox Adapt to network speed, and push up the slider to Best quality.

Problems connecting?

Make sure your ssh tunnel is open and you have set the right port numbers. On Windows, your PuTTY session must be open and you must have configured the tunnel as described above.

Automatic start-up of the VNC server

With the current settings, you will have to manually re-start the Vnc server on your instance every time after you reboot the instance. We can automate this to save you this work, and start up the Vnc server automatically.

The easiest way is to add an execution command in a file. Log into your instance (via ssh terminal) and follow these instructions:

  1. Verify the location of your VNC program by typing:
    which vncserver
    Most likely you will see /usr/bin/vncserver as result. We will assume this in the following. If you get something else, you will have to use your own path instead of this one.

  2. Create a file startVNC.sh (or call it whatever you want) in directory /etc/init.d:
    sudo nano /etc/init.d/startVNC.sh

  3. The editor will open. Put the following contents in the file (Note: If your username is other than ubuntu, you have to replace it in the 3rd line):
    #!/bin/bash    
    PATH="$PATH:/usr/bin/"    
    export USER="ubuntu"    
    su ${USER} -c "/usr/bin/vncserver -geometry 1024x768 -depth 24"

    Note: If your vncserver is located somewhere else, as you have found out in step 1, you have to use your own path here instead of /usr/bin/vncserver

  4. Close the editor with Ctrl + X (and confirm saving the file with “yes”).

  5. Now we have to make this file executable
    sudo chmod +x /etc/init.d/startVNC.sh

  6. Add this file to be executed at startup:
    sudo update-rc.d startVNC.sh defaults

Optional:

You may also set up a “service” out of the vnc server. After you have set it up, it allows you to start the vnc server any time with the following command:

service startVNC.sh start

(or shut it down with stop instead of start). This is probably the more proper way to do it, but the file startVNC.sh is a bit more complex:

#!/bin/bash    
PATH="$PATH:/usr/bin/"    
export USER="ubuntu"    
DISPLAY="1"    
DEPTH="32"    
GEOMETRY="1024x768"    
OPTIONS="-depth ${DEPTH} -geometry ${GEOMETRY} :${DISPLAY} -localhost"    
. /lib/lsb/init-functions    

case "$1" in    
   start)    
      log_action_begin_msg "Starting vncserver for user '${USER}' on localhost:${DISPLAY}"    
      su ${USER} -c "/usr/bin/vncserver ${OPTIONS}"    
;;    
   stop)    
      log_action_begin_msg "Stopping vncserver for user '${USER}' on localhost:${DISPLAY}"    
      su ${USER} -c "/usr/bin/vncserver -kill :${DISPLAY}"    
;;    
   restart)    
      $0 stop    
      $0 start    
;;    
esac   
 
exit 0

Reconnect to the VNC afresh

Of course, you won’t have to do all the above steps every day when you want to connect to your remote VNC desktop. The only things you need to do each time are:

  1. Start the ssh tunnel, as described above in step 1. On Windows, if you are using PuTTY, you may have created a saved session which you can start easily.

  2. Open your VNC client and connect, as described in step 3 above.

Let’s do this now to see how easy this is. Close all your applications (all your terminals / PuTTY, and your open VNC viewer).

If on Unix/Mac, also kill your ssh tunnel (for instructions, see the notes about SSH tunneling in the Appendix).

Now, do the above steps 1. and 2. above to get back to your remote desktop.

Troubleshooting on Windows/Putty:

If you are Windows and PuTTY, check whether you have properly saved your session last time. When you click on your session and then click “Load”, you can check whether all the settings we have configured earlier (“Connect to the instance”, and “Step 1: Set up ssh tunneling”) are still there. If you do not have all the details saved, change them now and save your session again (click on Session in the navigation panel on the left, select your session in the Sessions pane, and click Save).
Then, try again.

Resources

Running several desktop environments

It is possible to have several desktop environments installed and use them interchangeably. You will have to install each desktop environment you want to use on your instance (as we have done in this section earlier.

Note that with VNC, you have to change your desktop environment on the server side (on your instance), and everyone who connects to the instance will automatically use the desktop which has been configured for this instance. To change the desktop environment, you have to edit the xstartup file and restart the vnc server, as described in the last section.

With X2Go, you can decide on the X2Go client which desktop to use — in your x2goclient application, you can choose from the drop-down box which desktop environment you want to run.

Graphical Interfaces on the ssh terminal

If you want to run a single graphical application, you can also do this without setting up a remote desktop. You can do this via the ssh command line and start the application directly.

The advantage is that you don’t need to set up a remote desktop server on your instance and install a client on your local machine.

The downsides are (1) it introduces some security issues, which will not be discussed in detail here, and are usually not a big worry (see security note below for a few details). And (2), this is (perhaps surprisingly) a lot slower than a remote desktop solution. This is because of the way the communication is encoded.

Note: This technique allows no desktop sharing, just remote application running (However you could use XNest, not discussed further at this point).

image

The security issue arising with running graphical applications via the ssh connection is that we have to enable X11 Forwarding. X11 is the base for the Linux graphical applications (e.g. opening windows). With X11 forwarding, the server is likely to gain shell access to the client (the ssh terminal you are running on your computer): the server can send X11 commands back to the client. X11 was not designed with security in mind, it was designed with the idea that all programs that you’re displaying are run by you and therefore trusted anyway. However ssh also has a SECURITY extension which disables some obvious attacks such as keyboard grabs and key injection. However it still allows other security issues like focus stealing.

In simple terms: Let’s say you open a SSH connection to <some-server>, with X11 forwarding enabled. The primary risk is that if <some-server> is malicious, then <some-server> can do all sorts of things to the windows/applications you have open on your own computer.

For instance, <some-server> can open up windows on your computer, can close other windows you have open, can spy on the contents of other windows you have open and on the keys you type into other windows. It can inject spoofed keystrokes and mouse events into other windows you have open, and generally just mess with every other window you have open on my machine — even if some of those other windows are local applications running locally.

In general, because you log on to your own remote machine and not just <some-server>, this security risk is not much of a worry — only if your instance was infected by malicious software, this would be a potential security lead.

Step 1. Check that your Instance allows X11 Forwarding

We have to enable X11 forwarding. It may already be enabled, but make sure it is. Go to your ssh terminal and type:

sudo nano /etc/ssh/sshd_config

to edit the configuration file.

Scroll down until you find the line

X11Forwarding yes

(it may be a no instead of the yes). If this line starts with a #, remove the #. Make sure it is set to yes. If you cannot find the line, add it at the end of the file, just as it is shown above.

Scroll further to find the following line:

X11DisplayOffset 10

and make sure it is set to 10.

Exit the editor with Ctrl+X (and confirm saving the file with ‘Y’).

If you did not have to change the file, you can jump to the next step.

Otherwise, you now have to restart ssh on your instance. You can do by typing:

sudo /etc/init.d/ssh restart

Step 2. Log in to your instance via ssh

Linux

Open up a terminal. We will connect to the instance with the ssh command that we used in an earlier section, but this time you have to add the -X option to the command, to enable graphics mode.

ssh -i <your-private-key-file> -X ubuntu@NNN.NNN.NNN.NNN

Mac

You first need to install XQuartz on your Mac. If you don’t have it already, download from the XQuartz website

Note: If you have previously followed the steps to install the X2Go Client, you will already have XQuartz installed!

After you installed XQuartz, open up a Terminal on your local computer. We will connect to the instance with the ssh command we used in an earlier section, but this time you have to add the -X option to the command, to enable graphics mode.

ssh -i <your-private-key-file> -X ubuntu@NNN.NNN.NNN.NNN

Windows

If you use Windows you will need to install a X11 server locally on your windows computer. You can use for example Xming.

Open your PuTTY session and load your settings that you previously saved (to do this, select Session in the navigation pane on the left, select your saved session from the list, and click Load).

Then, go to “Connection > SSH > X11” in the navigation pane. You need to check Enable X11 forwarding and specify the X display location. If you did not change anything in the default configuration of Xming, then set the display location to “localhost:0”.

Otherwise, if you change the Xming configuration using the tool Programs > Xming > XLaunch, make sure to set the parameters for the display accordingly.

image

Step 3. Install X Windowing system on your instance

Your instance still needs a concept of graphical interfaces, so we need to install it.

image

If you already have previously installed a desktop environment (e.g. LXDE, Gnome, KDE, XFCE) as we have done in an earlier section, then you can skip this step, because the windowing system is already installed.

To install the very basic windowing system, go to your ssh session into your instance and type:

sudo apt-get install xauth

sudo apt-get install x11-apps

It is likely the xauth package (first command) already installed, but we are doing this just in case. If it is installed, the first command will not re-install any packages.

Step 4. Run a graphical application

We will now test if this works. Go to your ssh terminal (or on Windows, the PuTTY terminal) and type:

xcalc &

This should open up a simple calculator application that is installed by default. You may have to wait quite a while until it opens up, as said earlier, this way to run graphical applications pretty slow.

image

Note: The & after the command causes the calculator to open up and detach from the terminal. If you skip it, the calculator will open, but you won’t be able to type other commands in the terminal.

If this worked, try a larger application, like firefox. Go to your ssh terminal (or PuTTY on Windows) and type:

sudo apt-get install firefox

(It may already be installed)
Then, type

firefox &

and wait a while until firefox opens. You may notice the amount of time you have to wait is much longer than when you have a remote desktop open (as an earlier section described). To start with, ignore some warnings and error messages that may come up — it really takes forever for firefox to come up.

Troubleshooting on Unix/Mac

If you receive errors trying to run graphical applications, try the option -Y (instead of -X) in your ssh command (in Step 2 above).

Install a web server

In this section we are going to enable a web server and create our first simple web site.

To install the web server, go to the ssh terminal and type:

sudo apt-get install apache2

Confirm installation of the package with ‘Y’.

Now, open the web browser on your local computer, and type your instance’s IP address in the URL field. We will see the default apache website of the browser.

Let’s now start our own simple website. In your ssh terminal, type:

nano /var/www/html/index.html

the directory /var/www/html is where your website lives by default, and the file index.html is the start page which we want to edit. In the editor that comes up, remove all the text inside and replace it with the text below.

Tip: Copy and paste will not work with many terminals. If the traditional copy & paste does not work, you can instead try the following: copy the text as usual; then click with your mouse on the place you want to insert the text in your ssh terminal editor; Then, click the right or middle mouse button — this will work on many systems to “paste” the contents of your clipboard.

<!DOCTYPE html PUBLIC "-//IETF//DTD HTML 2.0//EN">    
<HTML>    
  <HEAD>    
    <TITLE>    
      A Small Hello    
    </TITLE>    
  </HEAD>    
<BODY>    
  <H1>Hi</H1>    
  <P>This is very minimal "hello world" HTML document.</P>    
</BODY>    
</HTML>

Now, go to your browser again and refresh the page. You should now see a very crude website displaying

Hi, this is a very minimal “hello world” HTML document.

Managing storage

image

Prerequisite: You should know what types of storage are available on the NeCTAR cloud. This was discussed in Module 6. You should also be familiar with connecting to your instance via an ssh terminal, as described in an earlier section.

In this section, we will go through the steps to set up various types of storage on our instance.

A brief recap of the available storage types:

Requesting storage

The NeCTAR trial account only includes object storage and ephemeral storage. You will have to file an allocation request to get access to Volume storage.

The allocation process was described in Module 5, and more information can also be found on the NeCTAR support website.

image

If you don’t have an allocation yet, you can complete the next section, On-Instance storage, and section Object storage. You will have to wait for your allocation until you can follow instructions for Volume Storage. However you can still read through the Volume storage instructions to see that it is easy, and in parts very similar to handling the secondary disk with the On-Instance storage.

On-instance storage

On-Instance storage appears as two separate hard disks. For example, in the flavor “m1.small” (you selected the flavor when launching the instance), this is

  1. a primary disk (10GB) which is saved with snapshots you make of the instance (see Module 9 for details about snapshots), and

  2. a secondary disk (30GB) which appears as a second hard disk on your instance, and is not backed up with snapshots. You will have to back up your data separately, which is a topic we will discuss later in Module 9.

The primary disk (1) is usually used for the operating system, you should not use it primarily to store data. Your “home directory” is also located within this disk space (when you log on with ssh, you start off in your “home directory” /home/ubuntu/). To store large and important files, you should use the secondary disk (2). The advantage of this disk, as opposed to the volume storage, is that you don’t have to create it separately: it already comes with your instance. Depending on the flavor you choose, you can get up to 480GB of storage for the secondary disk.

To get access to the secondary disk, we will have to “mount” it first. Mounting is a technique which essentially makes your hard drive accessible to you at a certain location (in a folder). Usually, the secondary disk is available on your system path /dev/vdb. However this is a device file, so you cannot read files there as if it was a folder. For this, we have to “mount” /dev/vdb to a folder first.

image

On some images, e.g. on the Ubuntu image we use, the device /dev/vdb will already have been mounted for you, most likely in the folder /mnt. However we will still go through the steps of how to mount a device to another folder (name it for example /data), in order to learn all steps involved in preparing storage. You will benefit from this knowledge when setting up the Volume storage.

To see where your secondary disk is mounted (or if it is mounted), go to your ssh terminal and type:

mount | grep vdb

You may get output that looks similar to this:

ubuntu@myfirstinstance:~$ mount | grep vdb    
/dev/vdb on /mnt type ext3 (rw)

This means that vdb is mounted on mnt and has the ext3 file system with read (r) and write (w) permissions. If you don’t get an output at all, vdb is not mounted.

Alternatively, try this command to check the mount point and size of your disks:

sudo lsblk -l

This should give you an output similar to this:

ubuntu@myfirstinstance:~$ sudo lsblk -l 
NAME  MAJ:MIN   RM  SIZE  RO TYPE MOUNTPOINT 
vda   253:0     0   10G   0  disk             
vda1  253:1     0   10G   0  part        /
vdb   253:16    0   30G   0  disk        /mnt

This tells us that we have two hard drives, vda and vdb, of sizes 10G and 30G respectively. The primary disk (vda) is further partitioned into a sub-disk called vda1 (using the full 10G) and this is mounted under the mountpoint “/”. In Linux terms, this means this disk is used for the operating system. The secondary disk (vdb) has 30G and is mounted on /mnt. Many OS, for example Ubuntu, already have mounted the secondary disk, but it may also not be mounted yet, in which case nothing is specified under MOUNTPOINT.

If your disk vdb has already been mounted, take note of what is specified as mount point for it. Then, unmount the disk using the command:

sudo umount <mount point>

replacing <mount point> with your mount point. For example, if your disk was mounted on /mnt, do

sudo umount /mnt

You may also be interested in the file system type which your secondary drive is currently using.

image

A file system is used to control how how the storage of data (files, folders, etc.) is organized on a computer disk (hard disk, floppy disk, CDROM, etc.).

The large amounts of data on a hard drive needs to be organized, similar to a library of books which needs to be catalogued, otherwise books cannot be found efficiently. Libraries have developed standards to organise their books, and similarly, computer systems have. Windows uses the vFat or the NTFS systems, while Linux has others, e.g. ext3 and ext4 are current standards.

File systems have different properties, e.g. maximum file size or partition size that is supported, and whether they journal or not. A journaling file system is more reliable when it comes to data storage: it prevents inconsistency, and is much faster at file system checks than non-journaled file systems.

For more information about the different Linux file systems, refer to this Ubuntu help page.

To find out which file system type your drives use, type:

sudo lsblk -f

which will print a different view of the information we have seen in the last command (because we use the -f option instead of -l). It should look similar to this:

ubuntu@myfirstinstance:~$ sudo lsblk -f
NAME     FSTYPE   LABEL        UUID               MOUNTPOINT 
vda                                                            
└─vda1   ext4                  1b85cad6-...       /           
vdb      ext3     ephemeral0   24cf7b45-77...

From this, we can see that vda1 is a partition of vda. We can also see that vda1 uses the file system type ext4, while vdb is formatted with ext3.

In the above example, the disk devices are available on the paths /dev/vda and /dev/vdb (you have to put the /dev/ before the disk name to get the path).

Follow these instructions below to set up your secondary hard disk. You should do this every time you start up a new instance and want to prepare the secondary disk for usage.

Step 1. Formatting

The first time you start up an instance, you may want to format the hard drive of your secondary disk to make sure you have the right file system type and start from a clean state.

image

Formatting erases all data that may already be on the hard disk. If you are working with a new instance, nothing of importance will be on it, e.g. only a default system Lost+Found folder, or something similar. You should only be careful with this step if you have used the disk before and have data on it. Then, you should back up this data or skip this step (if you skip this step, you will have to keep the file system which is already used on the disk).

One way to format the drive with the ext4 file system type is:

sudo mkfs.ext4 /dev/vdb

(if your secondary drive is at another path than /dev/vdb, which you have just found out before, you have to replace this here).

Confirm the formatting with ‘y’. You should see the status message

Writing superblocks and filesystem accounting information:

Wait a while until it says “done.”, and then your hard-drive is formatted.

Now, you may want to verify that you have formatted the drive with the ext4 file system. Again, use the command

sudo lsblk -f

which prints information similar to this:

ubuntu@myfirstinstance:~$ sudo lsblk -f
NAME    FSTYPE  LABEL       UUID               MOUNTPOINT  
vda                                                                 
└─vda1  ext4                1b85cad6-...       /            
vdb     ext4                24cf7b45-77...

You should now see that vdb is formatted with ext4. Only vda is mounted on ”/”, and this should stay this way, because the OS is saved under ”/”.

We will mount the secondary drive in the next step.

Step 2. Mounting

You can now mount your hard drive, so you can access it. First, create a folder which will be used to access your data. For example, we can name it /data:

sudo mkdir /data

Then, mount it:

sudo mount /dev/vdb /data

image

This will mount the disk if the type of file system can be guessed. Otherwise you may have to specify the file system type with the -t option, e.g.
sudo mount /dev/vdb /data -t ext4.

If you did not create and format the Volume then you can look up the filesystem type:
sudo parted -l

This maps the device dev/vdb to the folder /data. Now, your disk is mounted at /data, and you can add, remove and access files on your secondary disk at the path /data.

You can list the contents of this directory to see what is on your disk (alternatively, if you are using a remote desktop, you can open it in your file explorer. It should be displayed as a disk symbol in the left navigation panel of your file browser).

ls /data

You may see that there is a Lost+Found folder (this is a default system folder). If nothing is printed, you have no files there yet.

Step 3. Write permissions

By default, you do not have write permissions on the secondary drive. You can try to copy a file onto /data, but you will get an error message about write permissions (Note: This is not due to our folder not being located in our home directory). So we need to add our write permissions now.

First, find out which user group you belong to is. Type

id ubuntu

and find what is written behind “gid=”. This may be gid=1000(ubuntu). So your group name is also ubuntu and has the ID 1000.

Now, there are two options: You can 1) enable write permission only for yourself (your username ubuntu), or 2) for the user group you are in. This way, you can share the drive with other users that you may give access to your instance and who you put in the same user group. The instructions below detail what you need to do for both alternatives.

Option 1)

sudo chown ubuntu /data

Option 2)

sudo chgrp ubuntu /data
sudo chmod g+rwx /data

This should do it. You can now write and execute files from your secondary drive.

You will only need to do this once for the secondary drive on this instance.

Step 4. Auto-mounting

Every time you reboot your machine, you have to mount (not format!) the secondary drive again (so only do Step 2 above, not Step 1). We can avoid this by auto mounting it. For this, we have to edit a file called fstab:

sudo nano /etc/fstab

Look for a line which starts with /dev/vdb. You may find one that looks like this:

/dev/vdb  /mnt    auto    defaults,nobootwait,comment=cloudconfig   0  2

If you find such a line, this means your secondary hard disk is already automatically mounted at system start. On the Ubuntu distribution we are using in this course, this is the case: it is auto-mounted to the folder /mnt, as we can see in this line. If you find this line, you can edit it and just change the /mnt *to */data. If you cannot find this line, just add this line to the file at the bottom:

/dev/vdb   /data  auto   defaults,nobootwait   0  2

Exit the nano file editor with Ctrl+X and confirm saving of the file with ‘Y’. Next time you reboot your instance, you should already find your secondary drive in your folder /data, and you won’t need to do any of the Steps described here to access it.

Object storage

You can connect to your object store with a variety of clients, as long as they support OpenStack Swift or Amazon S3. We will discuss some example clients which support Swift in this section.

image

Security Warning

Swift does NOT provide encryption of the data it stores.

If you have sensitive data that requires encryption you must encrypt the data files before upload them to your object store.

Access to your object store requires your password, but this password may be saved in a file that people who have access to your local computer can read. You must be careful to not give anyone access to your computer — similarly, they would get access to your private ssh key that you save on your computer to log in to your instance, so be careful with giving people access to your computer.

image

The NeCTAR implementation of Swift is geodistributed across Nodes of the Research Cloud so that availability is not reliant on any one datacentre or network infrastructure. Each collection of Swift nodes/hardware is known as a region, which may or may not include a Swift proxy server (the Internet facing and serving component of Swift). With some Swift clients/APIs users can explicitly choose which proxy to connect to, this might be useful e.g. for speeding up writes to object storage by choosing the nearest proxy. Due to NeCTARs Swift having multiple regions, some clients/APIs require explicit configuration of a default region, which should be “Melbourne” for most users (see e.g. the Cyberduck profile below).

Get your OpenStack credentials

Before you start, you should get your login details. Note that you will not need to do this if you just want to upload files via the Dashboard (as described in the next section). But if you want to use other tools, and to complete this tutorial, you should take note of your credentials now.

You can download your credentials from the Dashboard

  1. Your tenant ID is the name of your project as reported in the left hand panel of the dashboard (e.g. pt-12345)

  2. Your username is reported at the top of the dashboard on the right (e.g, logged in as: user@uni.edu.au)

  3. Open the Dashboard API tab: get there via Compute > Access & Security > API Access.

  4. Look up what is specified for the ‘Identity’ Service in the Dashboard API tab. It will be a URL like
    https://keystone.rc.nectar.org.au:5000/v2.0/
    Take note of the text between https:// and the colon (keystone.rc.nectar.org.au in the example). This will be the Server. Take note of the number (5000 in the example). This will be the Port. You will need the Keystone Server and Port to authenticate your access. Also take note of the keystone version which you are using, in this case its v2.0.

  5. Get your credential file (this is called the OpenStack RC file for OpenStack, and EC2 file for Amazon). You can download by clicking on the buttons on the top right on the same window: Download OpenStack RC File. Download it and save it somewhere on your computer where you can easily find it later.

  6. You will also need your OpenStack password. This is not the same password that you use to log in to the Dashboard! If you have never used the OpenStack password before, you need to generate it first. You can do this in the Settings of the Dashboard: click next to your user name (your e-mail) on the top right and a drop down menu will open. Click on Settings. You need to reset your password by clicking Reset Password on the left panel of the settings. A long combination of numbers and letters will be displayed. This is your OpenStack password. Copy the text and save it somewhere safe.

    image

    Your OpenStack password will be a long combination of numbers. At the time of writing, unfortunately this password cannot be changed yet, but this should be changed in a future update of OpenStack on NeCTAR.

image

You will not need your OpenStack RC file in this Module, but you will need it later in Module 10. So while you are on the Dashboard API page already, you may download it now, and remember for later where you put it.

All platforms: Access via the Dashboard

We will upload some files to our Object Store using the Dashboard. This is probably the easiest method to upload files.

First, select any file on your computer — some image, text file, or anything not so important. If you are unsure which to choose, just create a new text file, write something in it, and save it to the Desktop (or anywhere else where you can find it.

In this example, we are using a file called SampleObject.txt with the text “I am a sample object file” in it, saved on the Desktop.

Step 1: Create a container

Go to the Dashboard and then to Object Store > Containers. You will get to a page which shows the overview of your containers.

image

First of all, you will have to create a Container.

image

A Container is like a folder in which you can put files. The difference to a normal folder is that a Container is intended for a flat hierarchy, so you cannot create containers within a container — however you may create “Pseudo Containers” within the Container, so you can still organize your files hierarchically.

Click on Create Container. As a Name, type in MyObjectContainer (or you may use your own container name, in which case you have to consider this in the instructions below). Keep the access on private.

image

Click on Create Container. Your Container MyObjectContainer will then be listed on your Dashboard:

image

Step 2: Upload a file

Click on Upload Object on the right side of your container overview. A window will open in which you can Browse for your file.

Note: If you cannot see the “Upload Object” field, you have to click on your container MyObjectContainer to select it.

image

You may also rename it in the given text field if you wish, for now we are going to keep the name SampleObject.txt.

Click Upload Object to close the window and upload the file to your object store. It will now be displayed on the right side of the overview. We can see that it has 27 Bytes.

image

If you wanted to download this file (e.g. when accessing the store from another computer), you can do this here by clicking on Download.

Windows/Mac: Access with Cyberduck

  1. Download the newest version possible of the application Cyberduck (you need at least version 4.4): http://cyberduck.ch/

  2. Install Cyberduck.
    • On a Mac, you will download a .zip file which contains the app. Save it to the Desktop (or any other folder). Extract the zip file by double-clicking on it.
    • On Windows, you will download an installer. You may keep the default options, if unsure. You don’t need to install Bonjour, if you are asked.
  3. Create a text file on your local computer (e.g. save it on the Desktop) with the following contents. Name it NeCTARObjectStore.cyberduckprofile (or any other name, it just has to end with .cyberduckprofile).
    <?xml version="1.0" encoding="UTF-8"?>    
    <!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">    
    <plist version="1.0">    
       <dict>    
          <key>Protocol</key>    
          <string>swift</string>    
          <key>Vendor</key>    
          <string>NeCTAR Swift (Keystone)</string>    
          <key>Context</key>    
          <string>/v2.0/tokens</string>    
          <key>Region</key>    
          <string>Melbourne</string>    
          <key>Description</key>    
          <string>NeCTAR Swift Melbourne (Keystone)</string> 
          <key>Scheme</key>
          <string>https</string>
          <key>Default Hostname</key>
          <string>keystone.rc.nectar.org.au</string>
          <key>Default Port</key>
          <string>5000</string>
          <key>Hostname Configurable</key>
          <true/>
          <key>Port Configurable</key>
          <true/>
          <key>Username Placeholder</key>    
          <string>tenantId:username</string>    
          <key>Password Placeholder</key>    
          <string>password</string>      
       </dict>    
     </plist>
  4. Double-click on your file NeCTARObjectStore.cyberduckprofile.

  5. A Dialogue should open where you can specify Server etc. If it does not, open Cyberduck and go to Bookmarks > New Bookmark.
    image

  6. Choose NeCTAR Swift Melbourne (Keystone) from the drop-down box, if it is not selected yet. If it does not exist, something has gone wrong in importing the NeCTAR profile (your file NeCTARObjectSTore.cyberduckprofile). The image shows how it looks on a Mac, on Windows your interface will look a bit different, but the fields are the same.

  7. Nickname: give the connection a name of your choice, e.g. ‘NeCTAR Object Store

  8. Confirm that Server and Port match the OpenStack credentials you took note of. If they don’t match use the values you took note of.

  9. TenantID:Access ID (or username on a Mac): Type in your tenant ID and username (you took note of this before) separated by a colon, for example pt-12345:user@uni.edu.au.

  10. Expand the window image by clicking on More Options. You can choose a Download Folder to which your files will be downloaded by default. As Path you can (but do not have to) specify the version number of keystone, e.g. /v2.0/.

  11. Leave all other options in the default setting, and close the window (by clicking on the window cross)

  12. Double click on the symbol of your connection and Cyberduck will try to connect to your object store.
    You will be asked to type your OpenStack password (not the password you use to login to the Dashboard!) This is the password in the OpenStack credentials you took note of. You might also be required to re-enter your tenant ID and username, again separated by a colon image

  13. You can now drag and drop files into your object store container (drag in on the folder symbol), and from the object store container to your local computer (onto your desktop or into an open folder in your file explorer). You can also right-click and Download (it will download into the folder you selected as Download Folder in the connection dialogue).

  14. You can synchronize files to a folder on your local computer. Right-click on the container you want to synchronize, and select Synchronize. You will be prompted to select a folder to synchronize into.
    image

    Attention! Do not select the same folder which you have specified as Download Folder in the connection dialogue! This will create duplicate and nested directories.

    Select your folder, and it will start to synchronize your files. You can then access your object store files in this folder, edit them from there, synchronize with Cyberduck again (you have to select the same folder again), and your Object Store will be up-to-date!

  15. You can edit files directly by right-clicking on them and select “Edit”. Then you may choose a program to edit it with.

Resources

NeCTAR support website

Cyberduck and OpenStack HowTo

Windows only: Access with Cloudberry

  1. If you do not already have CloudBerry Explorer then download and install it on your computer from the official website.

  2. Open CloudBerry Explorer

  3. Click File > OpenStack > New Account. Then, click Add.

  4. Display name: use a meaningful name to this account (e.g., the name you supplied as your container in the step above)

  5. User name: your username, which is your e-mail (e.g., user@uni.edu.au)

  6. API key: your OpenStack password

  7. Authentication Service: This is the server, port and version of keystone that you took note of with your credentials, e.g. https://keystone.rc.nectar.org.au:5000/v2.0

  8. Tick Use Keystone authentication

  9. Select the Tenant name option from the drop-down menu and insert your tenant ID (e.g., pt-12345)

  10. Close the Window with ‘OK’, and close the next Window with “Close”.

  11. On the right window, select your display name from the drop down box. CloudBerry will connect to your object store. On the left window, you see your files of your local computer.
    image

  12. You can drag and drop files between both windows.

  13. You can directly edit a file by Right click > Open.

Linux and Mac OS X: Using any FTP client

Prerequisite: You should know how to use an FTP client.

You can use any FTP client, for example Filezilla. For this, we will have to run a program that acts like a FTP server on our own local machine. This FTP server will then communicate with the SWIFT object storage on our cloud and provide the file exchange via the FTP service.

The following instructions are for a Debian based distribution, e.g. Ubuntu. For more information, refer to the official project page.

Open a Terminal on your local computer to type in the following commands.

  1. Install the ftp server package
    Linux
    sudo apt-get install ftp-cloudfs
    or use your other package manager if you are on another distribution than ubuntu.
    Mac OS X
    • Install python’s pip (unless you already have it) using the homebrew package manager:
      brew install python
      image

      Alternative: If you don’t have homebrew, you may use the easy_install command, however there may be problems with running the ftp-cloudfs command with this. You may try anyway, and if it does not work, then uninstall pip again (sudo pip uninstall python-swiftclient) and get homebrew instead.
      sudo easy_install pip

    • Upgrade setuptools
      sudo pip install --upgrade setuptools
    • Install python-swiftclient
      sudo pip install python-swiftclient
      sudo pip install python-keystoneclient
    • Install ftp-cloudfs pip install ftp-cloudfs
  2. Start the local FTP server
    ftpcloudfs --foreground --keystone-auth --auth-url https://keystone.rc.nectar.org.au:5000/v2.0 --keystone-tenant-separator=: &

  3. Start your favourite FTP client, for example FileZilla. If you don’t have FileZilla, you can install it with
    sudo apt-get install filezilla
    Your login details for the FTP client are as follows:
    • Server (or Host): 127.0.0.1
      This IP address is the one of your local computer (not an IP address you can use to access your computer from outside though) — Remember, you are connecting to the FTP server we just launched on our local machine.
    • User: <tenant>:<username>
      Replace <tenant> and <username> with your OpenStack credentials.
    • Password: Your OpenStack password
    • Port: 2021
  4. You can now browse your files and drag and drop them between both windows, as you are used to when using the FTP client. On the left side, you can see the file system of your local computer, and on the right side, you can see your object store.

image

You will have to do steps 2 and 3 every time you restart your local computer.

Linux [Advanced]: Access with CloudFuse

CloudFuse is a lightweight open source application that may be installed on Linux servers to create a local mount point to a customers existing Cloud Files container.

You can find more information and documentation about cloudfuse on the cloudfuse github page

To set up cloudfuse for access to your object store, launch a terminal on your local computer and follow the steps below.

  1. Install necessary packages
    sudo apt-get install build-essential libcurl4-openssl-dev libxml2-dev libssl-dev libfuse-dev git

  2. Clone the CloudFuse source
    cd
    git clone git://github.com/redbo/cloudfuse

  3. Compile the source code and install it
    cd cloudfuse
    ./configure
    make
    sudo make install
    modprobe fuse

  4. Create a file called $HOME/.cloudfuse
    sudo nano $HOME/.cloudfuse

  5. Put the following contents into your file (replace <your-tenant-name>, <your-username> and <your-password> with your own credentials):
    authurl=https://keystone.rc.nectar.org.au:5000/v2.0/    
    tenant=    
    username=    
    api_key=    
    cache_timeout=60
    
    
  6. Set the right permissions:
    chmod 600 $HOME/.cloudfuse

  7. Mount your Swift storage with the following commands
    sudo mkdir /mnt/cloudfiles/
    cloudfuse /mnt/cloudfiles/
    Of course you may choose another name instead of cloudfiles, and/or create the directory somewhere else.

  8. You should now be able to access your files on the directory /mnt/cloudfiles just as if they were on your local computer.
    image ls -l /mnt/cloudfiles/
    Or open your file explorer to find your mounted drive there (see image to the right). You may even drag and drop files to and from your object storage, as if it was a regular hard drive or folder on your computer.

  9. Every time you restart your computer, you will have to re-connect the object store with the command:
    cloudfuse /mnt/cloudfiles/

Troubleshooting:

Your username (of your local computer) may need to be part of the fuse user group. This can be accomplished with the following command in your computer’s terminal:

sudo usermod -a -G fuse <username>

All platforms: Access with the openstack command

You may also use the openstack command to list the files in our object storage that we just created. This will be discussed in Module 10.

Access via APIs

We will discuss more options on how to connect to the object store in Module 10.

Volume storage

This section is going to go through the steps required to use Volume storage from your instance. You are going to

  1. Create a Volume storage
  2. Attach the Volume to the instance
  3. Prepare the Volume for use within your instance
image

You need to submit an allocation request in order to be able to use volumes. The trial account does not include quota for volumes.

You can only use volume storage from instances running in the same Availability Zone which you have also chosen with your allocation. Therefore, you need to select the same Availability Zone when launching your instance. If you haven’t done so when launching the instance, you’ll have to launch a new one, as described in this earlier section, and select your allocation zone in the Availability zone tab.

There are two options how to manage volumes: (1) on the Dashboard or (2) via command line.

In this module, we will only cover managing volumes via the Dashboard. Module 10 will show how we can manage volumes with commands.

Step 1: Creating the volume.

Before you can use a volume, you have to create it.

Go to Compute > Volumes and click on the + Create Volume button on the top right.

image

A window opens in which you can give your Volume a name, and add a description.

image

There are also other options which you can specify:

Volume Source: You can either choose to create an empty volume, or create a volume from an image. The latter is useful if you want to create a volume which already has specific data on it. It is possible to save (or back up) a volume by creating an image of it - we will see how to do this in Module 9.

You may want to restore a volume that you previously have saved as an image, or someone else may have sent you an image with data on it which you should have access to. When you create a volume from an image, your volume will have all the data on it that was on the volume when the image was created of it.

In this tutorial, we will create an empty volume.

Type: This field isn’t currently used, it’s there for future expansion. Leave this at “No volume type”.

Size(GB): Type in the desired size (in GB) of your volume.

Availability zone: Select your availability zone. It must be the same in which your instance is running. Creation of the volume will only work if you have a volume allocation at this availability zone.

Click “Create volume” and you will see your new volume in the list of volumes. The Status should switch to “Available” after it has been created. This can take a few minutes.

Step 2: Attaching / Detaching volumes.

Before you can access the volume, you have to attach it to an instance. On the Dashboard, go to “Compute → Volumes” in the left navigation pane if you are not already on the page which shows the list of your volumes. On the right side next to your volume there is a drop-down box.
Select Edit attachments.

image

A window will open in which you can select your instance. If you don’t see any instance, you have no instance running in the same Availability Zone. In this case, create one as described in this earlier section.

image

Select your instance and click Attach Volume. The status of your instance will switch to “Attaching”, and when it has been attached successfully, it should display “In-Use”. You should also see the Server and Device that the volume is attached to. In the example below, it is attached to MyFirstInstance and to the device /dev/vdc.

image

In order to detach the volume again, you would go to the drop-down next to your volume again and select Edit attachments. A window opens in which you may detach the volume again. Your data won’t be lost, it will only be detached from the particular instance. You can re-attach it later again to any instance.

image

Step 3: Preparing your volume for use.

After you have created your Volume and attached it to your instance you will have to take a few extra steps to prepare that volume for use: you will have to (1) format it and (2) mount it.

image

The section about On-Instance Storage has more information about formatting and mounting drives.

Log in to your instance via ssh (or use a terminal on your remote desktop) to type in the following commands.

Step 3.1. Look up where your volume has been created

On your Dashboard, go to Compute > Volumes and you should see your volume. In the column Attached to it should display something like

“Attached to <YourInstanceName> on /dev/vdc”

It may also be attached to /dev/vdb, /dev/vdd or something else. Take note of the device name that it has been attached to and use this instead of /dev/vdc in the following.

Step 3.2: Format the volume.

image

Formatting erases all data. All your data on this volume will be lost. You should do this only the first time you use the volume, or if you are sure that you want to erase all data. If you don’t want to erase the data, skip to the next step.

A new Volume may not have a file system (depending on how it was created) and you need to create one before mounting.

The exact command syntax is depends on the instance’s operating system and the type of filesystem you require. In this example we will create an ext4 file system (a popular file system on linux) for an Ubuntu 14.10 instance.

To format the drive, type:

sudo mkfs.ext4 /dev/vdc

Display information about your disks with

sudo lsblk -f

or

sudo parted -l

This should now show that /dev/vdc has the ext4 file system.

Step 3.3: Mount the volume.

Mounting a disk has already been discussed in detail for the secondary drive of the Ephemeral storage, so at this point, only a summary is provided:

To mount your volume, you will have to create a folder first to which the device will be mounted, e.g.:

mkdir ~/MyVolumeStore

Then, mount this folder:

sudo mount /dev/vdc ~/MyVolumeStore

image

This will mount the disk if the type of file system can be guessed. Otherwise you may have to specify the file system type with the -t option, e.g.
sudo mount /dev/vdc ~/MyVolumeStore -t ext4.

If you did not create and format the Volume then you can look up the filesystem type:
sudo parted -l

You can now access your volume on ~/MyVolumeStore (MyVolumeStore in your home directory). You may not have write permission on the Volume yet, you can allow write access for the user ubuntu with:

sudo chown ubuntu ~/MyVolumeStore

Alternatively, you may grant write access to the <user group> (replace <user group> with the name of the user group, e.g. ubuntu):

sudo chgrp <user group> ~/MyVolumeStore
sudo chmod g+rwx ~/MyVolumeStore

Copying files across

You may want to easily access files on your ephemeral or volume storage from your local office computer. We will discuss two ways to do this:

  1. Using a SFTP client

  2. On the command line, using a secure copy command

Option 1: Using a SFTP client

You may use any FTP client which also supports the secure SFTP protocol, which uses ssh to secure the connection. There are many clients you may use. In this tutorial, we are going to describe the setup for the popular FileZilla FTP client.

Step 1. Install FileZilla

Step 2. Enable ssh agent to handle your private key.

It is possible to add a key file to the FileZilla client (On Edit > Settings, or a Mac, this is FileZilla > Settings, and then add the key file on Connection > SFTP), and then use the ‘Interactive’ logon type. However, when adding the key directly to FileZilla, it is converted to an unprotected one. This is not recommended. So we will use another method: We will use a SSH authentication agent, which holds your private keys in memory, already decoded, so that you can use them without needing to type a passphrase.

Step 3. Configure your site in FileZilla

Open FileZilla and go to File > Site Manager. image

Specify the following details:

On the left-hand panel, you may rename your site to something more meaningful, e.g. “NeCTAR”.

Click Connect to connect to your instance and close the site manager. Whenever you want to reconnect, you have to open the Site Manager again. You can also access it via the symbol on the top right of the window.

Then, you will be asked whether you want to trust the host. You can answer with Yes. image

You should now be connected to your instance and see your files:

image

You can drag and drop between them or drag and drop to/from file windows on your computer.

When you are done, close the window to disconnect.

Option 2: Use the command line

Instructions vary between Linux/Mac and Windows. Follow the instructions for your OS below.

Windows

You may use the PuTTY command line tool pscp.

First, open the Windows Command prompt: On Windows earlier than 8, click on the Start button locate the Search filed. On newer Windows systems, click Search in the right-hand pane. Type “cmd” in the search field and your command line prompt will open.

Download pscp.exe from the PuTTY website and save it in your C:\WINDOWS\system32 folder.

image

Saving pscp.exe in the system32 folder is the easiest way to do it. If you cannot access your system32 folder, you may add the folder where you downloaded pscp.exe to the systems PATH environment variable. Say you have saved pscp.exe for example in D:\Documents\pscp.exe. Go to your Windows command line that you just opened, and type:

set PATH=%PATH%;"D:\Documents\pscp.exe"

Now go to your Windows command prompt.

The pscp.exe command takes two paths as arguments:

image

If you have another username than ubuntu, replace ubuntu in the following commands with your user name.

To copy a file to your instance, go to your windows command prompt and type:

pscp.exe -i <path-to-ssh-key> <local-file-path> ubuntu@NNN.NNN.NNN.NNN:<instance-file-path>

replacing the N’s with your IP address. You need to specify the path where you have saved your nectar ssh key in <path-to-ssh-key>.

image

Note that if you are using Pageant (as described in the Appendix), you can skip the option -i <path-to-ssh-key> in all pscp commands.

For example, to copy the local file c:\documents\foo.txt to your instance and save it there in the folder /data, if your private key is saved in c:\documents\Nectar_Key.ppk, you would type:

pscp -i c:\documents\Nectar_Key.ppk c:\documents\foo.txt ubuntu@NNN.NNN.NNN.NNN:/data

image

Note that the local path to c:\documents\foo.txt has to be given in Windows syntax, using backslash \ instead of forward slash /. The <instance-file-path> on your instance however uses the Linux notation with the forward slash / separator.

Note: If you leave <instance-file-path> empty, it will copy your file into your ubuntu users home directory.

To copy a file from your instance to your local computer, you have to swap the order of the <local-file-path> and ubuntu@NNN.NNN.NNN.NNN:<instance-file-path> in the command above:

pscp.exe -i <path-to-ssh-key> ubuntu@NNN.NNN.NNN.NNN:<instance-file-path> <local-file-path>

Note: If you use ”.” as <local-file-path>, it will copy your file into your current working directory, which is the folder your PuTTY terminal is currently operating from.

For example, if you want to copy the file /data/foo.txt to your C:\documents folder on your local computer, and your private ssh key is located in C:\documents\Nectar_Key.ppk:

pscp.exe -i C:\documents\Nectar_Key.ppk ubuntu@NNN.NNN.NNN.NNN:/data/foo.txt C:\documents

image

If the pscp command is not working, make sure you have no error in the paths to your ssh key or your OpenRC file. On Windows, white spaces are problematic. If you path has white spaces, put the whole path in quotation marks, e.g. “D:\My Files\pscp.exe”. Also, note that you have to use the backslash \ in windows, instead of the forward slash / on Linux systems.

Linux or Mac OS X

You can use the scp command to copy files across.

The scp command takes two paths as arguments:

image

If you have another username than ubuntu, replace ubuntu in the following commands with your user name.

To copy a file to your instance:

scp -i ~/.ssh/Nectar_Key <local-file-path> ubuntu@NNN.NNN.NNN.NNN:<instance-file-path>

replacing the N’s with your IP address.

Note: If you leave <instance-file-path> empty, it will copy your file into your ubuntu users home directory.

For example, if you want to copy the file /home/username/foo.txt to your /data folder on the instance:

scp -i ~/.ssh/Nectar_Key /home/username/foo.txt ubuntu@NNN.NNN.NNN.NNN:/data

To copy a file from your instance to your local computer, you have to swap the order of the <local-file-path> and ubuntu@NNN.NNN.NNN.NNN:<instance-file-path> in the command above:

scp -i ~/.ssh/Nectar_Key ubuntu@NNN.NNN.NNN.NNN:<instance-file-path> <local-file-path>

Note: If you use ”.” as <local-file-path>, it will copy your file into your current working directory.

For example, if you want to copy the file /data/foo.txt to your /home/username/ folder on your local computer:

scp -i ~/.ssh/Nectar_Key ubuntu@NNN.NNN.NNN.NNN:/data/foo.txt /home/username/

Summary

Well done!!

You have completed the probably longest and hardest Module and are on your way to becoming an experienced Cloud user.

This module has provided hands-on instructions on how to launch an instance and connect to it. Two ways to connect have been presented: (a) with the command line (ssh) and (b) to a remote desktop. After having connected, we have gone through the first configurations necessary (e.g. updating the system) and then we went on to instructions on how to set up the storage so it can be accessed from the instance and remotely from the office computer.

Topics have covered:

You may be interested in taking a look at the Appendix which provides some more information and instructions for some troubleshooting.

You are now ready to go on with Module 8 which will discuss a few security-related topics.

Appendix

A number of topics are covered in the appendices. Scroll through the document to find what you may be looking for.

Create a new security group

On the Dashboard:

  1. Go to Access and Security

  2. Create Security Group

  3. Name it something meaningful, and add any description

  4. Click Create Security Group. You will see the new group in the list now.

  5. Click Manage rules at the right of your group.

  6. At the top right of the page, click Add rule

  7. You probably want an Incoming TCP rule.

  8. If you want to allow access from any computer, type 0.0.0.0/0 in the CIDR

    image

    The part before the slash ”/” is the address for localhost and the part after the slash specifies the subnet mask — you don’t need to worry about the meaning in this tutorial, but look it up online, e.g. google, if you want to restrict access to computers from only a certain range of IP addresses, e.g. only computers from within a university.

After you have created the security group, you can add it to the instance. Go to Compute > Instances and in the right-hand drop down menu for your instance, select Edit Security Groups.

Troubleshooting Linux and Mac: Problems connecting

Check which ports are open on your system:

If you get the error:

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@    
@  WARNING: UNPROTECTED PRIVATE KEY FILE! @    
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@    
Permissions 0664 for '<yourkey>' are too open.    
It is required that your private key files are NOT accessible by others.    
This private key will be ignored.    
bad permissions: ignore key: <yourkey>    
Permission denied (publickey).

Then you have changed the access rights to the key while copying it around. Fix this with:

chmod go-rw <yourkey>

Converting ssh keys from Linux for PuTTY

If you have generated your ssh key on a Linux system and now want to use it in Putty, you have to convert the keys first. If you have generated your key file with putty, you don’t need to do this. To convert the key, follow these steps:

  1. Start PuTTYGen.

  2. Click on the menu Conversions and select Import key.

  3. Select your private key and enter the passphrase.

  4. Click on the button Save private key and save your key file.

You may also want to simply create a separate set of keys for PuTTY (see this section). In this case, you have to add the new key as an authorized key on the instance. This is explained here in this Appendix.

Windows: Using Pageant to handle your ssh key

You may use a SSH authentication agent, which holds your private keys in memory, already decoded, so that you can use them without needing to type a passphrase.

PuTTY also offers a tool called Pageant, which is the PuTTY SSH authentication agent. Download it on the PuTTY website. If you have not generated your ssh key with PuTTY in the first place, you will also need to convert it to the PuTTY format. You can do this using the tool PuTTYgen (you can find instructions on how to do this in this Appendix).

To load your ssh key with Pageant:

Step 1.

image Start up Pageant (by clicking on pageant.exe which you downloaded). In your system tray, you’ll see the Pageant icon appear: A computer symbol with a hat on.

Step 2.

image Right-click the icon and select Add Key and select your private key (PPK) file. Enter your pass phrase. Now, your key is loaded in the Background and you can use FileZilla via the secure FTP protocol SFTP in the next step.

image

Don’t forget to close Pageant when you’re done! Otherwise any user with access to your computer can easily gain access with your ssh key without having to type the ssh key passphrase. To close pageant, go to the system tray icon, right click and Exit pageant.

See also the pageant tutorial on digitalocean.

Creating a new user account

If you want to create a new user account on your VM, type in the following command in your ssh terminal:

sudo adduser <username>

replacing <username> with the username you would like to create. It will prompt you to create a password for this user, and type in their name and contact details.

image

If you would like to just create the user without specifying password and details, use

sudo useradd <username>

instead. You then can set the user password with:

sudo passwd <username>

It is recommended you use a random password initially, and then give it to the user, with the strong recommendation of changing it (a user cannot set their own initial password themselves, hence this initial step is required). Users can change their own password with:

passwd

You have now created a user with minimal privileges. For example, the user cannot perform administration tasks which require the sudo command prefix. If you would like to add privileges for this user, you can add it to certain privileges groups, e.g. the sudo group:

sudo adduser <username> sudo

The user can then use the sudo command to perform administrative tasks. They will have to type in their own password to execute sudo commands (this is a security measure in case they left their keyboard unattended).

To check the right groups the user is assigned to, type

id <username>

image

Each user should have their own ssh keys. Ask them to create it, using their passphrase of choice, and then send you their public key. Add their public key to your VM by following instructions here in this appendix.

Adding a new ssh key

If you want to use a new key to connect to the instance, you have to add it to the file authorized_keys on the user’s account. If you are adding another key for the ubuntu user, the <username> in the command below is ubuntu, otherwise it is the user name you want to add a key for.

First, create the .ssh directory for the user, if it does not exist yet (if it exists, this command will return an error):

sudo mkdir /home/<username>/.ssh

Then, edit the file authorized_keys

sudo nano /home/<username>/.ssh/authorized_keys

Paste the contents of your public key as one line into this file.

image

Tip: Copy the content of the public key by opening it with a text editor, marking all text and copying with CTRL+C. Then, go to your PuTTY Window which has the editor open for authorized_keys. Pasting the content will not work with CTRL+V on PuTTY, but instead, clicking the right or middle mouse button should paste the content.

Similarly, if you want to remove a key, you have to delete the line for that key from the authorized_key file on your instance.

Make sure that the directories and files have the correct ownership and access:

sudo chown -R <username> /home/<username>/.ssh
sudo chgrp -R <username> /home/<username>/.ssh
sudo chmod 700 /home/<username>/.ssh
sudo chmod 500 /home/<username>/.ssh/authorized_keys

SSH Tunnelling explained

SSH Tunnelling is best explained with an example. Imagine you’re on a private network with very conservative firewall rules which don’t allow connections to a specific server. For example, blockedsite.com is not accessible from your network. You can of course ask your system administrator to free up this site for you, but you may have to wait for the approval. To get around this firewall rule we can redirect our connection via a port that does not block this website: We can create a tunnel through a server which is not on our network and which can access blockedsite.com. We call this server the gateway.

image

Let’s say blockedsite.com uses the http protocol (as most websites do) through Port 80. Instead of connecting our port 80 to the blocksite.com’s port 80 (which would not work), we will redirect all traffic from our port 80 through the secure ssh port 22. This one will not be blocked by the firewall, and nobody will be able to decipher the traffic going through it (unless they get access to your private key and passphrase of course). On the gateway, our connection will come in at Port 22, and the ssh server will then redirect the traffic to port 80 on blockedsite.com. We call this technique local port forwarding. Of course, the connection between the gateway and the server of blockedsite.com is not encrypted; only the tunnel between our computer and the gateway is.

The technique of local port forwarding through an ssh tunnel can also be used to create a secure connection all the way to the destination server, so that there is no point at which the connection is not encrypted. In this case, the gateway is also the destination server itself.
As an example, we are going to discuss the setup for ssh tunneling on port 5900 on our instance (the default VNC port), using the gateway localhost. The example is for Mac and Linux. Observe that the gateway is localhost, because when we specify the destination server, it is also the gateway in this case (this can be a bit confusing, because you would guess we specify the gateway first… but actually we first specify the destination server, and then the gateway through which we want to create the tunnel. From the gateway’s point of view, the destination server is localhost). The tunnelled service will be available on our localhost, port 6000 (we use different numbers so you can distinguish it in the text below). We could do this for any other port numbers.

  1. On your machine, create a tunnel from your machine (port 5900) to the other machine (port 5900)
    ssh -i <your-key-name> -L 6000:localhost:5900 -N -f -l ubuntu NNN.NNN.NNN.NNN
    replacing NNN.NNN.NNN.NNN with the IP address of the instance.

  2. On the instance: Install and start any service that will listen on port 5900 (e.g. the VNC Server).

  3. A quick test should show you that you cannot actually directly connect to port 5900 on the instance:
    telnet 144.6.227.50 5900
    This will try connect you to the remote port (without any useful application, but it’s a good way to test whether the port is open). The shell should just say “trying to connect…” but nothing happens: Because we have not allowed access to port 5900 from outside… only via the SSH tunnel! And the ssh tunnel is accessible via our localhost. End the connect trying with Ctrl+C. Instead, try:
    telnet localhost 6000
    or
    nc -v -w 5 localhost -z 6000
    And you should see a command prompt (no error). So now you are talking to the port and can specify commands specific to the server, but we don’t really want to do this now.
    To quit the telnet command prompt: Hit Ctrl+] (control and the right-bracket). That will return you to a telnet command prompt. From there you can enter the quit command (typing q is sufficient) and you will be returned to your usual Windows command prompt.
    If this still does not work, make sure there is a server listening on port 5900. Check this with:
    sudo netstat -tulpn
    There should be your service listed as “Listening” on the port you have meant it to provide its service. If your service is not listening on that port, it won’t be possible to connect to it either.

image

The ssh -L options are explained as follows.

Syntax:
-L <local-port-to-listen>:<remote-host>:<remote-port> <gateway>

  • -N : Do not execute a remote command i.e. just forward ports.

  • -f: Requests ssh to go to background just before command execution. Once password supplied it will go to background and you can use prompt for type commands on local system.

  • -l ubuntu: ubuntu is the user to log in as on the remote machine (sshserver.mydomain.com). Note: The -f -N options in the above command cause the tunnel to run in the background: once you have typed in the password, the command line prompt returns to your local shell. In this case, if you ever want to close your ssh connection, you have to kill the connection:
    ps aux | grep ssh
    Then, find your ssh tunneling command and note the number on the left, after your user name. To kill (close) the ssh tunnel, type:
    kill <the-ssh-tunnel-process-number>
    If you don’t use the -N -f options, you will be logged into an ssh terminal to your instance, and can use it as usual to type commands on your instance. If you type logout or close the terminal, the ssh tunnel will close as well.

This is the end of Module 7. Continue with Module 8