Operate Resources in Public and Private Clouds Thanks to OpenStack

To follow the lab you’ll need a Linux (tested under Ubuntu 20.04) with, at least, 8 GiB of RAM and 2 CPU. We are going to use Linux Dell R320 from room D011. If you have a different version of Ubuntu installed – use lsb_release -d to check it out – you should start a proper VM using, for instance, the Vagrantfile given in the Appendix. Alternatively, you can also use your own machine.

The lab makes use of Snap microstack: OpenStack in a Snap that you can run locally on a single machine. Snap is the Canonical’s App delivery mechanism. It enables developers to bundle all dependencies into a single app package. And so does Snap microstack for an all-in-one OpenStack on your machine.

An all-in-one OpenStack means that your machine will contain both services to operate and host virtualized resources. For instance, the nova-conductor to operate the boot of a VM, and nova-compute to host the VM. This is a good setup for a lab, but not for production. There are several other options such as DevStack, Puppet-OpenStack or Kolla-ansible and all matters. But, Snap microstack takes only 2 minutes to deploy OpenStack (instead of 30 minutes for other options).

First thing first, you have to connect to the Lab machine as invité and create your account.

Lab machines aren’t available publicly unfortunately. They are hidden behind Polytech private network. Fortunately, a frontend is available from the outside world at the address 163.172.58.84 that let us reach Lab machines. SSH on that frontend using the login and password given in the pad of the collaborative editor and then, SSH on your Lab machine. Password for invité on the lab machine is d012.

ssh -l <login> 163.172.58.84
ssh -l invité 192.52.86.XXX

From there, create your account on the Lab machine (use a valid univ-nantes email address).

sudo useradd -g 100 -G adm,sudo -m -s /bin/bash -c \
  '<Prénom> <Nom> <prénom.nom@etu.univ-nantes.fr>' <login>
sudo passwd <login>

And change user ID from invité to your <login>.

su <login>

The rest of this lab proceeds on the Lab machine.

Get the resources of the lab at https://github.com/Marie-Donnie/lectures/blob/master/2021-2022/os-polytech/tp.tar.gz.

curl https://github.com/Marie-Donnie/lectures/raw/master/2021-2022/os-polytech/tp.tar.gz -o tp.tar.gz -L
tar xzf tp.tar.gz
cd lab-os

The archive contains:

setup.sh

Script that sets up the lab.

teardown.sh

Script that uninstalls the lab.

rsc

Resource directory with bash scripts useful for the lab.

Install snap.

sudo apt update
sudo apt install snapd

Ensure OpenStack microstack is not already installed and remove it otherwise.

snap info microstack | fgrep -q installed && sudo snap remove --purge microstack

Install the last version from the snap store.

sudo snap install --channel=latest/beta microstack --devmode

Execute the setup.sh file with sudo to initialize OpenStack (setup networks, flavors, images, …).

sudo ./setup.sh

snap services microstack|sort

One service deployed is the OpenStack dashboard (Horizon). On your own machine, horizon is reachable from the web browser at https://<ip-of-your-lab-machine> with the following credentials:

• login: admin
• password: keystone

From here, you can reach Project > Compute > Instances > Launch Instance and boot a virtual machine given the following information:

• a name (e.g., horizon-vm)
• an image (e.g., cirros) and set the Create New Volume to “No”
• a flavor to limit the resources of your instance (we recommend m1.tiny)
• and a network setting (must be test)

You should select options by clicking on the big arrow on the right of each possibility. When the configuration is OK, the Launch Instance button should be enabled. After clicking on it, you should see the instance in the Active state in less than a minute.

Now, you have several options to connect to your freshly deployed VM. For instance, after clicking on its name, Horizon provides a virtual console under the Console tab. So, you can use the following credentials to access the VM:

• login: cirros
• password: gocubsgo

However, as a real DevOps, you will prefer to access to your VM by the command line interface …

While Horizon is helpful to discover OpenStack features, this is not the tool of choice for an operator. An operator prefers command line interface 😄. You are lucky, OpenStack provides one.

All operations to manage OpenStack are done through one unique command line, called openstack <service> <action> .... Doing an openstack --help displays the really long list of services/possibilities provided by this command. The following gives you a selection of the most often used commands to operate your Cloud:

List OpenStack running services

openstack endpoint list

List images

openstack image list

List flavors

openstack flavor list

List networks

openstack network list

List computes

openstack hypervisor list

List VMs (running or not)

openstack server list

Get details on a specific VM

openstack server show <vm-name>

Start a new VM

openstack server create --image <image-name> --flavor <flavor-name> --network <network-name> <vm-name>

View VMs logs

openstack console log show <vm-name>

$ openstack endpoint list
Missing value auth-url required for auth plugin password

openstack server list --os-auth-url=http://193.52.86.133:5000/v3/ \
                        --os-username=admin \
                        --os-password=keystone \
                        --os-project-name=admin \
                        --os-user-domain-name=Default \
                        --os-project-domain-id=default

source ./admin-openrc.sh

$ env|fgrep OS_|sort

OS_AUTH_URL=http://<ip-of-your-lab-machine>:5000/v3/
OS_IDENTITY_API_VERSION=3
OS_INTERFACE=public
OS_PASSWORD=keystone
OS_PROJECT_DOMAIN_ID=default
OS_PROJECT_ID=2bad71b9246a4a06a0c9daf2d8896108
OS_PROJECT_NAME=admin
OS_REGION_NAME=microstack
OS_USER_DOMAIN_NAME=Default
OS_USERNAME=admin

openstack server create \
  --image cirros \
  --flavor m1.tiny \
  --network test \
  cli-vm

Then, display the information about your VM with the following command:

openstack server show cli-vm

Note in particular the status of your VM (and how to extract that information from the command line with the -c and -f).

openstack server show cli-vm -c status -f json

This status will go from BUILD: OpenStack is looking for the best place to boot the VM; to ACTIVE: your VM is running. The status could also be ERROR if you are experiencing hard times with your infrastructure.

$ openstack server create --help | fgrep -A 6 "output formatters:"
output formatters:
  output formatter options

  -f {json,shell,table,value,yaml}, --format {json,shell,table,value,yaml}
                        the output format, defaults to table
  -c COLUMN, --column COLUMN
                        specify the column(s) to include, can be repeated

A VM in ACTIVE state still has to go through the boot process and init. Hence, you may still have to wait for one minute or two that your VM finishes to boot. You can check that your VM finished to boot by looking at its logs with openstack console log show cli-vm. A CirrOS VM finished to boot when last lines are:

=== cirros: current=0.4.0 latest=0.4.0 uptime=29.16 ===
  ____               ____  ____
 / __/ __ ____ ____ / __ \/ __/
/ /__ / // __// __// /_/ /\ \
\___//_//_/  /_/   \____/___/
   http://cirros-cloud.net


login as 'cirros' user. default password: 'gocubsgo'. use 'sudo' for root.
cli-vm login:

Any cirros VMs share the same credentials (i.e., cirros, gocubsgo) which is a security problem. As a IaaS DevOps, you want that only some clients can SSH on the VMs. Fortunately, OpenStack helps with the management of SSH keys. OpenStack can generate a SSH key and push the public counterpart on the VM. Therefore, doing a ssh on the VM will use the SSH key instead of asking the client to fill the credentials.

Make an SSH key and store the private counterpart in ./admin.pem. Then, give that file the correct permission access.

openstack keypair create --private-key ./admin.pem admin
chmod 600 ./admin.pem

Start a new VM and ask OpenStack to copy the public counterpart of your SSH key in the ~/.ssh/authorized_keys of the VM (i.e., note the --key-name admin).

openstack server create --wait --image cirros \
  --flavor m1.tiny --network test \
  --key-name admin cli-vm-adminkey

Attach it a floating IP.

openstack server add floating ip \
  cli-vm-adminkey \
  $(openstack floating ip create -c floating_ip_address -f value external)

Now you can access your VM using SSH without filling credentials.

openstack server ssh cli-vm-adminkey \
  --login cirros \
  --identity ./admin.pem

ssh -i ./admin.pem -l cirros $(openstack server show cli-vm-adminkey -c addresses -f value | sed  -Er 's/test=.+ (10\.20\.20\.[0-9]+).*/\1/g')

openstack server show cli-vm-adminkey -c addresses -f value | sed  -Er 's/test=.+ (10\.20\.20\.[0-9]+).*/\1/g'

Contextualizing is the process that automatically installs software, alters configurations, and does more on a machine as part of its boot process. On OpenStack, contextualizing is achieved thanks to cloud-init. It is a program that runs at the boot time to customize the VM.

You have already used cloud-init without even knowing it! The previous command openstack server create with the --identity parameter tells OpenStack to make the public counterpart of the SSH key available to the VM. When the VM boots for the first time, cloud-init is (among other tasks) in charge of fetching this public SSH key from OpenStack, and copy it to ~/.ssh/authorized_keys. Beyond that, cloud-init is in charge of many aspects of the VM customization like mounting volume, resizing file systems or setting an hostname (the list of cloud-init modules can be found here). Furthermore, cloud-init is able to run a bash script that will be executed on the VM as root during the boot process.

Vagrant is an open-source software to build development environment using a declarative approach. In addition to Vagrant, we recommend to install the Vagrant Libvirt Provider in order to take advantage of the hardware virtualization.

Copy the following content into a file named “Vagrantfile”. Review the content, especially the number of CPU, memory size and the private/public network.

# -*- mode: ruby -*-
# vi: set ft=ruby :

Vagrant.configure("2") do |config|
  # Ensure that the box is built with VirtIO Network Interface for
  # libvirt.  This is not the case of generic/ubuntu2004 which then
  # leads to issues with ovn geneve in OpenStack.
  config.vm.box = "peru/ubuntu-20.04-server-amd64"
  # config.vm.box = "peru/ubuntu-18.04-server-amd64"
  config.vm.box_version = "20201107.01"
  config.vm.box_check_update = false
  # Horizon dashboard
  config.vm.network :forwarded_port, guest: 80, host: 8080, host_ip: "*"
  # Horizon Spice Javascript console
  config.vm.network :forwarded_port, guest: 6082, host: 6082, host_ip: "*"
  # Optionally set the following private ip as a public ip of your
  # infrastructure
  config.vm.network :private_network, ip: "192.168.121.245"
  config.vm.synced_folder "./", "/vagrant", type: "rsync"

  config.vm.provider :virtualbox do |vb|
    # vb.memory = "8192"  # Minimum
    vb.memory = "16384"   # Much better
    vb.cpus = 4
    vb.gui = false
  end

  config.vm.provider :libvirt do |lv|
    # lv.memory = "8192"  # Minimum
    lv.memory = "16384"   # Much better
    lv.cpus = 4
    lv.nested = true
    lv.cpu_mode = "host-passthrough"
  end
end

Then start and connect to the lab machine with the following commands.

vagrant up --provider=libvirt  # remove `--provider=libvirt` if you don't support vagrant-libvirt
vagrant ssh

You can also synchronize the content of your current directory with the content of the /vagrant directory in the lab machine with the vagrant rsync command.

First start a VM with mpv and libcaca to output videos in the terminal. Download a video from youtube, so you can then broadcast it on a loop with mpv --vo caca --no-audio --loop vid.mp4 after SSH on it.

openstack server create migrate-vm \
          --wait --flavor m1.mini --network private \
          --image debian-10 \
          --key-name ronana \
          --user-data ./rsc/live-migrate-do-it.sh
ALLOCATED_FIP=$(openstack floating ip create -c floating_ip_address -f value external)
openstack server add floating ip migrate-vm "${ALLOCATED_FIP}"
echo "${ALLOCATED_FIP}"

Here is the script to setup the VM.

#!/usr/bin/env bash
apt update; apt install -y python3-pip mpv libcaca-dev
pip3 install --upgrade youtube_dl
# Pick a video under creative commons
youtube-dl --recode-video mp4 --verbose \
           --user-agent "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" \
           --output vid.mp4 \
           "https://www.youtube.com/watch?v=ZXsQAXx_ao0"

Then while broadcasting, live migrate your VM and also take a look at its state.

watch openstack server show migrate-vm -c 'OS-EXT-SRV-ATTR:host' -c 'OS-EXT-STS:task_state'
openstack server migrate migrate-vm --live-migration --block-migration