ganeti admin manual

248 Pages·2014·2 MB·English

Page | 1
GANETI
ADMIN MANUAL
CONTENT

Ganeti installation tutorial……………………………………………………………………………3
Page | 2
Ganeti administrator’s guide….……………………………………………………………………17
Ganeti walk-through..…………………………………………………………………………………50
Security in Ganeti………………………………………………………………………………………..69
Virtual cluster support..………………………………………………………………………………73
Moving instances between clusters…………………………………………………………….75
Ganeti Man Pages……………………………………………………………………………………….77
Ganeti installation tutorial
Documents Ganeti version 2.9
Page | 3
Contents
 Ganeti installation tutorial
o Introduction
o Installing the base system and base components
 Hardware requirements
 Installing the base system
 Hostname issues
 Installing The Hypervisor
 Xen settings
 Selecting the instance kernel
 Installing DRBD
 Installing RBD
 RADOS Cluster
 Kernel Modules
 Userspace Utils
 Configuration file
 Other required software
o Setting up the environment for Ganeti
 Configuring the network
 Configuring LVM
 Installing Ganeti
 What gets installed
 Installing the Operating System support packages
 Initializing the cluster
 Hypervisor/Network/Cluster parameters
 Joining the nodes to the cluster
 Separate replication network
 Testing the setup
 Cluster burnin
o Further steps
Introduction
Ganeti is a cluster virtualization management system based on Xen or KVM. This document
explains how to bootstrap a Ganeti node (Xen dom0, the host Linux system for KVM), create
a running cluster and install virtual instances (Xen domUs, KVM guests). You need to repeat
most of the steps in this document for every node you want to install, but of course we
recommend creating some semi-automatic procedure if you plan to deploy Ganeti on a
medium/large scale.
A basic Ganeti terminology glossary is provided in the introductory section of the Ganeti
administrator’s guide. Please refer to that document if you are uncertain about the terms we
are using.
Ganeti has been developed for Linux and should be distribution-agnostic. This documentation
will use Debian Squeeze as an example system but the examples can be translated to any
Page | 4
other distribution. You are expected to be familiar with your distribution, its package
management system, and Xen or KVM before trying to use Ganeti.
This document is divided into two main sections:
 Installation of the base system and base components
 Configuration of the environment for Ganeti
Each of these is divided into sub-sections. While a full Ganeti system will need all of the steps
specified, some are not strictly required for every environment. Which ones they are, and
why, is specified in the corresponding sections.
Installing the base system and base components
Hardware requirements
Any system supported by your Linux distribution is fine. 64-bit systems are better as they can
support more memory.
Any disk drive recognized by Linux (IDE/SCSI/SATA/etc.) is supported in Ganeti. Note that
no shared storage (e.g. SAN) is needed to get high-availability features (but of course, one can
be used to store the images). Whilte it is highly recommended to use more than one disk drive
in order to improve speed, Ganeti also works with one disk per machine.
Installing the base system
Mandatory on all nodes.
It is advised to start with a clean, minimal install of the operating system. The only
requirement you need to be aware of at this stage is to partition leaving enough space for a big
(minimum 20GiB) LVM volume group which will then host your instance filesystems, if you
want to use all Ganeti features. The volume group name Ganeti uses (by default) is xenvg.
You can also use file-based storage only, without LVM, but this setup is not detailed in this
document.
If you choose to use RBD-based instances, there’s no need for LVM provisioning. However,
this feature is experimental, and is not yet recommended for production clusters.
While you can use an existing system, please note that the Ganeti installation is intrusive in
terms of changes to the system configuration, and it’s best to use a newly-installed system
without important data on it.
Also, for best results, it’s advised that the nodes have as much as possible the same hardware
and software configuration. This will make administration much easier.
Hostname issues
Note that Ganeti requires the hostnames of the systems (i.e. what the hostname command
Page | 5
outputs to be a fully-qualified name, not a short name. In other words, you should use
node1.example.com as a hostname and not just node1.
Debian
Debian usually configures the hostname differently than you need it for Ganeti. For example,
this is what it puts in /etc/hosts in certain situations:
127.0.0.1 localhost
127.0.1.1 node1.example.com node1
but for Ganeti you need to have:
127.0.0.1 localhost
192.0.2.1 node1.example.com node1
replacing 192.0.2.1 with your node’s address. Also, the file /etc/hostname which configures
the hostname of the system should contain node1.example.com and not just node1 (you need
to run the command /etc/init.d/hostname.sh start after changing the file).
Why a fully qualified host name
Although most distributions use only the short name in the /etc/hostname file, we still think
Ganeti nodes should use the full name. The reason for this is that calling ‘hostname –fqdn’
requires the resolver library to work and is a ‘guess’ via heuristics at what is your domain
name. Since Ganeti can be used among other things to host DNS servers, we don’t want to
depend on them as much as possible, and we’d rather have the uname() syscall return the full
node name.
We haven’t ever found any breakage in using a full hostname on a Linux system, and anyway
we recommend to have only a minimal installation on Ganeti nodes, and to use instances (or
other dedicated machines) to run the rest of your network services. By doing this you can
change the /etc/hostname file to contain an FQDN without the fear of breaking anything
unrelated.
Installing The Hypervisor
Mandatory on all nodes.
While Ganeti is developed with the ability to modularly run on different virtualization
environments in mind the only two currently useable on a live system are Xen and KVM.
Supported Xen versions are: 3.0.3 and later 3.x versions, and 4.x (tested up to 4.1). Supported
KVM versions are 72 and above.
Please follow your distribution’s recommended way to install and set up Xen, or install Xen
from the upstream source, if you wish, following their manual. For KVM, make sure you have
a KVM-enabled kernel and the KVM tools.
After installing Xen, you need to reboot into your new system. On some distributions this
might involve configuring GRUB appropriately, whereas others will configure it
Page | 6
automatically when you install the respective kernels. For KVM no reboot should be
necessary.
Xen on Debian
Under Debian you can install the relevant xen-linux-system package, which will pull in both
the hypervisor and the relevant kernel. Also, if you are installing a 32-bit system, you should
install the libc6-xen package (run apt-get install libc6-xen).
Xen settings
It’s recommended that dom0 is restricted to a low amount of memory (512MiB or 1GiB is
reasonable) and that memory ballooning is disabled in the file /etc/xen/xend-config.sxp by
setting the value dom0-min-mem to 0, like this:
(dom0-min-mem 0)
For optimum performance when running both CPU and I/O intensive instances, it’s also
recommended that the dom0 is restricted to one CPU only. For example you can add
dom0_max_vcpus=1,dom0_vcpus_pin to your kernels boot command line and set dom0-cpus
in /etc/xen/xend-config.sxp like this:
(dom0-cpus 1)
It is recommended that you disable xen’s automatic save of virtual machines at system
shutdown and subsequent restore of them at reboot. To obtain this make sure the variable
XENDOMAINS_SAVE in the file /etc/default/xendomains is set to an empty value.
If you want to use live migration make sure you have, in the xen config file, something that
allows the nodes to migrate instances between each other. For example:
(xend-relocation-server yes)
(xend-relocation-port 8002)
(xend-relocation-address '')
(xend-relocation-hosts-allow '^192\\.0\\.2\\.[0-9]+$')
The second line assumes that the hypervisor parameter migration_port is set 8002, otherwise
modify it to match. The last line assumes that all your nodes have secondary IPs in the
192.0.2.0/24 network, adjust it accordingly to your setup.
If you want to run HVM instances too with Ganeti and want VNC access to the console of
your instances, set the following two entries in /etc/xen/xend-config.sxp:
(vnc-listen '0.0.0.0') (vncpasswd '')
You need to restart the Xen daemon for these settings to take effect:
$ /etc/init.d/xend restart
Selecting the instance kernel
Page | 7
After you have installed Xen, you need to tell Ganeti exactly what kernel to use for the
instances it will create. This is done by creating a symlink from your actual kernel to
/boot/vmlinuz-3-xenU, and one from your initrd to /boot/initrd-3-xenU [1]. Note that if you
don’t use an initrd for the domU kernel, you don’t need to create the initrd symlink.
Debian
After installation of the xen-linux-system package, you need to run (replace the exact version
number with the one you have):
$ cd /boot
$ ln -s vmlinuz-2.6.26-1-xen-amd64 vmlinuz-3-xenU
$ ln -s initrd.img-2.6.26-1-xen-amd64 initrd-3-xenU
By default, the initrd doesn’t contain the Xen block drivers needed to mount the root device,
so it is recommended to update the initrd by following these two steps:
 edit /etc/initramfs-tools/modules and add xen_blkfront
 run update-initramfs -u
Installing DRBD
Recommended on all nodes: DRBD is required if you want to use the high availability (HA)
features of Ganeti, but optional if you don’t require them or only run Ganeti on single-node
clusters. You can upgrade a non-HA cluster to an HA one later, but you might need to convert
all your instances to DRBD to take advantage of the new features.
Supported DRBD versions: 8.0-8.3. It’s recommended to have at least version 8.0.12. Note
that for version 8.2 and newer it is needed to pass the usermode_helper=/bin/true parameter to
the module, either by configuring /etc/modules or when inserting it manually.
Now the bad news: unless your distribution already provides it installing DRBD might
involve recompiling your kernel or anyway fiddling with it. Hopefully at least the Xen-ified
kernel source to start from will be provided (if you intend to use Xen).
The good news is that you don’t need to configure DRBD at all. Ganeti will do it for you for
every instance you set up. If you have the DRBD utils installed and the module in your kernel
you’re fine. Please check that your system is configured to load the module at every boot, and
that it passes the following option to the module: minor_count=NUMBER. We recommend
that you use 128 as the value of the minor_count - this will allow you to use up to 64
instances in total per node (both primary and secondary, when using only one disk per
instance). You can increase the number up to 255 if you need more instances on a node.
Debian
On Debian, you can just install (build) the DRBD module with the following commands,
making sure you are running the target (Xen or KVM) kernel:
$ apt-get install drbd8-source drbd8-utils
$ m-a update
$ m-a a-i drbd8
Page | 8
Or on newer versions, if the kernel already has modules:
$ apt-get install drbd8-utils
Then to configure it for Ganeti:
$ echo drbd minor_count=128 usermode_helper=/bin/true >> /etc/modules
$ depmod -a
$ modprobe drbd minor_count=128 usermode_helper=/bin/true
It is also recommended that you comment out the default resources (if any) in the
/etc/drbd.conf file, so that the init script doesn’t try to configure any drbd devices. You can do
this by prefixing all resource lines in the file with the keyword skip, like this:
skip {
resource r0 {
...
}
}

skip {
resource "r1" {
...
}
}
Installing RBD
Recommended on all nodes: RBD is required if you want to create instances with RBD disks
residing inside a RADOS cluster (make use of the rbd disk template). RBD-based instances
can failover or migrate to any other node in the ganeti cluster, enabling you to exploit of all
Ganeti’s high availabilily (HA) features.
Attention
Be careful though: rbd is still experimental! For now it is recommended only for testing
purposes. No sensitive data should be stored there.
You will need the rbd and libceph kernel modules, the RBD/Ceph userspace utils (ceph-
common Debian package) and an appropriate Ceph/RADOS configuration file on every VM-
capable node.
You will also need a working RADOS Cluster accessible by the above nodes.
RADOS Cluster
You will need a working RADOS Cluster accesible by all VM-capable nodes to use the RBD
template. For more information on setting up a RADOS Cluster, refer to the official docs.
If you want to use a pool for storing RBD disk images other than the default (rbd), you should
first create the pool in the RADOS Cluster, and then set the corresponding rbd disk parameter
named pool.
Page | 9
Kernel Modules
Unless your distribution already provides it, you might need to compile the rbd and libceph
modules from source. You will need Linux Kernel 3.2 or above for the kernel modules.
Alternatively you will have to build them as external modules (from Linux Kernel source 3.2
or above), if you want to run a less recent kernel, or your kernel doesn’t include them.
Userspace Utils
The RBD template has been tested with ceph-common v0.38 and above. We recommend
using the latest version of ceph-common.
Debian
On Debian, you can just install the RBD/Ceph userspace utils with the following command:
$ apt-get install ceph-common
Configuration file
You should also provide an appropriate configuration file (ceph.conf) in /etc/ceph. For the rbd
userspace utils, you’ll only need to specify the IP addresses of the RADOS Cluster monitors.
ceph.conf
Sample configuration file:
[mon.a]
host = example_monitor_host1
mon addr = 1.2.3.4:6789
[mon.b]
host = example_monitor_host2
mon addr = 1.2.3.5:6789
[mon.c]
host = example_monitor_host3
mon addr = 1.2.3.6:6789
For more information, please see the Ceph Docs
Other required software
Please install all software requirements mentioned in Ganeti quick installation guide. If you
want to build Ganeti from source, don’t forget to follow the steps required for that as well.
Setting up the environment for Ganeti
Configuring the network
Mandatory on all nodes.
Page | 10
You can run Ganeti either in “bridged mode”, “routed mode” or “openvswitch mode”. In
bridged mode, the default, the instances network interfaces will be attached to a software
bridge running in dom0. Xen by default creates such a bridge at startup, but your distribution
might have a different way to do things, and you’ll definitely need to manually set it up under
KVM.
Beware that the default name Ganeti uses is xen-br0 (which was used in Xen 2.0) while Xen
3.0 uses xenbr0 by default. See the Initializing the cluster section to learn how to choose a
different bridge, or not to use one at all and use “routed mode”.
In order to use “routed mode” under Xen, you’ll need to change the relevant parameters in the
Xen config file. Under KVM instead, no config change is necessary, but you still need to set
up your network interfaces correctly.
By default, under KVM, the “link” parameter you specify per-nic will represent, if non-
empty, a different routing table name or number to use for your instances. This allows
isolation between different instance groups, and different routing policies between node
traffic and instance traffic.
You will need to configure your routing table basic routes and rules outside of ganeti. The vif
scripts will only add /32 routes to your instances, through their interface, in the table you
specified (under KVM, and in the main table under Xen).
Also for “openvswitch mode” under Xen a custom network script is needed. Under KVM
everything should work, but you’ll need to configure your switches outside of Ganeti (as for
bridges).
Bridging issues with certain kernels
Some kernel versions (e.g. 2.6.32) have an issue where the bridge will automatically change
its MAC address to the lower-numbered slave on port addition and removal. This means that,
depending on the MAC address of the actual NIC on the node and the addresses of the
instances, it could be that starting, stopping or migrating instances will lead to timeouts due to
the address of the bridge (and thus node itself) changing.
To prevent this, it’s enough to set the bridge manually to a specific MAC address, which will
disable this automatic address change. In Debian, this can be done as follows in the bridge
configuration snippet:
up ip link set addr $(cat /sys/class/net/$IFACE/address) dev $IFACE
which will “set” the bridge address to the initial one, disallowing changes.
Bridging under Debian