Contents
Contents
Machine Overview
Machine |
Hostname |
IP Address |
Machine 1 |
natty1 |
192.168.122.100 |
Machine 2 |
natty2 |
192.168.122.101 |
Pacemaker/Corosync
[both] Install Pacemaker and Corosync
sudo apt-get install pacemaker corosync
[natty1] Generate Corosync keys
The cluster nodes, to be able to communicate within each other, they need a key to be able to authenticate the packages sent between them by corosync. This key is generated once and copied to the other nodes in the cluster:
sudo corosync-keygen
After executing the command, you need to generate entropy to have enough random bits to be able to generate the key. This can be achieved in many ways, such as keyboard pressing, mouse movement, or downloading files from the Internet, or even installing/upgrading packages of your system.
Once the key generation has finished, you need to copy the keys to the other nodes. From natty1 do as follows:
sudo scp /etc/corosync/authkey <user>@<cluster-node-ip>:/home/<user>
In natty2 do as follows:
sudo mv ~/authkey /etc/corosync
[both] Configure Corosync
Now, we need to configure corosync to be able to listen the messages that are going to be sent in the network. For this we edit the bindnetaddr field in /etc/corosync/corosync.conf and we set it to the "network address" of the subnet in used. In our case, is as follows:
[...]
interface {
# The following values need to be set based on your environment
ringnumber: 0
bindnetaddr: 192.168.122.0
mcastaddr: 226.94.1.1
mcastport: 5405
}
[...]
[both] Starting Corosync
Before being able to start corosync, we need to edit /etc/default/corosync and enable it by changing the no to yes:
START=yes
Once this is done, we can start corosync as follows:
sudo service corosync start
[natty1] Verifying Cluster Nodes
After starting corosync and a few seconds have passed, the nodes should be part of the cluster. To verify this we do as follows:
sudo crm_mon
And we will see similar to the following when the cluster nodes join the cluster:
============ Last updated: Fri Mar 18 16:27:48 2011 Stack: openais Current DC: natty1 - partition with quorum Version: 1.0.9-da7075976b5ff0bee71074385f8fd02f296ec8a3 2 Nodes configured, 2 expected votes 0 Resources configured. ============ Online: [ natty1 natty2 ]
DRBD
Prepare Partitions
The first step in setting up DRBD is to prepare the partitions/disks to be used as DRBD devices. We are assuming that we have a disk (vdb) on which we will create a partition table, and a primary partition of 1Gb for the DRBD device.
[natty1] Create the partitions
First, we create the partition table as follows:
sudo fdisk /dev/vdb
And we select the following options:
o -> To create an empty partition w -> To write the newly created partition table
Once, this is done, we create a primary partition of 1Gb:
sudo fdisk /dev/vdb
And we use the following options:
n -> New partition p -> Primary 1 -> Partition # 1 1 -> Where does partition starts +1024 -> Size of partition (1Gb = 1024 bytes) w -> To write the changes
[natty2] Copy partition table to natty2
First, on natty1 we issue the following:
sudo sfdisk -d /dev/vdb
This will generate an output such as:
# partition table of /dev/vdb unit: sectors /dev/vdb1 : start= 63, size= 1033137, Id=83 /dev/vdb2 : start= 0, size= 0, Id= 0 /dev/vdb3 : start= 0, size= 0, Id= 0 /dev/vdb4 : start= 0, size= 0, Id= 0
We need to copy the above output to natty2 to be able to create the partition table. First, we do the following:
sudo sfdisk -d /dev/vdb <<EOF
then we copy the output, and we finish it with EOF. This looks like:
sudo sfdisk -d /dev/vdb <<EOF > # partition table of /dev/vdb > unit: sectors > /dev/vdb1 : start= 63, size= 1033137, Id=83 > /dev/vdb2 : start= 0, size= 0, Id= 0 > /dev/vdb3 : start= 0, size= 0, Id= 0 > /dev/vdb4 : start= 0, size= 0, Id= 0 > > EOF
NOTE: In case the above doesn't work for any reason, we can create the partition for natty2 the same way the partition was create for natty1.
Installing and Configuring DRBD Resources
[both] Installing DRBD
First, we need to install drbd utils, as the DRBD kernel module is already in mainline kernel.
sudo apt-get install drbd8-utils
Then, we need to bring up the kernel module:
sudo modprobe drbd
And add it to /etc/modules:
[...] loop lp drbd # -> added
[both] Creating the DRBD resource
Now that DRBD is up, we need to create a resource. For this, we create /etc/drbd.d/export.res, and we copy:
resource export {
device /dev/drbd0;
disk /dev/vdb1;
meta-disk internal;
on natty1 {
address 192.168.122.100:7788;
}
on natty2 {
address 192.168.122.101:7788;
}
syncer {
rate 10M;
}
}The configuration file is explained as follows:
resource export: Specifies the name that identifies the resourced
device /dev/drbd0: Specifies the name of the DRBD block device to be used
disk /dev/vbd1: Specifies that we are using /dev/vdb1 for the device above.
on natty1 | on natty2: Note that these are the hostnames of the nodes and contains the respective IP address and port on which each DRBD node will communicate.
Once the configuration file is saved, we can test its correctness as follows:
sudo drbdadm dump export
Now, that we have created the resources, we need to create the metadata as follows:
sudo drbdadm create-md
And, then we have to bring it up:
sudo drbdadm up export
Once this command is issued, we can check that both DRBD nodes have made communication and we'll see that the data is inconsistent as no initial synchronization has been made. For this we do the following:
sudo drbd-overview
And the result will be similar to:
0:export Connected Secondary/Secondary Inconsistent/Inconsistent C r-----
[natty1] Initial DRBD Synchronization
Now, to be able to synchronize the data (and only if devices are identical) we do as follows:
sudo drbdadm -- --clear-bitmap new-current-uuid export
And this should show something similar to:
0:export Connected Secondary/Secondary UpToDate/UpToDate C r-----
In case the above command didn't work or wasn't used, then we can do as follows:
drbdadm -- --overwrite-data-of-peer primary export
Now, we have to make sure that natty1 is primary to be able to create the filesystem. For this we do as follows:
sudo drbdadm primary export sudo mkfs -t ext3 /dev/drbd0
Master/Slave Integration with Pacemaker
[both] Removing Startup symlinks
It is not necessary for DRBD init scripts to bring up the resources configured, given that now pacemaker will manage the resources. So we simply remove the symlinks as follows:
sudo update-rc.d -f drbd remove
[natty1] Configuring the resources
Now, we can configure the resources for pacemaker. The resources to configure are simply the resources needed to bring up the DRBD resources, and to be mount the DRBD partition into the filesystem. Additionally, other parameters are required to handle the master/slave states of drbd as well as when and where to become Master in one node. To be able to do so, we do as follows:
sudo crm configure
And we copy the following:
crm(live)configure# primitive res_drbd_export ocf:linbit:drbd params drbd_resource="export" # Configures the DRBD resource, specifying its name crm(live)configure# primitive res_fs ocf:heartbeat:Filesystem params device="/dev/drbd0" directory="/mnt/export" fstype="ext3" # Mounts the filesystem specifying the DRBD device and mount point. crm(live)configure# ms ms_drbd_export res_drbd_export meta notify="true" master-max="1" master-node-max="1" clone-max="2" clone-node-max="1" # Master/Slave notification. crm(live)configure# colocation c_export_on_drbd inf: res_fs ms_drbd_export:Master # Tells Cluster to Mount the filesystem after DRBD has become primary in the node. crm(live)configure# order o_drbd_before_nfs inf: ms_drbd_export:promote res_fs:start # Tells the cluster to promote DRBD, and after that start the Fielsystem. crm(live)configure# property stonith-enabled=false crm(live)configure# property no-quorum-policy=ignore crm(live)configure# commit crm(live)configure# exit
Then, to finish, we do the following to commit the changes:
commit
Now, to verify the changes we do:
sudo crm status
which should show something similar to:
============
Last updated: Fri Mar 18 17:12:36 2011
Stack: openais
Current DC: natty2 - partition with quorum
Version: 1.0.9-da7075976b5ff0bee71074385f8fd02f296ec8a3
2 Nodes configured, 2 expected votes
2 Resources configured.
============
Online: [ natty1 natty2 ]
Master/Slave Set: ms_drbd_export
Masters: [ natty1 ]
Slaves: [ natty2 ]
res_fs (ocf::heartbeat:Filesystem): Started natty1
HA Virtual IP Address
[natty1] Preparing other resources
Now, to be able to add the VIP to the above configuration, we are going to group resources. To be able to do so, the first thing we need to do is stop the res_fs so that we can group the VIP with it. To do that we do as follows:
sudo crm resource stop res_fs
[natty1] Adding a VIP Resource
Now. to be able to add the VIP resource, we are gonna use the crm command line. We do as follows:
sudo crm configure
And then we add the res_ip resource:
crm(live)configure# primitive res_ip ocf:heartbeat:IPaddr2 params ip="102.169.122.254" cidr_netmask="24" nic="eth0" crm(live)configure# commit crm(live)configure# exit
[natty1] Grouping resources
Now, given that we have to two resources (res_fs and res_ip) that we would like to bring up after DRBD, we need to group them. The grouping will bring up res_fs first, and after that, it will bring up res_ip. We do the following:
crm(live)configure# group rg_export res_fs res_ip crm(live)configure# commit crm(live)configure# exit
And you should see something like:
INFO: resource references in colocation:c_export_on_drbd updated INFO: resource references in order:o_drbd_before_export updated
This will update the colocation and order from using res_fs to rg_export. This means that it will automatically update the colocation and order rules to bring DRBD before bringing the group of resources. To verify, we check the status of the cluster (sudo crm status), and the result should be as follows:
:~$ sudo crm status
============
Last updated: Mon Mar 21 15:37:58 2011
Stack: openais
Current DC: natty1 - partition with quorum
Version: 1.0.9-da7075976b5ff0bee71074385f8fd02f296ec8a3
2 Nodes configured, 2 expected votes
2 Resources configured.
============
Online: [ natty1 natty2 ]
Master/Slave Set: ms_drbd_export
Masters: [ natty1 ]
Slaves: [ natty2 ]
Resource Group: rg_export
res_fs (ocf::heartbeat:Filesystem): Started natty1
res_ip (ocf::heartbeat:IPaddr2): Started natty1
HA NFS
[both] Installing and Preparing NFS
First, we need to install MySQL and disable the init script as it is going to be managed by pacemaker. To install we do:
sudo apt-get install nfs-kernel-server
And we disable the init script as follows:
sudo update-rc.d -f nfs-kernel-server remove
[natty1] Preparing DRBD
Now, we are going to use the DRBD export resource as the resource NFS will use. This will allow us to have the NFS export data replicated between the two nodes. We can, however, create another resource for NFS only and call it nfsexport. But, for demonstrations purposes we are going to use the previously created one.
This means that we are going to use the DRBD resource, mounted in /mnt/export, for every single share we create. In this case, we are just going to create the NFS4 virtual root filesystem share, which will NOT have any shared data, and we will create another directory that will contain the shared data, and assign it to the share, as follows:
/mnt/export # NFSv4 root filesystem /mnt/export/export1 # Export 1 /mnt/export/export2 # We can add more exports
Now, on natty1 (or where ever DRBD is primary) we do the following:
sudo mkdir /mnt/export/export1
Integrating with Pacemaker
[natty1] Creating a NFS resource
The first thing we need to do is add the new resource that will start the NFS daemon and add a clone. For this we do as follows:
sudo crm configure
And add the following resource and clone. The clone allows the resource to be running in both cluster nodes at the same time for a smooth transition.
crm(live)configure# primitive res_nfsserver lsb:nfs-kernel-server op monitor interval="30s" crm(live)configure# clone cl_nfsserver res_nfsserver crm(live)configure# commit
[natty1] NFSv4 root virtual filesystem
Now, we need to add resources for NFSv4 virtual root filesystem, so that both, NFSv4 and NFSv3 clients can mount the exports. For this we do as follows in the crm shell:
crm(live)configure# primitive res_exportfs_root ocf:heartbeat:exportfs params fsid=0 directory="/mnt/export" options="rw,crossmnt" clientspec="192.168.122.0/255.255.255.0" op monitor interval="30s" crm(live)configure# clone cl_exportfs_root res_exportfs_root
And we need to ensure that this clone gets started before the data to be exported. For this we do the following, which will start the exportfs root before the resource group of on which we will be starting the exported resources. We do as follows:
crm(live)configure# order o_root_before_nfs inf: cl_exportfs_root rg_export:start crm(live)configure# colocation c_nfs_on_root inf: rg_export cl_exportfs_root crm(live)configure# commit
Now, to verify that it has been started correctly we do the follows in both nodes:
sudo exportfs -v
The output should be similar to:
/mnt/export 192.168.122.0/255.255.255.0(rw,wdelay,crossmnt,root_squash,no_subtree_check,fsid=0)
[natty1] Non-root NFS Exports
Now, we need to add the exports using pacemaker, instead of manually doing it on /etc/exports. For that we do as follows:
crm(live)configure# primitive res_exportfs_export1 ocf:heartbeat:exportfs params fsid=1 directory="/mnt/export/export1" options="rw,mountpoint" clientspec="192.168.122.0/255.255.255.0" wait_for_leasetime_on_stop=true op monitor interval="30s"
And then we need to edit the rg_export resource:
crm(live)configure# edit rg_export
And add this newly created resource before res_ip:
group rg_export res_fs res_exportfs_export1 res_ip
Now, to verify we do:
sudo exportfs -v
And the output should be similar to:
/mnt/export/export1
192.168.122.0/255.255.255.0(rw,wdelay,root_squash,no_subtree_check,fsid=1,mountpoint)
/mnt/export 192.168.122.0/255.255.255.0(rw,wdelay,crossmnt,root_squash,no_subtree_check,fsid=0)
[client] Verifying
Now, to verify that we can mount the exported share from a client, we do the following using the VIP (res_ip):
sudo mount -t nfs 192.168.122.254:export1 /mnt
And the output of df should show:
:~$ df -h
[...]
192.168.122.254:export1
489M 32M 433M 7% /mnt
HA MySQL
[both] Installing and Preparing MySQL
First, we need to install MySQL and disable it's upstart job so that it doesn't start on boot. To install we do:
sudo apt-get install mysql-server
Once installed, we need to disable it's upstart job in /etc/init/mysql.conf by changing start on to stop on, as follows:
[...]
stop on (net-device-up
and local-filesystems
and runlevel [2345])
[...]This prevents MySQL from been started on boot, allowing Pacemaker to do so when necessary.
Now, we also need to disable AppArmor's profile form MySQL, otherwise AppArmor won't allow MySQL to get installed into the DRBD resource. To do this we do the following:
sudo ln -s /etc/apparmor.d/usr.sbin.mysqld /etc/apparmor.d/disable/ sudo apparmor_parser -R /etc/apparmor.d/usr.sbin.mysqld
Other way to achieve this is to edit the MySQL's AppArmor profile and add the directory of the DRBD resource where we are going to install MySQL's new data dir as detailed bellow (/mnt/export)
[natty1] Preparing DRBD
Now, we are going to use the DRBD export resource as the resource MySQL will use. This will allow us to have the MySQL data replicated between the two nodes. We can, however, create another resource for MySQL only and call it mysql. But, for demonstrations purposes we are going to use the previously created one.
Now, on natty1 (or where ever DRBD is primary) we do the following:
sudo mysql_install_db --datadir=/mnt/export sudo chown -R mysql:mysql /mnt/export
You should not see any errors at this point.
[natty1] Integrating with Pacemaker
Finally, we need to tell pacemaker to manage the MySQL. For this we will add a res_mysql1 resource and add the newly created resource to the resource group. For this we first enter to the crm shell:
sudo crm configure
Then we add the resource:
crm(live)configure# primitive res_mysql1 ocf:heartbeat:mysql params config=/etc/mysql/my.cnf datadir=/mnt/export/mysql binary=/usr/bin/mysqld_safe pid=/var/run/mysql/mysql1.pid socket=/var/run/mysql/mysql1.sock log=/var/log/mysql/mysql1.log additional_parameters="--bind-address=192.168.122.254" op start timeout=120s op stop timeout=120s op monitor interval=15s
Once the resource has been added, we need to edit the resource group and add res_mysql1. We do this while in the crm shell above as follows:
crm(live)configure# edit group rg_export
The above command launches an editor, on which we add res_mysql1 at the end, as follows:
group rg_export res_fs res_ip res_mysql1
Finally, we save the changes on the editor, and from the CRM shell we commit the changes as follows:
crm(live)configure# commit crm(live)configure# exit
[natty1] Verifying MySQL
Now, the cluster status (sudo crm status) should look as follows:
============
Last updated: Tue Mar 22 15:27:07 2011
Stack: openais
Current DC: natty1 - partition with quorum
Version: 1.0.9-da7075976b5ff0bee71074385f8fd02f296ec8a3
2 Nodes configured, 2 expected votes
2 Resources configured.
============
Online: [ natty1 natty2 ]
Master/Slave Set: ms_drbd_export
Masters: [ natty1 ]
Slaves: [ natty2 ]
Resource Group: rg_export
res_fs (ocf::heartbeat:Filesystem): Started natty1
res_ip (ocf::heartbeat:IPaddr2): Started natty1
res_mysql1 (ocf::heartbeat:mysql): Started natty1