High-Availability NFS Pseudo Load-Balanced Using Round Robin DNS, Virtual IP and Heartbeat on Ubuntu 12.04 LTS

Goal of this setup is to provide fault tolerance, and increased read performance for an NFS share.

Prerequisites:

• Two servers with equal content to be shared using NFS
• DNS server 

 The setup:

• Server1: server1.example.com, IP address: 10.0.0.100
• Server2: server2.example.com, IP address: 10.0.0.101
• Virtual IP address: 10.0.0.200 preferred server1, failover at server2
• Virtual IP address: 10.0.0.201 preferred server2, failover at server1
• Virtual IP address: fe80::f1ee:dead:beef preferred server1, failover at server2
• Virtual IP address: fe80::baad:dead:beef preferred server2, failover at server1
• DNS entry with an A record pointing to both virtual ip (multihomed, resolved using round robin)

For my particular setup, I have replicated storage using DRBD with OCFS2 to ensure equal content across my two servers.

Result:

• Fault tolerant NFS share, against a single DNS
• If one server goes down, the other server will take over the virtual IP, providing high-availability
• Round robin balanced NFS mounts across the two servers

Set up two virtual ips

Virtual ips are set up and managed by Heartbeat. Set it up on both servers:

apt-get install heartbeat -y
echo -e "auth 3 \n3 md5 secretpasswordhere" > /etc/ha.d/authkeys
chmod 600 /etc/ha.d/authkeys

Server1:

nano /etc/ha.d/ha.cf

# How many seconds between heartbeats
keepalive 2

# Seconds before declaring host dead
deadtime 10
 
# What UDP port to use for udp or ppp-udp communication?
udpport 694

bcast  eth0
mcast eth0 225.0.0.1 694 1 0
ucast eth0 10.0.0.101

# What interfaces to heartbeat over?
udp     eth0
logfacility     local0

# Allow ip to float back when server recovers
auto_failback on

# Tell what machines are in the cluster
# node must match uname -n
node    server1.example.com
node    server2.example.com

  Server2:

nano /etc/ha.d/ha.cf  

#
#       keepalive: how many seconds between heartbeats
#
keepalive 2
#
#       deadtime: seconds-to-declare-host-dead
#
deadtime 10
#
#       What UDP port to use for udp or ppp-udp communication?
#
udpport        694
bcast  eth0
mcast eth0 225.0.0.1 694 1 0
ucast eth0 192.168.0.100
#       What interfaces to heartbeat over?
udp     eth0
#
#       Facility to use for syslog()/logger (alternative to log/debugfile)
#
logfacility     local0

# Allow ip to float back when server recovers
auto_failback on 

#
#       Tell what machines are in the cluster
#       node    nodename ...    -- must match uname -n
node    server1.example.com
node    server2.example.com

Now, on both servers - set up the virtual ips:

nano /etc/ha.d/haresources 

server1 10.0.0.200 IPv6addr::fe80:0000:0000:0000:0000:flee:dead:beef
server2 10.0.0.201 IPv6addr::fe80:0000:0000:0000:0000:baad:dead:beef

 Note: Be sure the name matches 'uname -n' output.

Restart heartbeat to make the changes take effect:

/etc/init.d/heartbeat restart 

Now server1 is the primary node for 10.0.0.200, and will take over 10.0.0.201 if server2 stops responding (and vice versa)

Configuring NFS

Install NFS and share /var/www on both servers:

apt-get install nfs-kernel-server -y
mkdir /var/www
echo "/var/www 10.0.0.0/19(rw,async,no_subtree_check)" >> /etc/exports

Now your clients can connect to the DNS pointing to both the virtual IPs, and enjoy the benefits of a highly available NFS server!

High-Availability Mysql with MyISAM support on Ubuntu 12.04

As it turns out, there's quite a few approaches to provide high-availability Mysql - of where the majority doesn't support MyISAM. This is quite annoying when the main use for the Mysql server is to host out-of-the-box open source applications, which utilize MyISAM full-text search more often than not.

For my solution, I've utilized:

• Mysql replication
• Multi-Master Replication Manager for MySQL (MMM)
• Heartbeat to make sure MMM only runs at one server at the time

 The setup:

• Server1: server1.example.com, IP address: 10.0.0.100
• Server2: server2.example.com, IP address: 10.0.0.101
• Virtual IP address: 10.0.0.200 preferred server1, failover at server2
• Virtual IP address: 10.0.0.201 preferred server2, failover at server1

Setting up circular Mysql master replication managed by mmm

The official mmm site has a good tutorial which will get you started. There are however a couple of things I've done differently:

• Ubuntu 12.04 comes with mysql-mmm in package repos: apt-get install mysql-mmm-agent mysql-mmm-monitor
• I only utilize two servers, one for primary write, and one for primary read - thus I have agent and monitor on both servers
• My config for replication is done in /etc/mysql/conf.d/replication.cnf which is included by default for mysql-server package in Ubuntu 12.04
• REMOVE startup links for mysql-mmm-monitor such that only heartbeat will start it: update-rc.d -f mysql-mmm-monitor remove

Setting up high availability for the mmm monitor

The monitor can only run on one node at the time, to avoid collisions. I'm using heartbeat to ensure that the monitor only runs at one node at the time.

apt-get install heartbeat -y
echo -e "auth 3 \n3 md5 secretpasswordhere" > /etc/ha.d/authkeys
chmod 600 /etc/ha.d/authkeys

Server1:

nano /etc/ha.d/ha.cf

# How many seconds between heartbeats
keepalive 2

# Seconds before declaring host dead
deadtime 10
 
# What UDP port to use for udp or ppp-udp communication?
udpport 694

bcast  eth0
mcast eth0 225.0.0.1 694 1 0
ucast eth0 10.0.0.101

# What interfaces to heartbeat over?
udp     eth0
logfacility     local0

# Don't allow ping pong effect on masters
auto_failback off

# Tell what machines are in the cluster
# node must match uname -n
node    server1.example.com
node    server2.example.com

  Server2:

nano /etc/ha.d/ha.cf  

#
#       keepalive: how many seconds between heartbeats
#
keepalive 2
#
#       deadtime: seconds-to-declare-host-dead
#
deadtime 10
#
#       What UDP port to use for udp or ppp-udp communication?
#
udpport        694
bcast  eth0
mcast eth0 225.0.0.1 694 1 0
ucast eth0 192.168.0.100
#       What interfaces to heartbeat over?
udp     eth0
#
#       Facility to use for syslog()/logger (alternative to log/debugfile)
#
logfacility     local0

# Don't allow ping pong effect on masters
auto_failback off 

#
#       Tell what machines are in the cluster
#       node    nodename ...    -- must match uname -n
node    server1.example.com
node    server2.example.com

Now, I'll set up heartbeat to prefer mysql-mmm-monitor script located in /etc/init.d/mysql-mmm-monitor running at server1 if possible, else it will spawn at any other node in the heartbeat cluster. Heartbeat utilize the return codes on the init script for start and stop to determine if it's running or not.

nano /etc/ha.d/haresources 

server1 mysql-mmm-monitor 

 Note: Be sure the name matches 'uname -n' output.

Restart heartbeat to make the changes take effect:

/etc/init.d/heartbeat restart 

High-Availability storage using DRBD with OCFS2 on Ubuntu 12.04

This howto is designed to solve the following problems:

• Files in sync over multiple servers in case of
• Power outage in a single location
• Hardware failure
• OS lockup
• Optimized read performance
• Mounted filesystem on multiple servers

Other considerations specific to my solution:

• Limited amount of writes
• Low latency and high bandwidth between locations

Prerequisites

• Two servers
• Ubuntu 12.04
• Equally sized raw harddrive partition on both servers
• NTP client installed and running

DRBD set up to act as raid 1 (mirroring)

All steps should be done on both nodes unless stated otherwise

Install and activate DRBD tools and kernel module:

apt-get install drbd8-utils
modprobe drbd

Set up a resource config:

sudo nano /etc/drbd.d/disk.res

# Config by Jon Skarpeteig -- 06.11.2012
resource r0 {
        protocol C;
        syncer { rate 1000M; }
        startup {
                wfc-timeout  15;
                degr-wfc-timeout 60;
                become-primary-on both;
        }
        net {
# allow-two-primaries - Generally, DRBD has a primary and a secondary node.
# In this case, we will allow both nodes to have the filesystem mounted at
# the same time. Do this only with a clustered filesystem. If you do this
# with a non-clustered filesystem like ext2/ext3/ext4 or reiserfs, you will
# have data corruption.
                allow-two-primaries;

# after-sb-0pri discard-zero-changes - DRBD detected a split-brain scenario,
# but none of the nodes think they're a primary. DRBD will take the newest
# modifications and apply them to the node that didn't have any changes.
                after-sb-0pri discard-zero-changes;

# after-sb-1pri discard-secondary - DRBD detected a split-brain scenario,
# but one node is the primary and the other is the secondary. In this case,
# DRBD will decide that the secondary node is the victim and it will sync data
# from the primary to the secondary automatically.
                after-sb-1pri discard-secondary;

# after-sb-2pri disconnect - DRBD detected a split-brain scenario, but it can't
# figure out which node has the right data. It tries to protect the consistency
# of both nodes by disconnecting the DRBD volume entirely. You'll have to tell
# DRBD which node has the valid data in order to reconnect the volume.
                after-sb-2pri disconnect;

                cram-hmac-alg sha1;
                shared-secret "secret";
        }
        on server1 {
                device /dev/drbd0;
                disk /dev/sdb1;
                address 10.0.0.101:7788;
                meta-disk internal;
        }
        on server2 {
                device /dev/drbd0;
                disk /dev/sdb1;
                address 10.0.0.102:7788;
                meta-disk internal;
        }
}

Note: Here, the server1 and server2 are from the output of 'uname -n', and must be resolvable in DNS with both A and PTR records. For good measure, we can add these in /etc/hosts directly as well, not to break the DRBD link on DNS issues.

Initialize DRBD volume, and start drbd daemon:

drbdadm create-md r0

/etc/init.d/drbd start 

Then make Server1 the primary node (run this on server1 only!)

drbdadm -- --overwrite-data-of-peer primary all

At this point, it should synchronize the disks from Server1 => Server2. You can view the progress using

/etc/init.d/drbd status

Once the sync is complete, you can make server2 primary as well (run on server2 only):

drbdadm primary r0

OCFS2 to allow for file system to be mounted more than one place

All steps should be done on both nodes unless stated otherwise
 
First get management tools:

apt-get install ocfs2-tools

My config in /etc/ocfs2/cluster.conf

cluster:
        node_count = 2
        name = www

node:
        ip_port = 7777
        ip_address = 10.0.0.101
        number = 1
        name = server1
        cluster = www

node:
        ip_port = 7777
        ip_address = 10.0.0.102
        number = 2
        name = server2
        cluster = www

Note: The 'name' parameter must match exactly the 'on' parameter from drbd, and resolve to the local ip

Configure ocfs2 cluster:

sudo dpkg-reconfigure ocfs2-tools

Now, you can create the filesystem (on one server only):

mkfs.ocfs2 -L "www" /dev/drbd0

And mounting:

mkdir /var/www
echo "/dev/drbd0  /var/www  ocfs2  noauto,noatime,nodiratime,_netdev  0 0" >> /etc/fstab
mount /dev/drbd0

Now you should be all set up! Go ahead and test how it behaves by creating files, and removing them - and notice how all the changes are replicated.

Installing Bugzilla 3.6 on Ubuntu 10.04 LTS

Unfortunately this isn't as easy as it should be, due to dependencies not getting installed properly out of the box. First, download and unpack bugzilla:

wget http://ftp.mozilla.org/pub/mozilla.org/webtools/bugzilla-3.6.8.tar.gz
tar xvf bugzilla-3.6.8
mkdir /var/www/bugzilla
mv bugzilla-3.6.8/* /var/www/bugzilla/

Then check what dependencies are missing with checksetup.pl script:

cd /var/www/bugzilla
./checksetup.pl

This is likely to produce a lot of red lines with missing dependencies, and a way to install it which is supposed to be helpful (/usr/bin/perl install-module.pl --all). In a perfect world, this would solve all problems, but unfortunately it doesn't, as some of the modules simply fails to install. Luckily Ubuntu comes with some precompiled per modules that can be installed using apt:

sudo apt-get update
sudo apt-get install mysql-server build-essential -y
####################
# Required modules #
####################
# DBD::Mysql
sudo apt-get install libdbd-mysql-perl -y
# DateTime
sudo apt-get install libdatetime-perl -y
# Template
sudo apt-get install libtemplate-perl  -y
/usr/bin/perl install-module.pl Template
####################
# Optional modules #
####################
# GD, Chart::Lines, Template::Plugin::GD::Image, GD::Text, GD::Graph
sudo apt-get install libtemplate-plugin-gd-perl -y
# Test::Taint
sudo apt-get install libtest-taint-perl -y
# Math::Random::Secure
sudo apt-get install libany-moose-perl libnamespace-clean-perl
/usr/bin/perl install-module.pl Math::Random::Secure
# mod-perl2
sudo apt-get install libapache2-mod-perl2 -y

Now we're ready to change localconfig, and set up database login for instance:

nano localconfig

Some changes:

# Ubuntu default webserver group
$webservergroup = 'www-data';

Then it's time to create database, and set up virtualhost for Apache:

./checksetup.pl
sudo nano /etc/apache2/sites-enabled/bugzilla

Paste the following config into it:

   DocumentRoot /var/www/bugzilla

   
      AddHandler cgi-script cgi pl
      Options +Indexes +ExecCGI
      DirectoryIndex index.cgi
      AllowOverride Limit
   

After a quick restart of apache, you should be able to browse your newly installed bugzilla.

sudo /etc/init.d/apache2 restart

If you want to install the Bugzilla REST api, you can check out my post at: http://rdstash.blogspot.com/2010/10/bugzilla-rest-importexport-api.html

SSH keypair exchange made easy

On the client side, only two commands are needed:

To create a local key:

ssh-keygen -t dsa

To add the key as a valid key to the remote machine:

ssh-copy-id -i $HOME/.ssh/id_dsa.pub username@remote.box

Installing Gearman PHP Extension 1.0.2 on Ubuntu 10.04 LTS, 11.04, 11.10 and 12.04 LTS from pecl

I started off nice and easy:

sudo apt-get install python-software-properties -y
sudo add-apt-repository ppa:gearman-developers/ppa 
sudo apt-get update
sudo apt-get install gearman-job-server libgearman-dev gearman-tools php-pear php5-dev -y
sudo apt-get install libevent-dev uuid-dev -y
pecl install gearman

Bug then I hit: configure: error: libgearman version 0.21 or later required

To fix it I needed to compile a newer gearman from source, which in turn has another couple of issues with Boost.

To fix this for Ubuntu 10.04 LTS

sudo apt-get install libboost-program-options1.40-dev libboost-thread1.40-dev -y

For Ubuntu 11.04:

sudo apt-get install libboost-thread1.42-dev libcloog-ppl0 libboost-program-options-dev -y

For Ubuntu 11.10 and 12.04:

sudo apt-get install libboost-thread1.46-dev libcloog-ppl0 libboost-program-options-dev -y

Now compile should work, in order for libgearman requirement to be met for pecl:

apt-get install uuid-dev libevent-dev -y
wget https://launchpad.net/gearmand/trunk/0.28/+download/gearmand-0.28.tar.gz
tar xfv gearmand-0.28.tar.gz
cd gearmand-0.28
./configure
make && make install
sudo ldconfig

If you want to use the newly compiled gearmand, you need to edit /etc/init.d/gearman-job-server to include /usr/local:

prefix=/usr/local

Retry pecl install and install module into php:

sudo pecl install gearman
echo "extension=gearman.so" > /etc/php5/conf.d/gearman.ini