MySQL Statsdb Switch¶

Statsdb switch is done by running the statsb_switch script. See below for a functionality overview.

Note:* The procedure for performing a statsdb2* switch is basically the same. Simply use --newstatsdb2 parameter.

Performing a statsdb switch¶

Simply run this command to eg switch the current statsdb to dbs004:

statsdb_switch --newstatsdb=dbs004.prd.fyber.com

Attention:* Contrary to the MySQL Master Switch script, a switch will be performed immediately, without waiting for confirmation!***

Note: Always use the fqdn for the new host!
Note2: Always run the script as a regular user, never as root! The user must have full sudo rights on all servers, however.

Tip:* Run mytop in a separate screen on the new statsdb to see if clients connect properly (see Troubleshooting*).

The script will run a sanity check, then perform the necessary steps to make dbs004 statsdb (see below).

It doesn't actually matter on which host the script is run. I () have made it a habit to always run the script on the soon-to-become-statsdb.

Troubleshooting¶

If no client connections start appearing immediately after the switch, it might be necessary to clear neighboring arp caches again.

For statsdb¶

sudo /usr/sbin/arping -q -c3 -U 10.99.36.26 -I bond0

For statsdb2¶

sudo /usr/sbin/arping -q -c3 -U 10.99.36.27 -I bond0

Finalizing the switch in chef¶

After the switch both servers (old and new statsdb) have to be cheffed to update their local configuration:

1. Edit roles/dbs00X.json for the new statsb, replace role[db_slave] with role[db_statsdb]
2. Edit roles/dbs00X.json for the old statsb, replace role[db_statsdb] with role[db_slave]
3. Chef the old statsdb
4. Chef the new statsdb
5. Chef mon003 (to make monitoring aware of the change).

Overview¶

Running the script without parameters simply displays the current configuration:

rwunderer@dbs004:~$ statsdb_switch

Checking configuration... 

dbs003.prd.fyber.com dbs002.prd.fyber.com dbs004.prd.fyber.com dbs005.prd.fyber.com  
WARN: No mha_manager host found!


####################### 
Current configuration: 

MASTER is dbs002.prd.fyber.com 
STATSDB is dbs004.prd.fyber.com 
STATSDB2 is dbs003.prd.fyber.com 
MHA_MANAGER is  


Current layout as reported by mha_manager: 

 +--dbs003.prd.fyber.com(10.99.36.33:3306)
 +--dbs004.prd.fyber.com(10.99.36.34:3306)
 +--dbs005.prd.fyber.com(10.99.36.35:3306)

(ignore the warning about the missing mha_manager, it would only be needed for automated failover)

Using --help gives an overview of the possible parameters:

rwunderer@dbs004:~$ statsdb_switch --help

Usage:
    /usr/local/bin/statsdb_switch [OPTIONS] [--oldstatsdb=name] [--newstatsdb=name] [--oldstatsdb2=name] [--newstatsdb2=name]

    -h, --help              Display this help and exit
    -q, --quiet             Do not display information about actions performed
    -f, --force             Continue even though sanity check of environment fails
    --oldstatsdb            Name of old statsdb host
    --newstatsdb            Name of new statsdb host
    --oldstatsdb2           Name of old statsdb2 host
    --newstatsdb2           Name of new statsdb2 host

Steps performed¶

1. Sanity check. If any of the following preconditions are not met, the script will abort. --force will override this and continue in any case
2. master, statsdb and statsdb2 can be identified correctly
3. statsdb and statsdb2 are not the same as master
4. the new master is actually different from current master
5. the master will not be moved to current statsdb or statsdb2
6. Take down the VIP on the old statsdb ( ip addr del )
7. Take up the VIP on the new master ( ip addr add ) and use arping -U to clear arp caches on neighboring nodes/switches.