Activate a Galera Arbitrator node in a Galera Cluster
In the previous posts Galera Cluster for MySQL and Automating the startup of Galera Cluster we focused on installing the Galera cluster on 2 nodes and starting it automatically using a script job in the crontab. Here I want to talk about the Galera Arbitrator module.
Galera Arbitrator
We have already focused about a minimum of 3 nodes recommended for a reliable Galera cluster, and this to avoid split-brain situations in case of connectivity problems between the nodes. Galera Arbitrator has exactly this role, it represents the third node of the cluster but does not host data, so it is not involved in replication operations. Instead, it participates in voting operations, and in case of loss of connectivity between the nodes it can vote the primary node, avoiding the occurrence of split-brain conditions.
By reference to the installed cluster, in my case the architecture will be as follows:
- galera-node-1 172.31.81.55
- galera-node-2 172.31.80.162
- galera-arbitrator 172.31.6.110
The servers 172.31.81.55 and 172.31.80.162 are the 2 Galera nodes with MySQL installed, while the third node will host only the Arbitrator.
Setup
For the installation we proceed in the same way as for any other node of a Galera cluster. Changes concern just the configuration and the daemon to be started, that for the Arbitrator node will be the daemon called garbd, and not MySQL engine. We only need to provide a repo file to access the Galera Cluster repository and install it yum install galera-4 mysql-wsrep-8.0. Once the installation is completed, make sure that MySQL startup is disabled systemctl disable mysqld.
The garbd daemon can be started from the shell, or as a service via systemctl. Assuming we choose this second option we have to configure the system by editing the /etc/sysconfig/garb file, specifying the cluster name and the nodes belonging to our cluster.
# Copyright (C) 2012 Codership Oy
# This config file is to be sourced by garb service script.
# A comma-separated list of node addresses (address[:port]) in the cluster
GALERA_NODES="172.31.81.55:4567,172.31.80.162:4567"
# Galera cluster name, should be the same as on the rest of the nodes.
GALERA_GROUP="galera-clusterdb"
# Optional Galera internal options string (e.g. SSL settings)
# see http://galeracluster.com/documentation-webpages/galeraparameters.html
# GALERA_OPTIONS=""
# Log file for garbd. Optional, by default logs to syslog
# LOG_FILE="/var/log/garbd.log"
Now we can start garbd using systemctl start garb.
The next step is to tell the two nodes that a third node now exists, although only for voting operations. We need to change the wsrep_cluster_address parameter, inside the /etc/my.cnf file, on which we will add the IP of the Arbitrator node.
wsrep_cluster_address="gcomm://172.31.81.55,172.31.80.162,172.31.6.110"
By checking the cluster size with SHOW STATUS LIKE 'wsrep_cluster_size'; we can verify that the cluster size is now 3.
+--------------------+-------+
| Variable_name | Value |
+--------------------+-------+
| wsrep_cluster_size | 3 |
+--------------------+-------+
To test the Arbitrator, using iptables we can block the communication between node 2 of the Cluster and the other nodes (node 1 and Arbitrator).
iptables -A INPUT -s 172.31.6.110 -j DROP; iptables -A OUTPUT -d 172.31.6.110 -j DROP
iptables -A INPUT -s 172.31.81.55 -j DROP; iptables -A OUTPUT -d 172.31.81.55 -j DROP
The Arbitrator detects that the node is no longer reachable.
INFO: (1940e1d4-8259, 'tcp://0.0.0.0:4567') connection to peer 339f8f1d-a4ea with addr tcp://172.31.80.162:4567 timed out, no messages seen in PT3S, socket stats: rtt: 14992 rttvar: 19552 rto: 1720000 lost: 1 last_data_recv: 3381 cwnd: 1 last_queued_since: 381820935 last_delivered_since: 3381574527 send_queue_length: 0 send_queue_bytes: 0 segment: 0 messages: 0
INFO: (1940e1d4-8259, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: tcp://172.31.80.162:4567
INFO: (1940e1d4-8259, 'tcp://0.0.0.0:4567') reconnecting to 339f8f1d-a4ea (tcp://172.31.80.162:4567), attempt 0
INFO: (1940e1d4-8259, 'tcp://0.0.0.0:4567') connection to peer 00000000-0000 with addr tcp://172.31.80.162:4567 timed out, no messages seen in PT3S, socket stats: rtt: 0 rttvar: 250000 rto: 2000000 lost: 1 last_data_recv: 2709114 cwnd: 1 last_queued_since: 3009114578953 last_delivered_since: 3009114578953 send_queue_length: 0 send_queue_bytes: 0
The size of the cluster is now reduced to 2, the Arbitrator has blocked operations on the database hosted by node 2, we can access mysql on node 2, but operations are not allowed.
Reactivating the connectivity between the nodes with iptables --flush, the Arbitrator detects that the node is reachable again and restores the original size of the cluster.
INFO: STATE_EXCHANGE: sent state UUID: 664cf725-d87b-11ed-bfec-c212275b2001
INFO: STATE EXCHANGE: sent state msg: 664cf725-d87b-11ed-bfec-c212275b2001
INFO: STATE EXCHANGE: got state msg: 664cf725-d87b-11ed-bfec-c212275b2001 from 0 (garb)
INFO: STATE EXCHANGE: got state msg: 664cf725-d87b-11ed-bfec-c212275b2001 from 1 (galera-node-2)
INFO: STATE EXCHANGE: got state msg: 664cf725-d87b-11ed-bfec-c212275b2001 from 2 (galera-node-1)
All the data written on node 1 are now synchronized on node 2.