12.3. Setting Up HA

Home Prev	cOS Core 14.00.17 Administration Guide	Next

12.3.1. Hardware Setup

The steps for the setup of hardware in an HA cluster are as follows:

Start with two identical Clavister firewalls of the same model and with the same set of available Ethernet interfaces. Both may be newly purchased or an existing hardware unit may have a new unit added to it to create the cluster.

Note that when performing HA setup with two virtual firewalls in a virtual environment, the bus, slot and port number of each interface in an interface pair should be the same otherwise unexpected behavior could occur.
Both master and slave units must be running the same version of cOS Core.
Both master and slave units must have valid cOS Core cluster licenses that have identical capabilities and which have the HA cluster option enabled.

Make the physical connections:

Connect the matching interfaces of master and slave through separate switches or separate broadcast domains. It is important to keep the traffic on each interface pair separated from other pairs.

Select one unique interface on the master and slave which is to be used by the units for monitoring each other. This will be the sync interface. It is recommended that the same interface is used on both master and slave, assuming they are similar systems.

	Caution: The sync interface must be unique
With some hardware, an interface may be part of a switch fabric which joins a set of interfaces together. If such an interface is used as the HA sync interface then the other interfaces connected to the same switch fabric cannot be used for other purposes.

Caution: The sync interface must be unique

With some hardware, an interface may be part of a switch fabric which joins a set of interfaces together.

If such an interface is used as the HA sync interface then the other interfaces connected to the same switch fabric cannot be used for other purposes.

Also keep in mind that there should be no cOS Core IP rule set entries configured that include the sync interface.

Connect together the sync interfaces. This should be done directly with a suitable crossover cable or through a separate switch (or broadcast domain).

Note that the sync interface packets use a default Ethernet protocol number of 0xc157. If network equipment blocks or alters this Ethernet type then it must be changed in cOS Core on both cluster nodes using the high availability setting HA Sync Protocol. A CLI example of doing this might be:
```
Device:/> set HighAvailability HASyncProtocol=0x8ffd
```

Note that physical separation of HA nodes (for example, in separate buildings) can increase cluster resilience but comes with limits and this is discussed further in Section 12.4, HA Issues and Troubleshooting.

Decide on a shared IP address for each interface in the cluster. Some interfaces could have shared addresses only while others could also have unique, individual IP addresses for each interface specified in an IP4 HA Address object. The shared and individual addresses are used as follows:

The individual addresses specified for an interface in an IP4 HA Address object allow remote management through that interface. These addresses can also be "pinged" using ICMP provided that IP rule set entries are defined to permit this (by default, ICMP queries are dropped by the rule set).

If either unit is inoperative, its individual IP addresses will also be unreachable. These IP addresses are usually private but must be public if management access across the Internet is required.

If an interface is not assigned an individual address through an IP4 HA Address object then it must be assigned the default address localhost (the loopback address) which is an IP address from the sub-network 127.0.0.0/8. The localhost object behaves as two addresses and uses 127.0.0.1 for the master and 127.0.0.2 for the slave.

ARP queries for the individual IP addresses specified in IP4 HA Address objects are answered by the firewall that owns the address, using the normal hardware address, just as with normal IP units.
One single shared IP address is used for routing and it is also the address used by dynamic address translation, unless the configuration explicitly specifies another address.

	Note: Master and slave management IPs must be different
The shared IP address cannot be used for remote management or monitoring purposes. For example, when using SSH for remote management of the firewalls in an HA Cluster, the individual IP addresses of each firewall's interfaces must be used and these are specified in IP4 HA Address objects as discussed above. For this reason the management IP addresses of the cluster units must be different. It is recommended to change the management IP address of the slave unit so it is different from the master. Changing the management IP is described in Section 2.1.2, Configuring Network Management Access.

Note: Master and slave management IPs must be different

The shared IP address cannot be used for remote management or monitoring purposes. For example, when using SSH for remote management of the firewalls in an HA Cluster, the individual IP addresses of each firewall's interfaces must be used and these are specified in IP4 HA Address objects as discussed above.

For this reason the management IP addresses of the cluster units must be different. It is recommended to change the management IP address of the slave unit so it is different from the master. Changing the management IP is described in Section 2.1.2, Configuring Network Management Access.

Typical HA Cluster Network Connections

The illustration below shows the arrangement of typical HA Cluster connections in a network. All interfaces on the master unit would normally also have corresponding interfaces on the slave unit and these would be connected to the same networks. This is achieved by connecting the same interfaces on both master and slave via a separate switch (or broadcast domain) to other network portions.

Figure 12.1. High Availability Setup

In the HA setup shown above, the lan interface on the master and the lan interface on the slave would be connected to the same switch which then connects to an internal network. Similarly the wan interface on the master and the wan interface would connect to a switch which in turn connects to the external Internet.

If the lan interface is connected to a switch block in the firewall hardware then the other interfaces connected to the block should not be used.

	Note: The illustration shows a crossover cable sync connection
	The illustration above shows a direct crossover cable connection between the sync interfaces of each unit. Alternatively, the connection could be via a switch or broadcast domain.

Wizard and Manual Software Setup

The software setup procedures are now divided into the two sections that follow:

Section 12.3.2, Wizard HA Setup for fast, simple setup.
Section 12.3.3, Manual HA Setup for step by step manual setup, without the wizard.

12.3.2. Wizard HA Setup

cOS Core provides a wizard to automate the HA setup procedure. The wizard needs to be run twice: once when connected to the master unit in the HA cluster, and a second time when connected to the slave unit in the cluster. The procedure for doing this with each unit is as follows:

Connect to the Clavister firewall through the Web Interface.
Go to: Objects > Address Book and create an IP4 HA Address object for each interface pair in the cluster. Each object will contain the master and slave IP addresses for the interface pair. Inclusion in an IP4 HA Address object is mandatory for any interface that will be used for remote management but optional for other interfaces.

The private address must be different in the IP4 HA Address object for the management interface of the master and slave unit.
Save and activate the new configuration before continuing to the next step.
Go to System > Device > High Availability in the web interface and press the Start Wizard button. Running the wizard for the master and slave units is described in A and B below.
Save and activate the new configuration.

A. Running the Wizard for Master Setup

Specify the NodeType, ClusterID and Interface that will be used for synchronization.
If the two hardware models used in the cluster are different, this should also be indicated (see Section 12.3.5, Unique Shared Mac Addresses below for an explanation of this option).
Select the shared IP address and, if desired, the individual IP4 HA Address objects created earlier. Make sure the management interface IPs are different for master and slave.
The wizard will confirm that master setup is complete.

B. Running the Wizard for Slave Setup

Specify the NodeType, ClusterID and Interface that will be used for synchronization.
If the two hardware models used in the cluster are different, this should also be indicated (see Section 12.3.5, Unique Shared Mac Addresses below for an explanation of this option).
The wizard will acknowledge that it will synchronize with the master unit.
Select the desired interface mapping.
The wizard will confirm that slave setup and synchronization is complete.

Installing a New Master Unit

It should be noted in the above process that the HA wizard synchronizes the slave from the master unit and not the other way around. This means that if the slave unit is swapped with a new unit, perhaps because of an equipment failure, the wizard can be used to perform the initial setup since it will copy over the configuration from the master.

But what if it is the slave unit that contains the configuration that is to be used and it is the master unit that is to be synchronized with it? This would be the case if the master unit suffered a fault and had to be swapped with a brand new unit.

There are two methods to resolve this:

Method A. Copying the slave configuration to the new master

The easiest and quickest way to configure a new master unit is as follows:

Use the normal configuration backup function to make a backup of the configuration that exists on the existing slave unit.
Restore the backup from the slave to the new master unit.
Through the management interface, change the new master unit's HA designation to be Master and rename the device so both do not have the same name.

Method B. Turning the slave into the master

A second, slightly more complicated approach, is to turn the slave unit into a master and then use the wizard as normal to copy the configuration across.

Changing the slave to the master is done through the management interface by changing the unit's HA designation to be Master. However, a remaining issue will be that the ARP caches of connected switches will not now be valid. To force an update of these caches either the switches should be restarted or the CLI command arp -notify could be issued from the new master (which was previously the slave).

This process of changing a slave to a master must be done quickly since there will be a reversion to the old configuration within the Validation Timeout period, which, by default, is 30 seconds. Within that time, the ARP cache problem must also be addressed. To solve this issue we can either commit the new configuration manually before dealing with the ARP issue, or lengthen the time available by increasing the advanced setting Validation Timeout.

12.3.3. Manual HA Setup

To set up an HA cluster manually, without the wizard, the steps are as follows:

Connect to the master unit with the Web Interface.
Go to: System > Device > High Availability.
Check the Enable High Availability checkbox.
Set the Cluster ID. This must be unique for each cluster.
Choose the Sync Interface.
Select the node type to be Master.
Go to: Objects > Address Book and create an IP4 HA Address object for each interface pair. Each must contain the master and slave interface IP addresses for the pair.

Creating an object is mandatory for an interface pair used for remote management, but optional for other interfaces (in which case the default loopback address localhost must be used and this is an IP address from the 127.0.0.0/8 sub-network). The IPv4 address for the management interfaces of the master and slave units must be different.
Optionally create an IP6 HA Address object for any relevant interface pairs. Management access or logging is not possible using an IPv6 address. However, a private IPv6 address could be pinged by incoming ICMP messages when the HA cluster is active or used as the source IP for outgoing ICMP ping messages when HA is not active.

Go to: Network > Interfaces and VPN > Ethernet and go through each interface in the list, entering the shared IP address for that interface in the IP Address field.

Also select the Advanced tab for each interface and set the High Availability, Private IP Address field to be the name of the IP4 HA Address object created previously for the interface (cOS Core will automatically select the appropriate address from the master and slave addresses defined in the object).

	Note: IP addresses could be public IPv4 addresses
	The term "private IPv4 address" is not strictly correct when used here. Either address used in an IP4 HA Address object may be public if management access across the Internet is required.

Save and activate the new configuration.
Repeat the above steps for the other Clavister firewall but this time select the node type to be Slave.

Making Cluster Configuration Changes

The configuration on both firewalls needs to be the same. When using the Web Interface or CLI for management, the configurations of the two cluster peers will be automatically synchronized, provided automatic synchronization is enabled. To change something in a cluster configuration, log into either the master or the slave unit as an administrator, make the change, then save and activate. The change is automatically made to both firewalls. Automatic synchronization is discussed in more depth in Section 12.1, Overview.

Alternatively, an HA Cluster can also be managed using InControl and changes are then deployed automatically to the two units by InControl. Once under InControl control, configuration changes should then not be made outside of InControl.

12.3.4. Verifying that the Cluster Functions Correctly

Important: Perform shutdown on both master and slave

	Important: Perform shutdown on both master and slave
After the cluster has been configured, it is highly recommended to perform a shutdown operation on both cluster units so they restart at approximately the same time. This is done separately for each unit through the Web Interface or using the CLI command: `Device:/>` shutdown This will ensure both units will try to correctly synchronize with each other. This step can be skipped but may be necessary later if synchronization does not succeed or other problems occur.

After the cluster has been configured, it is highly recommended to perform a shutdown operation on both cluster units so they restart at approximately the same time. This is done separately for each unit through the Web Interface or using the CLI command:

Device:/> shutdown

This will ensure both units will try to correctly synchronize with each other.

This step can be skipped but may be necessary later if synchronization does not succeed or other problems occur.

To verify that the cluster is performing correctly, first use the ha command on each unit. The output will look similar to the following for the master:

Device:/> ha
			
This device is an HA MASTER
This device is currently ACTIVE (will forward traffic)
HA cluster peer is ALIVE

Then use the stat command to verify that both the master and slave have about the same number of connections. The output from the command should contain a line similar to the following:

Connections 2726 out of 128000

The lower number on the left in this output is the current number of connections and the higher number on the right is the maximum number of connections allowed by the license.

The following points are also relevant to cluster setup:

If this is not the first cluster in a network then the Cluster ID must be changed for the cluster so that it is unique (the default value is 0). The Cluster ID determines that the MAC address for the cluster is unique.

Enabling the advanced setting Use Unique Share MAC is recommended so that each interface has its own MAC address. If this is not enabled, interfaces share a MAC address and this can confuse some third party switches.

Make sure that the advanced setting High Buffers (found in System > Advanced Settings > Misc. Settings in the Web Interface) is set to be automatic for both units in the cluster. This setting determines how memory is allocated by cOS Core for handling increasing numbers of connections. A restart of cOS Core is required for a change in this setting to take effect and this can be achieved with the CLI command:
```
Device:/> shutdown
```
Where an HA cluster has a very high number (for example, tens of thousands) of simultaneous connections then it may be necessary to set a high value for this instead of enabling the Dynamic High Buffers option. A very high value for High Buffers can suit situations with large numbers of connections but can have the disadvantage of increasing throughput.

See Section 13.10, Miscellaneous Settings for a full explanation of these settings.

12.3.5. Unique Shared Mac Addresses

For HA setup, cOS Core provides the advanced option Use Unique Shared MAC Address. By default, this is enabled and in most configurations it should not need to be disabled.

Enabling a Unique Shared MAC Address

The effect of enabling this setting is that a single, unique MAC address will be used for each pair of matching hardware interfaces so that, for example, the lan1 interface on the master unit will appear to have the same MAC address as the lan1 interface on the slave unit.

Problem Diagnosis

An HA cluster will function if this setting is disabled but can cause problems with a limited number of switch types where the switch uses a shared ARP table. Such problems can be hard to diagnose which is why it is best to always have the setting enabled.