Veritas Cluster Server overview: how VCS works
Veritas Cluster Server keeps applications running when a server fails. It watches every part of an application, from the network card up to the database, and moves the whole application to another server when something breaks, usually within a minute or two.
Failover, not load balancing
In the most common setup, a failover service group runs on one node at a time. The other node is a hot standby: VCS is running there and watching, but the application is offline. When the active node fails, or an administrator moves the group for maintenance, VCS stops the application on one node (if it still can) and starts it on the other. Clients reconnect to the same virtual IP address.
VCS also supports parallel groups, which run on several nodes at once (used with cluster file systems and Oracle RAC), and clusters of up to dozens of nodes.
Building blocks
- Node (system)
- A server in the cluster.
- Resource
- One thing VCS manages: a NIC, an IP address, a disk group, a volume, a mount point, a database instance, a listener, a process.
- Resource type and agent
- Each resource has a type (
IP,Mount,Oracle…) handled by an agent that knows how to bring it online, take it offline and monitor it. - Service group
- All resources an application needs, with the order they depend on each other. The group is the unit that fails over.
- Dependencies
- Rules such as "the mount needs the volume, the volume needs the disk group". VCS brings resources online from the bottom up and offline from the top down.
A typical database group, from bottom to top:
Oracle ── Listener
│ │
Mount IP
│ │
Volume NIC
│
DiskGroup
Communication: LLT and GAB
- LLT (Low Latency Transport) carries heartbeats and cluster traffic over dedicated private network links, configured in
/etc/llttab. Use at least two independent links, plus a low-priority link over the public network. - GAB (Group Membership and Atomic Broadcast) decides which nodes are members of the cluster and keeps their configuration in step.
/etc/gabtabcontains the command that starts it, for example/sbin/gabconfig -c -n2: wait until two nodes are present before forming the cluster. - HAD, the high-availability daemon, is the VCS engine.
hashadowrestarts it if it dies.
Split brain and I/O fencing
If all heartbeat links fail while both nodes are still running, each might think the other is dead and start the application, and both would write to the same disks. That is split brain, and it corrupts data. I/O fencing (vxfen) prevents it: coordination points (three small SAN disks with SCSI-3 persistent reservations, or coordination point servers) decide which side of a split cluster survives; the losing nodes are stopped from writing and panic. Production clusters with shared storage should always run with fencing enabled.
What happens at startup
- LLT starts and finds the other nodes over the private links.
- GAB waits until the number of nodes in
/etc/gabtabis present, then seeds the cluster. If a node boots alone (the others are down), it waits; an administrator can seed it manually (see debugging). - Fencing starts and registers with the coordination points.
- HAD starts, loads the configuration (from the running cluster if another node is up, otherwise from
main.cf) and probes every resource on every node. - Service groups with an
AutoStartListcome online on the first available node in that list, bringing up resources in dependency order. If a critical resource fails to come online, the group faults on that node and VCS tries the next.
Files and logs
| Path | Contents |
|---|---|
/etc/VRTSvcs/conf/config/main.cf | The cluster configuration: nodes, groups, resources, dependencies |
/etc/VRTSvcs/conf/config/types.cf | Definitions of the resource types |
/etc/llttab, /etc/llthosts | LLT links and node IDs |
/etc/gabtab | How GAB is started |
/etc/vxfenmode, /etc/vxfendg | I/O fencing configuration |
/var/VRTSvcs/log/engine_A.log | The main log: read this first |
/var/VRTSvcs/log/<Type>_A.log | One log per agent: Oracle_A.log, Mount_A.log, IP_A.log … |
Use ls -ltr /var/VRTSvcs/log to find the logs that changed most recently.
Changing the configuration
Change the configuration with commands while the cluster is running; VCS distributes the change to every node.
haconf -makerw # open the configuration for writing
hagrp -modify oragrp SystemList node1 0 node2 1
hares -modify ora_mnt MountPoint /u02
haconf -dump -makero # save to main.cf on all nodes and close
Only edit main.cf by hand when the whole cluster is stopped:
hastop -all -force # stop VCS on all nodes, leave applications running
cd /etc/VRTSvcs/conf/config
cp main.cf main.cf.$(date +%F)
vi main.cf
hacf -verify . # check syntax before starting again
hastart # on this node first, then on the others
A mistake in the configuration can take the application down or stop it from failing over when you need it. Keep a copy of every working main.cf, change one thing at a time, and run a failover test after any change (see testing failover).