SAP ASE HADR Fault Manager – Overview, Installation, and Configuration

This post provides a conceptual overview of the Fault Manager – a key component of the SAP Adaptive Server Enterprise HADR solution – along with a quick summary of the Fault Manager installation process, and instructions on using the command line to configure the Fault Manager.

Fault Manager Overview

The Fault Manager monitors the various components of the HADR environment – Replication Management Agent (RMA), ASE, Replication Server, applications, databases, and the operating system. Its primary job is to ensure the high-availability (zero data loss during failover) of the ASE cluster by initiating automatic failover with minimal manual intervention.

The Fault Manager also lets you:

• Check the state of the databases.
• Restart the databases.
• Check the health of ASE and RMA.
• Resume replication after the failover is complete.
• Avoid a split-brain condition.
• Send alerts through SAP ASE Cockpit.
• Troubleshoot the HADR system. For example, you can use the Fault Manager’s log for diagnostic information.

The Fault Manager resides on its own host, separate from the primary and standby companion servers. It monitors the health of the primary and standby servers, and triggers failover if the primary server becomes unavailable.

How Does the Fault Manager Check the State of Databases?

The Fault Manager uses the SAP Host Agent to check the state of the databases. The SAP Host Agent is a daemon process that runs on all participating nodes.

The HADR system uses the SAP Host Agent to perform several management tasks, such as operating-system monitoring, database monitoring, system instance control, and provisioning. If the primary server is down or if the primary node is down or unreachable, the Fault Manager triggers a failover using the SAP Host Agent.

Fault Manager and Zero Data Loss

The HADR system supports synchronous replication between the primary and standby servers for high availability, so the two servers can stay in sync with zero data loss. If the SAP Replication Server is in synchronous mode before the failure of the primary server, the Fault Manager automatically initiates failover (it checks the heartbeats from the heartbeat clients), ensuring zero data loss.

However, if the SAP Replication Server is in asynchronous mode before the primary server fails, use manual intervention to failover. Manual failover can, of course, result in some data loss.

Installing the Fault Manager

When you’ve configured both the primary and secondary companion sites, you can proceed with the installation of the Fault Manager. The Fault Manager is located on a machine that is separate from the machines hosting the primary and standby companion servers, and uses a separate installer. You don’t need to install the SAP ASE Server on this machine before installing the Fault Manager.

Install the Fault Manager using one of the following options:

• Use the SAP GUI installer with the Fault Manager Response file. In this option, the Fault Manager parameters are auto-populated. However, if required, you can edit these parameters to suit your system requirements.
• Use the SAP GUI installer, and manually enter the required parameters. That is, run the installer without any response file.
• Use the command-line with the Fault Manager Response file.

What’s a Response File?

The Fault Manager Response file is automatically generated when you complete the HADR configuration on the standby site. The response file contains various parameters that govern the Fault Manager installation. For instance, product edition, primary site name, installation directory etc.

This response file (fault_manager_responses.txt) is located in the $SYBASE/log directory. You’ll need to manually copy the response file from the standby site to the machine where you plan to install the Fault Manager.

Installing the Fault Manager using the SAP GUI

The Fault Manager installer (setup.bin) is located in the <ASE_installer>/FaultManager directory. As mentioned earlier, manually copy the response file from the secondary site to the node (host) where you plan to install the Fault Manager. Now, use this command to install the Fault Manager using the response file:
./setup.bin -f <fault_manager_responses.txt>

Follow the installation instructions to complete the installation. The values in the response file automatically show up in the wizard, but you can change those values if necessary.

When the Fault Manager has installed, you’ll see the Configure Fault Manager screen. Click Yes to configure it. If you click No, you can manually configure the Fault Manager later using the sybdbfm utility.

Configuring the Fault Manager from the Command Line (sybdbfm utility)

Use the sybdbfm utility to configure and run the Fault Manager. The sybdbfm utility is located in $SYBASE/FaultManager/bin on the machine hosting the Fault Manager.

Syntax:

sybdbfm [<options>]

Where <options> is one of:

• hadm pf= <full_path_to/SYBHA.pfl> – Starts the Fault Manager process. Requires a profile file.
• hahb pf= <full_path_to/SYBHA.pfl> – Starts the heartbeat process. Requires a profile file. The Fault Manager (running in monitoring mode) internally starts the process in heartbeat mode on the hosts running SAP ASE.
• install[pf= <full_path_to/SYBHA.pfl> ] – Configures the Fault Manager. You can use this parameter:
  • In interactive mode, during which you provide information to command-line prompts to install the Fault Manager.
  • With a profile file, which is provided by the pf= <full_path_to/SYBHA.pfl> parameter. The profile contains the details you would provide at the interactive command prompt.
• uninstall[pf= <full_path_to/SYBHA.pfl> ] – Uninstalls the Fault Manager. You can use this parameter:
  • In interactive mode, during which you provide information to command line prompts to uninstall the Fault Manager.
  • With a profile file, which is provided by the pf= <full_path_to/SYBHA.pfl> parameter. The profile contains the details you would provide at the interactive command prompt.
• check pf= <full_path_to/SYBHA.pfl> – Performs the basic bootstrapping of the Fault Manager to confirm if the details provided in the profile file and the user credentials in the SecureStore file are correct, and that the Fault Manager can run.
• hibernate – Hibernates the Fault Manager. Useful for pausing the Fault Manager for planned maintenance activities on the HADR components (for example, planned failover, and upgrades).
• resume – Resumes running the Fault Manager from hibernation.
• restart – Restarts the Fault Manager using an altered profile file. Used when any profile parameter is modified or added.
• stop – Stops the Fault Manager and all heartbeat processes spawned on the HADR system nodes. Execute stop from the same directory where you started the Fault Manager.
• status – Displays the status of the Fault Manager.
• version – Displays version information, along with other build and operating system version information supported by the Fault Manager.

Username and Password Encryption

The Fault Manager requires username and password combinations to connect to SAP ASE, RMA, SAP ASE Cockpit, and SAP Host Agent. These usernames and passwords are stored in encrypted format in SecureStore. During configuration, the Fault Manager creates usernames and passwords for the HADR users, such as SAP ASE user and RMA user.

You can use the rsecssfx utility to administer SecureStore, and update any changed usernames and passwords. To do so, stop the Fault Manager, update SecureStore using the rsecssfx utility, and then restart the Fault Manager. For more information, refer to the SAP ASE HADR User’s Guide.

Fault Manager Profile Parameters

The Fault Manager includes various profile parameters, which are stored in the Fault Manager profile file, SYBHA.PFL. You can find this file in the install directory of the Fault Manager.

Profile parameters determine the Fault Manager’s behavior during failover. For example, ha/syb/failover_timeout governs the timeout period for failover attempts, while ha/syb/allow_restart determines whether to allow a restart of the primary SAP ASE in case of failure while in ASYNC_REP mode.

Not all profile parameters appear in the profile file after you configure the Fault Manager. The profile file is initially populated with defaults, but you can modify these defaults to match your site’s configuration. For example, you can edit the parameters for the primary site, the companion site, and the parameters related to communication frequency check and timeout.

Restart the Fault Manager for the profile parameter changes to take effect.

Important Fault Manager Profile Parameters Used During Failover

The Fault Manager profile file contains many parameters, but only a couple of them are important from a failover perspective. Here’s what happens during failover:

After failover, you can use the RMA to restore data replication, either automatically (by the Fault Manager), or through manual intervention, depending on the value of the ha/syb/set_standby_available_after_failover profile parameter:

• If the profile parameter = 1, the Fault Manager makes the host available using the sap_host_available RMA command.
• If the profile parameter = 0, you manually issue the sap_host_available command from RMA.

Response File, Sample Response File, Profile File, and Log File

While using the Fault Manager, you create/edit various files, with each file/files serving a specific purpose. Here is a quick summary of these files:

• A response file determines the parameters required to install the Fault Manager. The HADR installer generates a response file named after the site you last created.
• A sample response file is an SAP-provided, ready-made template you can use to install the Fault Manager.
• A profile file determines the Fault Manager’s behavior during failover.
• A log file records all types of events, including startup and shutdown information, errors etc.