Technical Articles

Roman Gorbenko

December 20, 2019 9 minute read

SAP HANA Backup via backint interface. Step-by-step guide for Veritas NetBackup.

Prerequisites.

You want config backup via backint interface for your SAP HANA multitenant database.

This step-by-step guide provides information about Veritas NetBackup solution

N.B .If you want to use other backup tools, for example, EMS Networker, Veeam Plug-in for SAP HANA, Spectrum Protect for Enterprise Resource Planning, etc this guide will be useful for you too.

SAP Environment.

SAP HANA DB
SAP HANA Cockpit
SAP HANA Studio
Veritas Solution: NetBackup Agent, NetBackup Server and NetBackup OpsCenter

About.

SAP HANA database provides a backup interface, called Backint for SAP HANA database. This interface offers 3rd-party backup tool vendors to connect their product to the backup and recovery capabilities of the SAP HANA database. Backint for SAP HANA is fully integrated into SAP HANA database, i.e. data and log backups can be configured individually to be created and recovered in the 3rd-party backup tool.

Resolution.

Step 1: Install and Config NetBackup Server (Master Server)

Create Policies

NetBackup Management -> Policies

Step 2: Veritas NetBackup Agent installation on HANA DB-host

Substep 2.1. Run installation

Substep 2.2. During the installation process specify Veritas Master server FQDN from Step 1

Step 3: Configuration of Veritas NetBackup Agent

Substep 3.1. Create utl-files

Goto to a folder:

cd /usr/openv/netbackup/ext/db_ext/sap/scripts/sap_oracle/

Create directories for each tenant in HDB:

mkdir <SID>

Create a policy file for each SID from initial utl-file (initSAP.utl)

cp initSAP.utl <SID>/initSAP-<SID>-log.utl

cp initSAP.utl <SID>/initSAP-<SID>-data.utl

Substep 3.2. Edit created a utl-files:

Provide information about “policy” and “schedule” parameters in created utl-files:

nano /<SID>/initSAP-<SID>-log.utl

Substep 3.3. Create and edit node_names.txt file:

nano /usr/openv/ netbackup/ext/db_ext/sap/node_names.txt

provide here FQDN name of Master-server from Step1

Substep 3.4 Create a symbolic link to hdbbackint_script

ln -s /usr/openv/netbackup/bin/hdbbackint_script /usr/sap/<SID>/SYS/global/hdb/opt/ hdbbackint

Step 4. Settings in SAP HANA Cockpit / SAP HANA Studio

Goto section DB Administration-> Configure system properties

Settings for global.ini

[section – backup]

catalog_backup_using_backint = false

log_backup_using_backint = true

data_backup_parameter_file = /usr/openv/netbackup/ext/db_ext/sap/scripts/sap_oracle/<SID>/initSAP-data.utl (for SYSTEMDB / and each tenant)

log_backup_parameter_file= /usr/openv/netbackup/ext/db_ext/sap/scripts/sap_oracle/<SID>/initSAP-log.utl (for SYSTEMDB / and each tenant)

log_backup_using_backint = true (for SYSTEMDB and each tenant)

[section – persistence]

basepath_catalogbackup = /catalogbackup

basepath_databackup = /catalogbackup

N.B. In this case we setup Data/Log backup via backint.

catalogbackup via File to the NFS-share in mount point /catalogbackup

Settings for xsengine.ini

Add or Edit

[section – scheduler]

enabled = true

Step 5: Create Ad-Hoc backup for SYSTEMDB and each tenant

Goto section DB Administration-> Manage database backups -> Create backup

Backup Type: Complete

Destination Type: Backint

Status: Successful

Step 6. Create backup Schedule for SYSTEMDB and each tenant

Goto section DB Administration-> Manage database backups -> Create Schedule

Backup Type: Complete

Destination Type: Backint

Unique Schedule Name: <SID>@<SID>

Start on: Time

Every week on: Days of the week

Step 7. Recover SYSTEMDB and each tenant

Goto Overall Tenant Status.

Stop Tenant – Recover Tenant

Backup to be Used

Check Availability Backup – Backint

Initialize Log Area – No

Recovery – In Progress

Recovery – Done!

Step 8. Maintenance for catalogbackup.

As indicated in Step 4 in this approach:

For full and log backup is used a backint interface.

For catalogbackup is used a remote NFS folder (mount point /catalogbackup).

Keep in mind that information in catalogbackup is redundant

Each next catalogbackup-file contains information about full/log backups + small delta

In this case, we can use such a strategy for catalog-backup files:

Rotation. Keep in NFS -folder only the last three catalogbackup files. Backup to tape and cleanup NFS folder
Cleanup catalogbackup files itself

Consider catalogbackup cleanup-phase in more detail. As described in Note 2399996 we can use for this task HANA Cleaner.

2399996 – How-To: Configuring automatic SAP HANA Cleanup with SAP HANACleaner

On DB-level (Create Role/User):

CREATE ROLE HANA_CLEANING;

GRANT BACKUP ADMIN, CATALOG READ, LOG ADMIN, MONITOR ADMIN, RESOURCE ADMIN, TRACE ADMIN to HANA_CLEANING;

GRANT SELECT, DELETE ON “_SYS_STATISTICS”.”HOST_OBJECT_LOCK_STATISTICS_BASE” TO HANA_CLEANING;

GRANT SELECT, DELETE ON “_SYS_STATISTICS”.”STATISTICS_ALERTS_BASE” TO HANA_CLEANING;

GRANT SELECT, DELETE ON “_SYS_REPO”.”OBJECT_HISTORY” TO HANA_CLEANING;

CREATE USER HANACLEANER1 password <your password for HANACLEANER1 DB USER>;

GRANT HANA_CLEANING to HANACLEANER1;

ALTER USER HANACLEANER1 DISABLE PASSWORD LIFETIME;

On OS-level:

log-on as an <sid>adm os-user and add a new a key to hdbuserstore:

hdbuserstore SET HANACLEANER1KEY hostname:3<instance>15 HANACLEANER1 <your password>

log-on as a <sid>adm os-user and check hdbuserstore:

hdbuserstore LIST HANACLEANER1KEY

Download the Python-script hanacleaner.py from Note 2399996 and copy it to the Scripts-folder:

(/usr/sap/<SID>/HDB<Instance>/exe/python_support/)

Create a shell -file hanacleaner.sh

python /usr/sap/<SID>/HDB<instance>/exe/python_support/hanacleaner.py -fl 25 -fo true -tc 30 -td true -to true -es true -os false -ar 30 -ad true -bd 45 -zb 40 -lr 3 -eh 10 -eu 10 -fo true -op /usr/sap/<SID>/HDB<instance>/hostname/trace

where:

-fl 25 – fragmentation limit DataVolume %

-fo true – displays data volume statistics before and after defragmentation

-tc 30 – minimum retained days for trace files

-td true – displays the trace files that were deleted

-to true – displays trace files before and after the cleanup

-es true – Execute all crucial housekeeping tasks (useful to turn off for investigations with –os=true)

-os false – Prints all crucial housekeeping tasks (useful for debugging with –es=false)

-ar 30 – minimum number retained days of the alerts

-ad true -output deleted alerts

-bd 60 – minimum retained days of backup entries in the catalog

-zb 40 – backup logs compression size limit

-lr 3 – maximum number of free log segments per service

-eh 10 – minimum retained days for handled events

-eu 10 -minimum retained days for unhandled events

-fo true – output fragmentation displays data volume statistics before and after defragmentation

-op /usr/sap/<SID>/HDB<instance>/<hostname>/trace – output path full path of the folder where the hanacleaner logs are written

hanacleaner.sh was executed:

Create a cron job for this script:

crontab -e -u hsdadm 0 1 * * * /path/hanacleaner.sh >> /path/hanacleaner.log 2>&1

Step 9. Monitoring.

Monitoring via Veritas NetBackup OpsCenter (a web-based tool)

Monitor – > Jobs

Step 10. Troubleshooting.

For example In the HANA Cockpit / Hana Studio you see Error

Backup status: Failed

Additional Information: [447] backup could not be completed, [110512] Backint reported ‘BACKINT did not return result for [/usr/sap/<SID>/SYS/global/hdb/backint/DB_<SID>/2019_12_02_04_30_00_databackup_0_1]’ in file ‘/var/tmp/hdbbackint_<SID>.vkrhNE’, [110203] Not all data could be w.

Investigate SAP Notes about

2305517 – Backup with backint aborts after a few minutes

https://apps.support.sap.com/sap/support/knowledge/preview/en/2305517

2243428 – backint “failed with error code ERROR’ in file ‘/var/tmp..” when running HANA backup

https://apps.support.sap.com/sap/support/knowledge/preview/en/2243428

2538877 – Error: Backint exited with exit code 2 instead of 0. console output: No additional Information was received

https://launchpad.support.sap.com/#/notes/0002538877

Investigate backup.log / backint.log / nameserver.trc for additional information.

Log files are created and updated in specific directories. In SAP HANA you will find the trace and log files root directory at : [ /usr/sap/<SID>/HDB<Instance number>< hostname>/trace ] This trace and log root directory contains the SYSTEMDB trace and log files only. Each tenant database has it own directory with a trace and a log files. You can determine this information from a cdtrace command output.

Log-in as <SID>adm

Run the following command : cdtrace

or via such logic:

Path to backup.log (for SYSTEMDB) –

/usr/sap/<sid>/HDB<instance>/<host>/trace/backup.log

/usr/sap/HSP/HDB35/sp1hspdb/trace/backup.log

Path to backint.log (for SYSTEMDB) –

/usr/sap/<sid>/HDB<instance>/<host>/trace/backint.log

Path to backint.log (for TENANTS DB) –

/usr/sap/<sid>/HDB<instance>/<host>/trace/DB_<TANANT_DB_SID>/backup.log

/usr/sap/<sid>/HDB<instance>/<host>/trace/DB_<TANANT_DB_SID>/backint.log

Investigate the Veritas log files:

On the SAP client host, put the following line in the /usr/openv/netbackup/bp.conf file :

VERBOSE = 5

and create log directories :

/usr/openv/netbackup/logs/vnetd

/usr/openv/netbackup/logs/bpcd

/usr/openv/netbackup/logs/backint

chmod – R 0777 /usr/openv/netbackup/logs/backint

Investigate log files in these directories.

Miscellaneous.

NetBackup Agent operation commands:

Stop Agent

cd /usr/openv/netbackup/bin

./bp.kill_all

Start Agent

cd /usr/openv/netbackup/bin

./bp.start_all

Conclusion.

A backup procedure via backint interface was provided.

Useful resources about this topic.

Links:

1730932 – Using backup tools with Backint for HANA

https://launchpad.support.sap.com/#/notes/1730932

1913568 – Support Process for Veritas NetBackup

https://launchpad.support.sap.com/#/notes/1913568

Veritas AdminGuide

https://www.veritas.com/content/support/en_US/doc/18716246-132504715-0/v41275436-132504715

Veritas NetBackup InstallGuide

https://www.veritas.com/support/en_US/doc/27801100-130536338-0/index

My GitHub about this topic

https://github.com/basisteam-io/BasisTeam-Content/tree/master/blog/SAP%20HANA/SAP_BASIS_BACKUP_VIA_BACKINT

Update 10/01/2020.

The golden rule “The hosts file is a file located on your computer that is used as a first point of lookup for DNS hostname resolution.” not working for Veritas Solution. If the hostname of Media Server in /etc/hosts and in DNS are different you will receive hard to understand a permanent error

[447] backup could not be completed, [110507] Backint exited with exit code 1 instead of 0. console output: No additional information was received, [110203] Not all data could be written: Expected 4096 but transferred 0

My kindly advice – don’t use /etc/hosts for Media Server hostname resolution on HANA DB host. Only DNS

2 Comments

You must be Logged on to comment or reply to a post.

Mohamed Chérif BOUKHALFA-BENNAI

March 24, 2021 at 9:22 am

Hello Roman;
I thank you for this relevant presentation.
Is this scenario possible with Veritas BackupExec?

My greetings

Roman Gorbenko

Blog Post Author

April 1, 2021 at 10:05 pm

Hello, Mohamed,

Description from the official site:

"Backup Exec System Recovery is a standalone disk-based backup and recovery solution for small businesses. This simple, cost-effective solution helps to minimize downtime and avoid disaster by easily recovering individual data files/folders or complete Windows systems in minutes. It can scale to larger environments working in conjunction with Backup Exec"

It looks like this software for other purposes.