Prerequisites.
You want config backup via backint interface for your SAP HANA multitenant database.
This step-by-step guide provides information about Veritas NetBackup solution
N.B .If you want to use other backup tools, for example, EMS Networker, Veeam Plug-in for SAP HANA, Spectrum Protect for Enterprise Resource Planning, etc this guide will be useful for you too.
SAP Environment.
- SAP HANA DB
- SAP HANA Cockpit
- SAP HANA Studio
- Veritas Solution: NetBackup Agent, NetBackup Server and NetBackup OpsCenter
About.
SAP HANA database provides a backup interface, called Backint for SAP HANA database. This interface offers 3rd-party backup tool vendors to connect their product to the backup and recovery capabilities of the SAP HANA database. Backint for SAP HANA is fully integrated into SAP HANA database, i.e. data and log backups can be configured individually to be created and recovered in the 3rd-party backup tool.
Resolution.
Step 1: Install and Config NetBackup Server (Master Server)
Create Policies
NetBackup Management -> Policies
Step 2: Veritas NetBackup Agent installation on HANA DB-host
Substep 2.1. Run installation
Substep 2.2. During the installation process specify Veritas Master server FQDN from Step 1
Step 3: Configuration of Veritas NetBackup Agent
Substep 3.1. Create utl-files
Goto to a folder:
cd /usr/openv/netbackup/ext/db_ext/sap/scripts/sap_oracle/
Create directories for each tenant in HDB:
mkdir <SID>
Create a policy file for each SID from initial utl-file (initSAP.utl)
cp initSAP.utl <SID>/initSAP-<SID>-log.utl
cp initSAP.utl <SID>/initSAP-<SID>-data.utl
Substep 3.2. Edit created a utl-files:
Provide information about "policy" and "schedule" parameters in created utl-files:
nano /<SID>/initSAP-<SID>-log.utl
Substep 3.3. Create and edit node_names.txt file:
nano /usr/openv/ netbackup/ext/db_ext/sap/node_names.txt
provide here FQDN name of Master-server from Step1
Substep 3.4 Create a symbolic link to hdbbackint_script
ln -s /usr/openv/netbackup/bin/hdbbackint_script /usr/sap/<SID>/SYS/global/hdb/opt/ hdbbackint
Step 4. Settings in SAP HANA Cockpit / SAP HANA Studio
Login in SAP Hana Cockpit. Open SystemDB Overview
Goto section DB Administration-> Configure system properties
Settings for global.ini
[section - backup]
catalog_backup_using_backint = false
log_backup_using_backint = true
data_backup_parameter_file = /usr/openv/netbackup/ext/db_ext/sap/scripts/sap_oracle/<SID>/initSAP-data.utl (for SYSTEMDB / and each tenant)
log_backup_parameter_file= /usr/openv/netbackup/ext/db_ext/sap/scripts/sap_oracle/<SID>/initSAP-log.utl (for SYSTEMDB / and each tenant)
log_backup_using_backint = true (for SYSTEMDB and each tenant)
[section - persistence]
basepath_catalogbackup = /catalogbackup
basepath_databackup = /catalogbackup
N.B. In this case we setup Data/Log backup via backint.
catalogbackup via File to the NFS-share in mount point /catalogbackup
Settings for xsengine.ini
Add or Edit
[section - scheduler]
enabled = true
Step 5: Create Ad-Hoc backup for SYSTEMDB and each tenant
Login in SAP Hana Cockpit. Open SYSTEMDB / Tenant Overview
Goto section DB Administration-> Manage database backups -> Create backup
Backup Type: Complete
Destination Type: Backint
Status: Successful
Step 6. Create backup Schedule for SYSTEMDB and each tenant
Login in SAP Hana Cockpit. Open SYSTEMDB / Tenant Overview
Goto section DB Administration-> Manage database backups -> Create Schedule
Backup Type: Complete
Destination Type: Backint
Unique Schedule Name: <SID>@<SID>
Start on: Time
Every week on: Days of the week
Step 7. Recover SYSTEMDB and each tenant
Login in Hana Cockpit. Open SYSTEMDB Overview
Goto Overall Tenant Status.
Stop Tenant - Recover Tenant
Backup to be Used
Check Availability Backup - Backint
Initialize Log Area - No
Recovery - In Progress
Recovery - Done!
Step 8. Maintenance for catalogbackup.
As indicated in Step 4 in this approach:
For full and log backup is used a backint interface.
For catalogbackup is used a remote NFS folder (mount point /catalogbackup).
Keep in mind that information in catalogbackup is redundant
Each next catalogbackup-file contains information about full/log backups + small delta
In this case, we can use such a strategy for catalog-backup files:
Rotation. Keep in NFS -folder only the last three catalogbackup files. Backup to tape and cleanup NFS folder
Cleanup catalogbackup files itself
Consider catalogbackup cleanup-phase in more detail. As described in Note 2399996 we can use for this task HANA Cleaner.
On DB-level (Create Role/User):
CREATE ROLE HANA_CLEANING;
GRANT BACKUP ADMIN, CATALOG READ, LOG ADMIN, MONITOR ADMIN, RESOURCE ADMIN, TRACE ADMIN to HANA_CLEANING;
GRANT SELECT, DELETE ON "_SYS_STATISTICS"."HOST_OBJECT_LOCK_STATISTICS_BASE" TO HANA_CLEANING;
GRANT SELECT, DELETE ON "_SYS_STATISTICS"."STATISTICS_ALERTS_BASE" TO HANA_CLEANING;
GRANT SELECT, DELETE ON "_SYS_REPO"."OBJECT_HISTORY" TO HANA_CLEANING;
CREATE USER HANACLEANER1 password <your password for HANACLEANER1 DB USER>;
GRANT HANA_CLEANING to HANACLEANER1;
ALTER USER HANACLEANER1 DISABLE PASSWORD LIFETIME;
On OS-level:
log-on as an <sid>adm os-user and add a new a key to hdbuserstore:
hdbuserstore SET HANACLEANER1KEY hostname:3<instance>15 HANACLEANER1 <your password>
log-on as a <sid>adm os-user and check hdbuserstore:
hdbuserstore LIST HANACLEANER1KEY
Download the Python-script hanacleaner.py from Note 2399996 and copy it to the Scripts-folder:
(/usr/sap/<SID>/HDB<Instance>/exe/python_support/)
Create a shell -file hanacleaner.sh
python /usr/sap/<SID>/HDB<instance>/exe/python_support/hanacleaner.py -fl 25 -fo true -tc 30 -td true -to true -es true -os false -ar 30 -ad true -bd 45 -zb 40 -lr 3 -eh 10 -eu 10 -fo true -op /usr/sap/<SID>/HDB<instance>/hostname/trace
where:
-fl 25 – fragmentation limit DataVolume %
-fo true - displays data volume statistics before and after defragmentation
-tc 30 - minimum retained days for trace files
-td true - displays the trace files that were deleted
-to true - displays trace files before and after the cleanup
-es true - Execute all crucial housekeeping tasks (useful to turn off for investigations with –os=true)
-os false - Prints all crucial housekeeping tasks (useful for debugging with –es=false)
-ar 30 - minimum number retained days of the alerts
-ad true -output deleted alerts
-bd 60 - minimum retained days of backup entries in the catalog
-zb 40 - backup logs compression size limit
-lr 3 - maximum number of free log segments per service
-eh 10 - minimum retained days for handled events
-eu 10 -minimum retained days for unhandled events
-fo true - output fragmentation displays data volume statistics before and after defragmentation
-op /usr/sap/<SID>/HDB<instance>/<hostname>/trace - output path full path of the folder where the hanacleaner logs are written
hanacleaner.sh was executed:
Create a cron job for this script:
crontab -e -u hsdadm 0 1 * * * /path/hanacleaner.sh >> /path/hanacleaner.log 2>&1
Step 9. Monitoring.
Monitoring via Veritas NetBackup OpsCenter (a web-based tool)
Monitor - > Jobs
Step 10. Troubleshooting.
For example In the HANA Cockpit / Hana Studio you see Error
Backup status: Failed
Additional Information: [447] backup could not be completed, [110512] Backint reported 'BACKINT did not return result for [/usr/sap/<SID>/SYS/global/hdb/backint/DB_<SID>/2019_12_02_04_30_00_databackup_0_1]' in file '/var/tmp/hdbbackint_<SID>.vkrhNE', [110203] Not all data could be w.
Investigate SAP Notes about
2305517 - Backup with backint aborts after a few minutes
2243428 - backint "failed with error code ERROR' in file '/var/tmp.." when running HANA backup
2538877 - Error: Backint exited with exit code 2 instead of 0. console output: No additional Information was received
Investigate backup.log / backint.log / nameserver.trc for additional information.
Log files are created and updated in specific directories. In SAP HANA you will find the trace and log files root directory at : [ /usr/sap/<SID>/HDB<Instance number>< hostname>/trace ] This trace and log root directory contains the SYSTEMDB trace and log files only. Each tenant database has it own directory with a trace and a log files. You can determine this information from a cdtrace command output.
Log-in as <SID>adm
Run the following command : cdtrace
or via such logic:
Path to backup.log (for SYSTEMDB) -
/usr/sap/<sid>/HDB<instance>/<host>/trace/backup.log
/usr/sap/HSP/HDB35/sp1hspdb/trace/backup.log
Path to backint.log (for SYSTEMDB) -
/usr/sap/<sid>/HDB<instance>/<host>/trace/backint.log
Path to backint.log (for TENANTS DB) -
/usr/sap/<sid>/HDB<instance>/<host>/trace/DB_<TANANT_DB_SID>/backup.log
/usr/sap/<sid>/HDB<instance>/<host>/trace/DB_<TANANT_DB_SID>/backint.log
Investigate the Veritas log files:
On the SAP client host, put the following line in the /usr/openv/netbackup/bp.conf file :
VERBOSE = 5
and create log directories :
/usr/openv/netbackup/logs/vnetd
/usr/openv/netbackup/logs/bpcd
/usr/openv/netbackup/logs/backint
chmod - R 0777 /usr/openv/netbackup/logs/backint
Investigate log files in these directories.
Miscellaneous.
NetBackup Agent operation commands:
Stop Agent
cd /usr/openv/netbackup/bin
./bp.kill_all
Start Agent
cd /usr/openv/netbackup/bin
./bp.start_all
Conclusion.
A backup procedure via backint interface was provided.
Useful resources about this topic.
Links:
1730932 - Using backup tools with Backint for HANA
1913568 - Support Process for Veritas NetBackup
Veritas AdminGuide
Veritas NetBackup InstallGuide
My GitHub about this topic
Update 10/01/2020.
The golden rule "The hosts file is a file located on your computer that is used as a first point of lookup for DNS hostname resolution." not working for Veritas Solution. If the hostname of Media Server in /etc/hosts and in DNS are different you will receive hard to understand a permanent error
[447] backup could not be completed, [110507] Backint exited with exit code 1 instead of 0. console output: No additional information was received, [110203] Not all data could be written: Expected 4096 but transferred 0
My kindly advice - don't use /etc/hosts for Media Server hostname resolution on HANA DB host. Only DNS
- SAP Managed Tags:
2 Comments
