Encrypting an SAP System on a Db2 for LUW Database – Reduce Downtime by Exploiting the HADR Feature

You have an SAP system running on an unencrypted Db2 database and want to encrypt your existing database? To do so, you can create a backup image and restore it into an encrypted database.
But what about potential downtimes that come with it?

The Downside (aka Downtime)

During the restore, your database is not available. The downtime depends on the size of the database and the time it needs to restore.

The Upside: You Can Reduce the Downtime

To reduce the downtime needed for setting up Db2 native encryption and the database restore, you can use the Db2 HADR feature. For a short introduction, have a look at Configuring native encryption in an HADR environment – IBM Documentation. However, this procedure is not very detailed. That’s why in this blog, I’d like to provide you with a more detailed and easier to use description adapted to the SAP environment.

Overview

The procedure will only have two short downtime windows where the connectivity of your SAP system is switched to the second host and back to your original database host. Since HADR will only be used temporarily, I do not install a cluster manager for automatic failover orchestration.

My SAP system is running with a database on host A, which is not encrypted.
The idea is to add host B as a standby database and encrypt the standby database while setting up the HADR system. The database on host A is still available during this step.

The next step is to use host B as the database server by switching the HADR roles. During this change to your system landscape, there will be a short downtime for configuration changes and the role switch of the HADR members.

While your SAP system runs with host B as the database server, you stop HADR. To encrypt the database on host A, you set it up as a new standby database for the database on host B with an encrypted database.

You can then switch back to host A as the primary database server with an encrypted database. There will be a second short downtime for configuration changes and the role switch of the HADR members.

Optimize Your Downtime Even More

For further optimization, check out the SAP Graceful Maintenance Tool (GMT): The GMT enables you to make use of the micro-outage feature to optimize the takeover for further reduction of the downtime while encrypting your existing database (for more information, see Graceful Maintenance Tool (GMT)).

At this point, your SAP system has an encrypted database on host A, and you can remove the temporary HADR setup.

Read on for a more granular step-by-step description of the process of encrypting the database of your SAP system with a reduced downtime.

Procedure

Let’s start on host A with the unencrypted database <DBSID> and with SSL client-server-communication configured between the SAP application server and the IBM Db2 database.

Preparations

1. Take an online backup on host A:

   db2 backup database <DBSID> online to /db2/db2<dbsid>/backup.Host_A/online.backup compress include logs

2. Export SAP profiles from the global host to the new database standby server (host B) and mount them on the standby:

   mkdir -p /sapmnt/<SAPSID>
   mount -t nfs  Host_A:/sapmnt/<SAPSID> /sapmnt/<SAPSID>

Installation of HADR standby

3. Set up your HADR standby on host B:

   Use the SAP installer (SWPM) to set up the HADR standby to get a database instance that is configured for encryption. In this example, I will use a local keystore for Db2 native encryption.
   You can also use a centralized key manager. Please refer to the IBM Db2 documentation for more information about keystore selection.
   Optionally, you can configure SSL communication between your primary and standby servers, which I would recommend. In this example, I will skip this step as it is not relevant for the procedure. For more information, see https://www.ibm.com/docs/en/db2/11.5?topic=tcd-configuring-tls-communication-between-primary-standby-hadr-servers.

   Run SWPM on host B:
   <Product> – > System Copy -> Target System -> Distributed System -> Based on AS ABAP -> Database Instance

   1. Choose custom mode.
   2. Specify the profile directory.
   3. In the dialog step “SAP System Database”, choose copy method “Homogeneous System Copy (Database Copy Method)”:
   4. Specify the following as on the source system (host A):
      – ABAP database schema and ABAP connect user
      – User ID for db2<dbsid>
      – User ID for sap<sid>
      – Group IDs for db<dbsid>adm, db<dbsid>ctl, db<dbsid>mnt, and db<dbsid>mon
      – User ID for sapadm
      – RDBMS DVD (same level as on the source system)
   5. In the dialog step “IBM Db2 for Linux, UNIX, and Windows – Encryption“, select “Use Db2 Native Encryption”. This will configure the Db2 instance for native encryption.
      Select “Use Secure Sockets Layer (SSL)” if you use this on your source system.
   6. In the dialog step “IBM Db2 for Linux, UNIX, and Windows – Db2 Native Encryption”, choose the type of keystore to use:
      “Use Local Keystore” or “Use Centralized Keystore”
      (Choose “centralized keystore” only if you have a keystore manager.)
      In this example here, I’m choosing the option “Use Local Keystore”:
   7. Specify the keystore directory. The example uses the default provided by SWPM.
   8. Enter a password for your local keystore and keep it in a safe place.
   9. Specify the “Master Key Label”. I’m using the SAP default here:
   10. When using SSL client server communication, specify the following:
       1. “Password of Local Keystore for SSL Client-Server Communication”
       2. “Self-Signed Certificate Label”
   11. In the dialog step “IBM Db2 for Linux, UNIX, and Windows – System Copy for HADR Purposes”, select “Using system copy with backup/restore for HADR purposes”.
       This will stop SWPM to allow you to restore your database.
   12. In the dialog step “IBM Db2 for Linux, UNIX, and Windows – Database Communication”, specify the port values as on your source system:
       “Database Communication Port”
       Communication Between Partitions “First Port” and “Last Port”
   13. In the exit step with the message “Sapinst will exit now. Restore the database backup. After restoring, continue with your HADR configuration.”, cancel SWPM:
4. Restore your unencrypted backup to an encrypted database.
   1. Create the storage paths used by your source database, for example, as user db2<dbsid>:

      mkdir /db2/<DBSID>/sapdata1 (1 through 4)
      mkdir /db2/<DBSID>/saptmp1 (1 through 4)

   2. Now restore the database.
      To do so, you must specify the master key label (see SWPM dialog) and the encryption options.
      Example:

      db2 restore database <DBSID> from /mnt/backup.Host_A/ ENCRYPT CIPHER AES KEY LENGTH 256 MASTER KEY LABEL 'sap_db2<dbsid>_<Host_B>_dbencr_000'

   3. Check encryption after the restore:

      db2 get db cfg for <DBSID> | grep -i encrypt
      Encryption Library for Backup                 (ENCRLIB) = libdb2encr.so
      Encryption Options for Backup                (ENCROPTS) = CIPHER=AES:MODE=CBC:KEY LENGTH=256
      Encrypted database                                      = YES

5. Set up the specific HADR configuration. For more information about HADR, see https://ibm.github.io/db2-hadr-wiki/.
   1. Create ports for HADR on both hosts.
      Example:

      db2<dbsid>_hadr 5920/tcp

      On host A and host B:

      db2 update db cfg for <DBSID> using HADR_REMOTE_SVC db2<dbsid>_hadr

   2. On the primary and standby database servers, set hadr_local_host, hadr_local_svc, and hadr_syncmode configuration parameters.

      On the primary (host_A), run the following as user db2<dbsid>:

      db2 "update db cfg for <DBSID> using
        HADR_LOCAL_HOST Host_A
        HADR_REMOTE_HOST Host_B
        HADR_LOCAL_SVC db2<dbsid>_hadr
        HADR_REMOTE_SVC db2<dbsid>_hadr
        HADR_REMOTE_INST db2<dbsid>"

      On the standby (host B), run the following as user db2<dbsid>:

      db2 "update db cfg for <DBSID> using
        HADR_LOCAL_HOST Host_B
        HADR_REMOTE_HOST Host_A
        HADR_LOCAL_SVC db2<dbsid>_hadr
        HADR_REMOTE_SVC db2<dbsid>_hadr
        HADR_REMOTE_INST>db2<dbsid>"

6. On the standby start HADR operations (host B):

   db2 START HADR ON DB <DBSID> AS STANDBY

    

7. On the primary start HADR operations (host A):

   db2 START HADR ON DB <DBSID> AS PRIMARY

8. Check that HADR is running:

   db2pd -hadr -db <DBSID>

Now you have an HADR system with the primary database on host A and an encrypted standby database on host B.

9. If your system is configured for SSL client-server communication, you need to copy the SSL certificate from host A. During the setup of the HADR standby with SWPM, the Db2 instance on host B was configured for SSL client-server communication. You can either export the SSL certificate from host A and import it on host B, or copy the keystore file from host A to host B.
   In our example, I copy the keystore from host A to host B.
   1. On host B as user db2<dbsid>:

      db2 deactivate db <DBSID>
      cd keystore
      mv sapdb2<dbsid>_ssl_comm.kdb sapdb2<dbsid>_ssl_comm.kdb.backup
      mv sapdb2<dbsid>_ssl_comm.sth sapdb2<dbsid>_ssl_comm.sth.backup
      scp db2<dbsid>@Host_A:/db2/db2<dbsid>/keystore/sapdb2<dbsid>_ssl_comm.kdb sapdb2<dbsid>_ssl_comm.kdb
      scp db2<dbsid>@Host_A:/db2/db2<dbsid>/keystore/sapdb2<dbsid>_ssl_comm.sth sapdb2<dbsid>_ssl_comm.sth
      db2 update dbm cfg using SSL_SVR_LABEL sap_db2<dbsid>_<Host_A>_ssl_comm_000
      db2stop
      db2start
      db2 activate db <DBSID>

   2. Check your HADR status:

      db2pd -hadr -db <DBSID>

      The HADR_STATE should be PEER.

 

Switch the database host – This a DOWNTIME phase

10. Now you must adapt your SAP system to use host B as its database server.
    1. Prepare the db2cli.ini of your SAP system to use the standby on host B as the database server.

       [<DBSID>]
       Database=<DBSID>
       Protocol=tcpip
       Hostname=<Host_A> <= replace here with <Host B>
       Servicename=5912
       Security=ssl
       SSLServerCertificate=/usr/sap/<SAPSID>/SYS/global/SSL_client/sap_db2_<Host_A>_ssl_comm_000.arm
       [COMMON]
       Diagpath=/usr/sap/<SAPSID>/SYS/global/db6/db2dump

    2. Stop your SAP application servers.
    3. Switch the HADR roles. Host B will now be the primary database.
       As user db2<dbsid> on host B, run the following:

       db2 takeover hadr on db <DBSID>

    4. Check your HADR status:

       db2pd -hadr -alldbs

    5. Test the connection on an application server.
       As user <sid>adm, run the following:

       R3load -testconnect

    6. Start your SAP application servers.
11. Make sure your SAP system is running successfully with host B as the database server.

 
This is the end of the first downtime phase.
 

Replace the Database on Host A with an Encrypted Database

The next steps will replace the database on host A with a new backup from host B to encrypt the database on host A.

12. Take a new backup on host B.
    1. Optional: Set ENCRLIB to use encryption and compression for your backup.

       db2 connect to <DBSID>
       db2 update db cfg for <DBSID> using ENCRLIB ~/sqllib/lib64/libdb2compr_encr.so
       db2 terminate

    2. Backup your database.

       db2 backup database <DBSID> online to /db2/db2<dbsid>/backup.<Host_B>/online/ include logs

13. Stop HADR.
    1. On host B, as db2<dbsid>:

       db2 stop hadr on db <DBSID>

    2. On host A, as db2<dbsid>:

       db2 deactivate db <DBSID>
       db2 drop db <DBSID>

14. Copy the keystore file used for encryption from host B to host A.
    1. On host B, as db2<dbsid>:

       scp ~/keystore/sapdb2<dbsid>_db_encr.* db2<dbsid>@<Host_A>:/db2/db2<dbsid>/keystore/

       If you use a centralized key manager, you need to make sure that both hosts can access the key manager.

    2. On host B, retrieve the values:

       db2 get dbm cfg | grep -i keystore_
       Keystore type         KEYSTORE_TYPE) = PKCS12
       Keystore location     (KEYSTORE_LOCATION) = db2/db2<dbsid>/keystore/sapdb2<dbsid>_db_encr.p12

    3. On host A, update DBM CFG:

       db2 update dbm cfg using
         KEYSTORE_TYPE PKCS12
         KEYSTORE_LOCATION /db2/db2<dbsid>/keystore/sapdb2<dbsid>_db_encr.p12
       db2stop
       db2start

15. Restore the database on host A and adapt the HADR settings.
    1. Restore the database:

       db2 restore database <DBSID> from /mnt/backup.<Host_B>/online/ encrypt

    2. On host A update the DB configuration and start HADR:

       db2 "update db cfg for <DBSID> using
         HADR_LOCAL_HOST Host_A
         HADR_REMOTE_HOST Host_B
         HADR_LOCAL_SVC db2<dbsid>_hadr
         HADR_REMOTE_SVC db2<dbsid>_hadr
         HADR_REMOTE_INST db2<dbsid>"
       db2 start hadr on database <DBSID> as standby

    3. On host B start HADR:

       db2 start hadr on database <DBSID> as primary

    4. Check on host A for the encrypted database:

       db2 get db cfg for <DBSID> | grep -i encr
       Encryption Library for Backup (ENCRLIB) = /db2/db2<dbsid>/db2_software/lib64/libdb2compr_encr.so.1
       Encryption Options for Backup                (ENCROPTS) = CIPHER=AES:MODE=CBC:KEY LENGTH=256
       Encrypted database                                      = YES

Switch the database host back – This a DOWNTIME phase

16. Now you must adapt your SAP system to use host A as the database server.
    1. Stop your SAP application servers.
    2. Prepare the ini of your SAP system to use the standby on host A as the database server:

       [<DBSID>]
       Database=<DBSID>
       Protocol=tcpip
       Hostname=<Host_B> <= replace here with <host A>
       Servicename=5912
       Security=ssl
       SSLServerCertificate=/usr/sap/<SAPSID>/SYS/global/SSL_client/sap_db2_<Host_A>_ssl_comm_000.arm
       [COMMON]
       Diagpath=/usr/sap/<SAPSID>/SYS/global/db6/db2dump

    3. Switch the HADR roles. host A will now be the primary database.
       As user db2<dbsid> on host A:

       db2 takeover hadr on db <DBSID>

    4. Check your HADR status with db2pd -hadr -alldbs.
    5. Test the connection on an application server.
       As user <sid>adm, run the following:
       R3load -testconnect

Now you have an encrypted database on host A and HADR with host B as standby.

17. Start your SAP application servers.
18. Verify that your SAP system is running successfully with host A as the database server.

 
Note: In this example, I don’t take care of the log files that are created while host B has the primary role in the HADR setup. If you want to have a continuous log stream, you must make sure to archive the logs created on host B.
The HADR setup will also work without shared log archives, but manual intervention may be required in certain operations. If the archived logs are not accessible to both the HADR primary and standby hosts, some processes may take longer, e.g., remote catchup or the manual copying of log files.
For more information, see Log archiving configuration for Db2 high availability disaster recovery (HADR) – IBM Documentation and Db2 HADR Wiki (ibm.github.io).

 
This is the end of the second downtime phase.
 

Clean up the HADR setup

19. Now you can remove the HADR standby and get back to your standalone database server.
    Since I used HADR in this example only to reduce the downtime while encrypting the database, you now clean up the HADR standby on host B.
    1. On host A, as db2<dbsid>:
       1. Stop HADR.

          db2 stop hadr on database <DBSID>

       2. Delete the HADR parameter values in the database configuration that you have configured on host A. In this example, the HADR parameters are:
          HADR_LOCAL_HOST, HADR_REMOTE_HOST, HADR_REMOTE_SVC, HADR_REMOTE_INST

          db2 update db cfg for HAT using
            HADR_LOCAL_HOST NULL
            HADR_LOCAL_SVC NULL
            HADR_REMOTE_HOST NULL
            HADR_REMOTE_SVC NULL
            HADR_REMOTE_INST NULL

    2. On host B, as db2<dbsid>:
       1. Stop HADR.

          db2 deactivate database <DBSID>
          db2 stop hadr on database <DBSID>

       2. Unmount the file systems from the application servers that are no longer needed:

          umount /sapmnt/<SAPSID>

    3. Clean up the database server on host B.
       1. Drop the database on host B by running the following command as user db2<dbsid>:

          db2 drop database <DBSID>
          db2stop

       2. Delete the Db2 instance on host B by running the following command as user root:

          db2idrop db2<dbsid>

       3. Uninstall Db2 software on host B.
       4. Clean up the file systems on host B:

          /db2/<DBSID>
          /db2/db2<dbsid>

       5. Delete the following SAP users on host B:
          <sid>adm, sap<sid>, db2<dbsid>

     

Conclusion

To encrypt an existing database, you can restore from a backup. The downtime is the time needed to restore your database and may result in a longer downtime of your SAP system.
In this example, I used a temporary HADR setup to minimize the downtime as the database is still accessible during the restore. There are only two short downtimes when the connection of the applications servers is being switched from the database on host A to host B and vice versa.

You can also use an existing HADR setup with unencrypted databases by following the basic idea of this example to first build up the standby database with an encrypted database and then switch the roles of the HADR hosts before restoring the other host also with an encrypted backup as standby.

For further optimization of your downtime, you might want to use a virtual IP for your database host.  This can be helpful because you would switch the virtual IP address to the other database host when the HADR takeover happens. This eliminates the need to update the db2cli.ini files on the application servers. Additionally, you can use the SAP Graceful Maintenance Tool (GMT).

Combining these procedures could potentially further reduce your downtime.