DMO: optimizing system downtime is timeless…

Last Changed: Q4/2020

Increasing the DMO performance

Blogs:

– Optimizing DMO Performance
– DMO: Handling table comparison checksum errors
– Downtime minimization when upgrading BW systems
– Note 2350730 – Additional Info – SAP BW 7.50 SP04 Implementation

Since the availability of the DMO functionality within the SUM framework at lot of things happened under the hood as well. Beside the Availability of the SAP First Guidance – Migration BW on HANA using the DMO option in SUM (which is updated constantly …)

The latest Version of the SUM/DMO executables – https://support.sap.com/sltoolset

Especially Oracle RDBMS based Systems which covers the majority of our Customer base need to address with some Performance Improvements right from the beginning to ensure a stable and performant migration to BW on HANA despite it´s sizes or other challenges on the way.

 Performance Optimization: Table Migration Durations

You can provide the Software Update Manager with the information about table migration durations from a previous DMO run. SUM uses this data to optimize the performance of subsequent DMO runs on the same system. Although SUM does consider the table sizes for the migration sequence, other factors can influence the migration duration. In other words, more criteria than just the table size have to be considered for the duration of a table migration.

The real durations are the best criteria, but they are only known after the migration of the tables.

ℹ this can improve the downtime up to 50%, due to the more “aggressive” table splitting after the first run and the results in the XML files. During a migration, SUM creates text files with the extension XML which contain the information about the migration duration for each migrated table. The files are created in directory SUM/abap/htdoc/ and are called (create them beforehand, if you want to use the setting from the beginning)

– MIGRATE_UT_DUR.XML for the uptime migration, and
– MIGRATE_DT_DUR.XML  for the downtime migration

Copy the above-mentioned XML-files directly to  the download folder. The creation of the specific file location in the file SAPup_add.par in directory SUM\abap\bin become obsolete with the current Versions of the SUM Versions 1.0/2.0 (September 2020).

➕ DMO: background on table split mechanism

whatever you will find out: You can trust the DMO optimization when it comes to the table splitting. the Algorithm behind is smarter than you think. Overruling this via the manual setup of the file EUCLONEDEFS_ADD.LST is not possible.
In the SAP First Guidance – Using the new DMO to migrate to BW on HANA is also mentioned, how to use the file SUM/abap/htdoc/UPGANA.XML to optimize the runtime further.

ℹ don´t forget the BW Housekeeping Task before you start the DMO procedure, and don´t underestimate the importance saving time and space! the Blog SAP BW-Performance Tuning through System Tables gives you a good overview about the “waste” you have collected in your BW System. Together with the manual table splitting option, you can take over the tables without content.
SAP First Guidance – BW Housekeeping and BW-PCA

Introducing the DMO Benchmark tool

➕ DMO: introducing the benchmarking tool

start the tool with the following URL – https://<host>:1129/lmsl/migtool/<SID>/doc/sluigui

Parallel Processes Configuration

http://server.wdf.sap.corp:1128/lmsl/sumabap/<SID>/set/procpar

Note 1616401 – Parallelism in the Upgrades, EhPs and Support Packages implementations

❗ Do not use more than 32 R3trans Processes and 24 SQL Processes as maximum for the Start. The R3load Processes can be increased/decreased as you have the resources available. Consider their Values divided by 2: e.g. 96 Export and 96 Import Processes on the AppServer will lead to 192 Processes. If you want to increase the migration performance, only the downtime processes are relevant. As the ABAP DDIC is the almost same size despite the DB size, SQL PROCESSES and PARALLEL PHASES can be even on the default values.

❗ For the configuration phase you can use higher values for the uptime/downtime R3load process to allow a higher split value of the tables. Before the downtime process is starting, use the much lower value for R3load to ensure that you always in a optimal range of CPU/RAM/Network usage. Increasing the R3load step by step (e.g. with the value 20) is more effective, than starting with a high value and reducing them. The little Monitoring tool called “nmon” (e.g. AIX, Linux64, etc.) can be very useful in monitoring Hardware and Network resources during the migration process.

– Phases behind DMO R3load parallel export/import during UPTIME and DOWNTIME to target HANA DB
– SUM: introduction to shadow system

ℹ

For the detailed monitoring of the HANA resources you can use the following SAP Notes and enable the HANA Configuration Check Tool

• Note 1943937 – Hardware Configuration Check Tool – Central Note
• Note 1991051 – Hardware Configuration Check Tool – Extended documentation for network test module

If you want to increase the Value for the ABAP Processes, you have to maintain the Instance Parameter 
rdisp/tm_max_no=2000 in the shadow Instance as in the primary Instance to prevent SAP Gateway errors like as follows. 
This also is related to the Value for the PARALLEL PHASES.

 *** ERROR => GwRqDpSendTo: DpLogon failed [gwdp.c       3580]
 *** WARNING => DpLogon: overflow of logon table [dpxxtool2.c  4047]
 *** LOG Q0P=> DpLogon, tm_overflow ( 200) [dpxxtool2.c  4050]

Create a graphical output after the DMO run

Once the files EUMIGRATEUTRUN.LOG and EUMIGRATEDTRUN.LOG are created, you cannot only improve the next DMO run, you can also use these files as input into a graphical representation of the DMO run. With the Release of SUM 1.0 SP14 the extended UI allows you to do realtime monitoring and detailed analysis of existing DMO run´s. See also the Document – SAP First Guidance – Using the new DMO to Migra… | SCN

Example for a DMO run explaining the different phases:

Oracle: Suppress Long-Running Phases

EU_CLONE_DT_SIZES/EU_CLONE_UT_SIZES

During the update with DMO, the following phases can be long-running:
– EU_CLONE_DT_SIZES
– EU_CLONE_UT_SIZES

In the course of these phases, the system updates the database statistics regarding the usage of space that the tables need on the database. The aim is a better distribution of the tables during the system cloning. Before you start the update, you have the option to suppress these long-running phases using the following procedure:

1. Log on to the host where the Oracle database instance is running. Use ora<dbsid> for UNIX system, or user <sapsid>adm for Windows.

2. Open a command prompt and execute the following commands:

brconnect -u / -c -f stats -t oradict_stats -p 8

brconnect -u / -c -f stats -t system_stats -p 8

brconnect -u / -c -f stats -o <schema_owner> -t all -f allsel,collect,space -p 8 -c 5 -force

brconnect -u / -c -f stats -t all -f monit -p 8

3. Add to file SAPup_add.par the following line:
/ORA/update_spacestat = 0
The file SAPup_add.par is located in the subdirectory SUM/abap/bin of the SUM-directory. If this file does not exist yet, create it.

❗ Especially Oracle based RDBMS (which is still the largest SAP customer implementation) need special attention for the DB statistics despite which version you are running. “An old statistic is a dead statistic”. Old could mean 10 seconds or an empty table as well. You can always see in transaction SM50, which table is busy and run an updated statistic with transaction DB20. This can help already a lot, but of course can be time consuming. so have a look to the following SAP Notes as well. Oracle is the RDBMS which need the most attention before you start the DMO process.

❗ Also manually created additional Indexes on the source database can lead to errors in the HANA import described in SAP Note 2007272.There is an Update for the current 7.42 R3load available – See SAP Note 2144285

ℹ

Don´t go for the SAP Notes title and don´t mix up with manual heterogeneous system copy recommendations. DMO is highly optimized is a way, a custom build migration script or monitor would never reach DMO performance and is not supported in this context anyway.

Note 936441 – Oracle settings for R3load based system copy

init.ora/spfile

filesystemio_options              =  setall
disk_asynch_io                    =  true
log_buffer                        =  1048576           
parallel_execution_message_size   =  16384
parallel_threads_per_cpu          =  1
parallel_max_servers              =  number of CPU’s * number of concurrent R3load processes * 2
processes                         =  processes + parallel_max_servers
SQL*Net configuration
Increase SDU_SIZE in listener.ora to 64KB
Increase SDU_SIZE in tnsnames.ora to 64KB

Specify tcp.nodelay = yes in sqlnet.ora

Note 1045847 – ORACLE DIRECT PATH LOAD SUPPORT IN R3LOAD
Note 1741015 – Repartitioning of BW tables
Note 1413928 – Index corruption/wrong results after rebuild index ONLINE
Note 1635605 – CLIENT HANGS ON INSERT INTO TABLE WITH SECUREFILE LOB
Note 1813548 – Database Migration Option (DMO) of SUM
Note 1875778 – Performance Optimization for System Copy Procedures
Note 1918774 – Performance issues when running a SAP Installation / System Copy
Note 1967819 – R3load tools on HANA DDLHDB.TPL: create primary key after load
Note 1981281 – Oracle: Add. Information – Software Update Manager 1.0 SP12
Note 2008547 – Error during conversion of sub-partitioned tables
Note 2007272 – Identify Duplicated Records in Oracle tables
Note 2130908 – SUM tool error to connect HANA DB with target release 7.4 SP8 or higher

➕   so for the most of the systems this example of the file SUM/abap/bin/SAPup_add.par would increase the performance a lot.

*/clonepar/imp/procenv = HDB_MASSIMPORT=YES
*/clonepar/indexcreation = after_load
*/clonepar/clonedurations = <absolute_path>/MIGRATE_UT_DUR.LST,<absolute_path>/MIGRATE_DT_DUR.LST
/ORA/update_spacestat = 0

* retired values/now default values with the current R3load versions (Note 2118195) and the latest SUM Versions (see below).

SAP Kernel handling – always use the latest version of the R3* tools and LibDBSL

During the migration to SAP HANA DMO of SUM has to deal with 3 kernel versions. Those are in detail:

• Kernel currently used by the system for the source DB (e.g. 7.20 EXT for Oracle)
• Kernel for the target release and source DB (e.g. 740 for Oracle – used for shadow system)
• Kernel for the target release and SAP HANA
The kernel currently used by the system can usually be found in /sapmnt/<SID>/exe/…
The other two target kernel versions (for AnyDB and SAP HANA) can be found in the SUM directory.
At the beginning of the migration process those directories look like this:
• SUM/abap/exe < contains the target kernel for AnyDB
• SUM/abap/exe_2nd < contains the target kernel for SAP HANA

During downtime (phase MIG2NDDB_SWITCH) the directories will be switched. After the switch it looks like this:

• SUM/abap/exe (target kernel for AnyDB) has been moved to SUM/abap/exe_1st
• SUM/abap/exe_2nd (target kernel for SAP HANA) has been moved to SUM/abap/exe
  As usual in SUM later on (phase KX_SWITCH) the kernel will be copied from SUM/abap/exe to the system

❗ together with the R3* tools you always exchange the LibDBSL as well for source and target DB. Currently for Kernel 7.42 (which is needed for SAP BW 7.40 SP08 and higher) these latest patches are needed:

Note 2054965 – R3load: TOC for logical table is incomplete in declustering mode after restart
Note 2124912 – R3load sporadically produces empty task files
Note 2118195 – R3load aborts during unicode conversion and declustering
Note 2144285 – R3load can produce duplicates at export if bulk fetch mode is active
Note 2130541 – SAP HANA: Deactivate fast data access in a non-Unicode system
Note 2144274 – R3load migration to SAP HANA database Revision 85.02 can lead to a disk full event

Attention: The Software Update Manager 2.0 SP01 is part the Software Logistics Toolset 1.0
SP stack 22 and exists in parallel to the Software Update Manager 1.0 SP21

* Attention: The Software Update Manager 2.0 SP02 is part the Software Logistics Toolset 1.0 SP stack 23 and exists in parallel to the Software Update Manager 1.0 SP22.

** Attention: The Software Update Manager 2.0 SP03 is part the Software Logistics Toolset 1.0 SP stack 24 and exists in parallel to the Software Update Manager 1.0 SP23.

*** Attention: The Software Update Manager 2.0 SP04 is part the Software Logistics Toolset 1.0 SP stack 25 and exists in parallel to the Software Update Manager 1.0 SP24

Which SUM/DMO Version for which scenario?

(Blog/Graphic by Boris Rubarth, SL tools)

Note 2223738 – Central Note – Software Update Manager 1.0 SP17 [lmt_008]
Note 2277058 – Central Note – Software Update Manager 1.0 SP18 [lmt_004]
Note 2328500 – Central Note – Software Update Manager 1.0 SP19 [lmt_005]
Note 2418924 – Central Note – Software Update Manager 1.0 SP21 [lmt_007]
Note 2472928 – Central Note – Software Update Manager 1.0 SP22 [lmt_004]
Note 2428168 – Central Note – Software Update Manager 2.0 SP00 [lmt_020]
Note 2472850 – Central Note – Software Update Manager 2.0 SP01 [lmt_021]
Note 2529257 – Central Note – Software Update Manager 2.0 SP02 [lmt_022]

One Word about the HANA Version

To keep the additional complexity to minimum, try to stick with the current HANA 1.0 SP12 Revisions, when running the DMO procedure. To find a list of the available SAP HANA Revisions have a look to the 

SAP First Guidance – SAP BW on HANA – Edition 2017
Nevertheless, some of the mentioned HANA settings here are beneficial, despite of the used SAP HANA Version.

MERGE DELTA of <tablename> FORCE REBUILD;

Note 2112732 – Pool/RowEngine/MonitorView allocates large amount of memory

ALTER SYSTEM ALTER CONFIGURATION ('global.ini', 'system') set ('expensive_statement', 'use_in_memory_tracing') = 'false';

Note 1912332 – SQL array DML operations fail with error code “-10709”

ALTER SYSTEM ALTER CONFIGURATION('indexserver.ini','SYSTEM') SET ('distribution', 'split_batch_commands') = 'off' WITH RECONFIGURE;

Note 2136595 – Dump DBIF_DSQL2_SQL_ERROR when doing a BW data load

dbs/hdb/connect_property = CONNECTTIMEOUT=0

Note 2105761 – High memory consumption by RANGE-partitioned column store tables due to missing optimize compression

ALTER SYSTEM ALTER CONFIGURATION ('indexserver.ini', 'system') set ('optimize_compression', 'min_hours_since_last_merge_of_part') = '0'

Note 2039810 – Table does not get merged automatically
Note 1930853 – HdbUserStore contains too many SAP HANA nodes
Note 2229248 – Long runtime in background job “RSUPGRCHECK” during SAP EHP upgrade
Note 2026343 – SP34: DMO/Housekeeping: Performance improvement of PSA Housekeeping tasks

ℹ with the release of SUM 1.0 SP14 and higher, there will also a new and functionality improved and consolidated UI5 for all SUM/DMO procedures (except Dual Stacks) be available.

See also the Blog – Upgrade was never been easier …

Roland Kramer, SAP Platform Architect for Intelligent Data & Analytics
@RolandKramer

“I have no special talent, I am only passionately curious.”