Phases behind DMO R3load parallel export/import during UPTIME and DOWNTIME to target HANA DB
After completed several DMO projects (BW, CRM and ERP), in order to refresh and a good way retaining memory, is to write and share knowledge gained on DMO parallel export/import phase, by observing and studying its behavior via logs, migration and command files in each and every project done apart from those good sources from sap guide and notes, great blogs and doc in SCN.
In this blog, will merely focusing on how DMO works its magic on migration upgraded/updated shadow repository during uptime and application data during downtime to target HANA DB as depicted in picture A and B, step 2a and 2b (highlighted in red)
Picture A: (Source credit to @Roland Kramer and @Boris Rubarth, Thanks!)
Picture B: (Source credit to @Roland Kramer and @Boris Rubarth, Thanks!)
Picture C shown DMO specific phases behind uptime migration during Preprocessing/ Shadow Import. Will talk about the phases highlighted in red, such as how shadow repository and its objects are created and move to HANA Database, how DMO knows which R3load, kernel and binaries to use whilst there are 2 different databases (in our case, source = oracle and target = HDB).
From the above, we know that upgraded/updated shadow repo created in source is ready to move to HANA Database, where clone size will be calculated in 2 groups: UT and DT and based on the object in PUTTB_SHD tables.
UT = system table (eg: DD*, REPO* and etc.)
DT = data table
PUTTB_SHD = control tables for shadow import during upgrade, tables that needed to copy and import to shadow
Example syntax for (variables are vary from phase to phase)
Selecting from ‘PUTTB_SHD’ with condition ‘( ( ( CLONE == “U” or CLONE == “B”) and ( SRCTYPE == “J” or SRCTYPE
== “T” or SRCTYPE == “P” or SRCTYPE == “C” ) and SRCFORM == “T” ) or ( ( CLONE == “S” or CLONE == “C” or CLONE == “F”
or CLONE == “G” or CLONE == “T” or ( CLONE == “U” and FLAGNEW == “X”) ) and ( DSTTYPE == “J” or DSTTYPE == “T” or DSTTY
PE == “P” or DSTTYPE == “C” ) and DSTFORM == “T” ) )’.
Selecting from ‘PUTTB_SHD’ with condition ‘( ( CLONE == “D” or CLONE == “C” or CLONE == “G” or CLONE == “T” )
and ( SRCTYPE == “J” or SRCTYPE == “T” or SRCTYPE == “P” or SRCTYPE == “C” ) and SRCFORM == “T” )’.
Directory migrate_ut & migrate_dt will be created by phase EU_CLONE_MIG_UT_PRP and EU_CLONE_MIG_DT_PRP subsequently in /SUM/abap/
Both migrate_ut & migrate_dt directory contains .CMD, .STR and other files generated by R3ldctl. .TSK files will generated by R3load during export/import with migration result for each table (EXP = Export files ; IMP = IMPORT files)
EU_CLONE_MIG_*T_PRP : Prepare the tables COUNT (*); split tables with certain threshold, produce list of shadow tables and views to be imported, and other details information into the bucket file – MIGRATE_UT.BUC
EU_CLONE_MIG_*T_CREATE: R3load (HANA) run to create table structure in HANA.
How to verify?
There are just MIGRATE_UT_CREATE_*_IMP.TSK but no *_EXP.TSK exist in SUM/abap/migrate_ut_create and SUM/abap/migrate_dt_create. You’ll see object type (T) and action (C) in .TSK files.
Example: Random check on several .TSK file return with object type Table and action Create (in bold)
T SXMS_SEQUENCE C ok
T SXMS_EO_RETRY_ER C ok
T SXMS_CUST_HDR C ok
T SXMSPLSRV C ok
T SXMSCONFDF C ok
T SXI_LINK C ok
T ARCH_IDX_S C ok
T CRMC_ICSS_REG C ok
T CRMD_DHR_HSLSQUO C ok
T PAT13 C ok
T ARCH_OCLAS C ok
T CRMC_ICSS_IO_ATR C ok
Further explain the syntax in .TSK file:
EU_CLONE_MIG_UT_RUN (UPTIME): Entries of *UT* group tables are exported from shadow repo and imported to HANA in parallel. R3load pairs are doing the export and import. The first R3 load (part of the shadow kernel) is exporting the data, the second R3load (part of the target kernel) is importing the data into SAP HANA DB.
Both R3loads are running in parallel on the same host. No export files (dump files) are created because the data transfer between the R3load pair happens through the main memory of the host. This R3load option is called memory pipes (currently only for non-windows hosts).
To understand more, refer to 2 great blogs shared by Boris Rubarth DMO: technical background and DMO: comparing pipe and file mode for R3load
This is proven in MIGRATE_UT_*_EXP.CMD and MIGRATE_UT_*_IMP.CMD file as you can see, ‘PIPE’ is used:
Also, you can see the updated time for export and import .TSK is identical or close to each other.
Mar 22 11:16 MIGRATE_UT_00010_IMP.TSK
Mar 22 11:16 MIGRATE_UT_00010_EXP.TSK
Mar 22 11:16 MIGRATE_UT_00001_IMP.TSK
Mar 22 11:16 MIGRATE_UT_00008_IMP.TSK
Mar 22 11:16 MIGRATE_UT_00008_EXP.TSK
Mar 22 11:16 MIGRATE_UT_00009_IMP.TSK
Mar 22 11:16 MIGRATE_UT_00009_EXP.TSK
Mar 22 11:17 MIGRATE_UT_00014_IMP.TSK
Mar 22 11:17 MIGRATE_UT_00014_EXP.TSK
By the way, how SUM-DMO knows which R3loads/binaries to use since there’s shadow kernel and target HANA Kernel?
DMO distinguish them with source DB (Shadow Kernel) extracted to SUM/abap/exe whilst Target HANA Kernel to SUM/abap/exe_2nd/ during configuration phase.
Result end of SUM Configuration Phase:
R3load_25-10012508.SAR PATCH UNPACK_EXE OK SAP kernel patch: R3load ,Release: 741
R3load_25-10012508.SAR PATCH UNPACK_EXE2ND OK SAP kernel patch: R3load ,Release: 741
dw_25-10012457.sar PATCH UNPACK_EXE OK SAP kernel patch: disp+work ,Release: 741
dw_25-10012457.sar PATCH UNPACK_EXE2ND OK SAP kernel patch: disp+work ,Release: 741
Above phases are run during UPTIME, and only EU_CLONE_MIG_UT_RUN was executed but not EU_CLONE_MIG_DT_RUN. Again, refer to step 2b in both picture A and B, application data will only move to target Database (HANA) once enter to DOWNTIME.
Picture D: Application data migrated to target Database (HANA) via phase EU_CLONE_MIG_DT_RUN:
EU_CLONE_MIG_DT_RUN (DOWNTIME): At downtime, entries of application data table (DT) are exported from shadow repo and imported to HANA in parallel, using the pairs of R3load same as phase EU_CLONE_MIG_UT_RUN.
Lastly, consistency of migrated content is checked by COUNT(*) on each table in the source and in the target database. These can be maintain/manipulate in /bin/EUCLONEDEFS_ADD.LST, reference to /bin/EUCLONEDEFS.LST with option below:
Ignlargercount -> apply when table might change during cloning (reposrc)
Igncount -> table count will be ignored
Nocontent -> table doesn’t exist in HANA (DBATL; SDBAH – DB specific tables)
noclone -> table doesn’t exist (/BIC* – BW temp table)
Hope this blog will helps other to understand more on DMO. Please rectify me for any incorrect statement. Extra input and info share are greatly welcome!