Optimizing DMO Performance
When migrating an existing SAP system to the SAP HANA database using SUM with database migration option (DMO), several ways exist to optimize the performance and reduce the downtime.
This blog covers the topics benchmarking, optimization and analysis step-by-step. Therefore you should read and follow the steps in sequence
The following graphic gives you an overview of the migration process and will be used below to visualize the performance optimization options.
Optimizing the standard DMO performance
Preparation steps
The DMO uses tables from the nametab. Therefore it is recommended to clean up the nametab before starting a DMO run. Proceed as follows:
1.) Start transaction DB02 (Tables and Indexes Monitor) and choose “Missing tables and Indexes”
2.) Resolve any detected inconsistencies
If you do not perform this step, the DMO run may stop with warnings in the roadmap step “Preparation”.
Benchmarking Tool
Before you start a complete DMO test run, we highly recommend using the benchmarking tool to evaluate the migration rate for your system, to find the optimal number of R3load processes and to optimize the table splitting.
Start the benchmarking mode with the following addresses:
http://<hostname>:1128/lmsl/migtool/<sid>/doc/sluigui
or
https://<hostname>:1129/lmsl/migtool/<sid>/doc/sluigui
This opens the following dialog box:
Benchmark Export
Use this option when you want to simulate the export of data from the source system.
Proceed as follows in the dialog box:
1.) Select the option “Benchmark migration”
2.) Select the option “Benchmark export (discarding data)”
This selection will run a benchmark of the data export and discard the data read
from the source database (source DB).
Note:
a) Always start with the benchmark of the export to test and optimize the performance of your source DB.
Since almost the complete content of the source DB needs to be migrated to the SAP HANA database, additional load is generated on the source DB, which differs from the usual database load of a productive SAP system.
This is an essential part for the performance of the DMO process, since on the one hand parts of the data is already transferred during the uptime while users are still active on the system. On the other hand the largest part of the data is transferred during the downtime. Therefore you have to optimize your source DB for the concurring read access during uptime to minimize the effect on active business users and also for the massive data transfers during the downtime to minimize the migration time.
b) Always start with a small amount of data for your first benchmarking run.
This will avoid extraordinary long runtimes and allow you to perform several iterations.
The idea behind this is that performance bottlenecks on the source DB can already be found with a short test run while more iterations are useful to verify the positive effects of source DB configuration changes on the migration performance.
However, too short runtimes should also be avoided, since the R3load processes and the database need some time at the beginning to produce stable transfer rates.
We recommend about 100GB or less than 10% of the source database size for the first run.
The ideal runtime of this export benchmark is about 1 hour.
3.) Select the option „Operate on all tables“.
Define the sample size as percentage of the source database size.
- Example: your source database has a size of 1 TB, then using 10% as “percentage of DB size for sample” will result in around 100 GB size for the sample.
Define the size of the largest table in the sample as percentage of the source database size. The tool will then only consider tables for the sample which size is smaller than this percentage of the DB size.
- Example: your source database has a size of 1 TB. One of the tables <TABLE1> has a size of 15 GB. You have chosen 1 % as “Size of largest table in sample” which is equivalent to around 10 GB. The tool will then not select <TABLE1> as part of the sample because the size exceeds the given limit.
4.) Also select “Enable Migration Repetition Option”.
This option enables you to simply repeat the migration benchmark without changing the set of tables. This is especially useful for finding the optimal number or R3load processes for the migration.
5.) Define a high number of R3load processes in your first test iteration to get enough packages from the table spitting to be able to play around with the number of parallel running R3load processes later on. For detailed information of the table splitting mechanism see the blog DMO: background on table split mechanism
Use 10 times the number of CPU cores available on the SUM host (usually the Primary Application Server) as the number of R3load processes here.
The R3loads for “UPTIME” are used for the preparation (determine tables for export), the R3loads for “DOWNTIME” are used for the export (and import, if selected), so UPTIME and DOWNTIME are no indication for uptime or downtime (concerning the configuration of R3load processes).
6.) Directly before starting the roadmap step “Execution”, in which the actual data migration will take place, reduce the R3load processes to 2 times the number of CPU cores available on the SUM host.
You can change the SUM process parameters during the run by means of the DMO utilities:
7.) Start the roadmap step “Execution”.
While monitoring your network traffic and CPU load, raise the number of R3load processes step by step, always waiting 10 to 15 seconds until they are started.
When either the CPU load or the network traffic reach 80% to 90%, you have found the optimal number of R3load processes for this system landscape.
8.) If you repeat the benchmarking run, avoid database caching.
This can either be realized by flushing the cache or by restarting the database.
If you want to change the table set, finish the current benchmarking run and start the test from the beginning. To avoid database caching, you can also select bigger tables that exceed the database cache.
Benchmark Export + Import
Use this option when you want to simulate the export of data from the source system and the import of data into the target system.
After you have executed at least one export benchmark, you can continue with benchmarking the migration export and import in combination. In this way you can find out if your target SAP HANA database is already running at peak performance or if it needs to be optimized for the mass import of migrated data.
The behavior of this combined benchmark is very similar to a real migration run since the exported data is really imported into the target HANA database. Only after a manual confirmation at the end of the migration benchmark the temporarily created database schema is dropped from the target HANA database.
Proceed as follows in the dialog box:
1.) Select the option “Benchmark migration”
2.) Select the option “Benchmark export and import”
Automatically optimize Table Splitting
1.) Perform a benchmark migration of the whole database to generate a durations file, which contains the migration runtimes of the most significant tables.
Set the percentage of the DB size as well as the size of the largest tables to 100% and enable the “Migration Repetition Option”.
On the process configuration screen, input the optimal number of R3load processes, identified beforehand.
2.) Repeat the migration phase to run the full migration benchmark again.
This time the benchmarking tool makes use of the durations file from the first full run to automatically optimize the table splitting, which should result in a shorter overall migration runtime.
Analysis
After a complete migration run, you can analyze the migrated data volume and the migration speed.
The SUM creates a summary at the end of the file ../SUM/abap/log/EUMIGRATERUN*.LOG:
Total import time: 234:30:20, maximum run time: 2:31:41.
Total export time: 222:31:49, maximum run time: 2:31:42.
Average exp/imp/total load: 82.0/87.0/168.9 of 220 processes.
Summary (export+import): time elapsed 2:41:40, total size 786155 MB, 81.05 MB/sec (291.77 GB/hour).
Date & Time: 20150803161808
Upgrade phase “EU_CLONE_RUN” completed successfully (“20150803161808”)
In this example
– 220 R3load processes have been used (110 Export, 110 Import)
– the downtime migration phase took 2 hours 41 minutes
– total migration data volume was: 786155 MB (786 GB)
– migration speed was: 81 MB/s (291 GB/h)
– the migration phase ended without issues: “completed successfully”
In general, a good migration speed is above 300 GB per hour.
R3load Utilization
In the DMO Utilities, analyze the R3load utilization after a migration run.
1.) Open the DMO utilities and navigate to “DMO Migration Post Analysis -> Charts”.
2.) Select the file “MIGRATE_*PROC*”
3.) Check for a long tail at the end of the migration, in which only a small number of R3loads still process remaining tables.
For a definition of this tail and examples for a long and a short tail, see the blog
DMO: background on table split mechanism
If such a long tail is found, analyze the durations file to find out which tables cause it.
Durations file
1.) Open the file SUM/abap/htdoc/MIGRAT*_DUR.XML
(or SUM/abap/doc/analysis/MIGRAT*_DUR.XML if you are using SUM 2.0 SP 02 or newer) with a browser to get a graphical representation of the runtimes of the migrated tables.
2.) Look for long-running tables at the end of the migration phase.
In this example, the table RFBLG has a very long runtime. It is running from the beginning of the migration phase until the end.
R3load logs
Analyze the R3load logs to identify the origin of performance bottlenecks of long-running tables.
1.) Open the R3load log summary file SUM/abap/log/MIGRATE_RUN*.LOG
2.) Search for the problematic tables
3.) Analyze the R3load runtimes to identify the origin of the performance bottlenecks.
You will find R3load statistics of the time spend in total (wall time), in CPU in user mode (usr) and in the kernel system calls (sys).
There are separate statistics available for the database and memory pipe of the exporting R3load (_EXP) and the importing R3load (_IMP).
#!—- MASKING file “MIGRATE_00009_RFBLG_EXP.LOG”
(STAT) DATABASE times: 1162.329/4.248/0.992 93.6%/36.9%/47.6% real/usr/sys.
(STAT) PIPE times: 79.490/7.252/1.092 6.4%/63.1%/52.4% real/usr/sys.
#!—- MASKING file “MIGRATE_00009_RFBLG_IMP.LOG”
(STAT) DATABASE times: 702.479/213.625/4.896 56.6%/96.6%/86.3% real/usr/sys.
(STAT) PIPE times: 539.445/7.620/0.780 43.4%/3.4%/13.7% real/usr/sys.
In this example the exporting R3load spend 1162 seconds on the source DB reading data.
79 seconds were required to copy the data to the memory pipe.
The importing R3load spent 702 seconds on the target SAP HANA DB to write the data and it spend 539 seconds on the memory pipe waiting for data.
Conclusion: In this example the source DB was the bottleneck, because the importing R3load has been waiting for data on the pipe most of the time.
In this case you should ask the administrator of the source DB if he can do a performance analysis of this table.
Extended Analysis
If you still experience low migration speeds, an extended analysis of the following factors during a migration run might help to find bottlenecks:
CPU Usage
As already mentioned in the R3log analysis example, the R3loads usually wait for the database most of the time while the actual processing of the data only takes a small amount of time.
Therefore it should not happen, that the R3load processes use more than 90% of the CPU time on the application server. If this is the case, either reduce the number of R3load processes or equip the server, on which SUM is running (usually the application server), with more CPUs, if feasible.
Memory Usage
Analogous to the CPU usage on the server where SUM is running, enough main memory should be available for the R3load processing.
Otherwise the operating system will apply paging mechanisms that significantly slow down the migration performance.
The minimum memory usage of a single R3load process during the migration of a standard table is about 60 MB.
Especially when declustering is necessary (for target releases 7.40 and higher), the memory required by R3load is very content dependent.
Therefore it makes sense to monitor the actual memory usage during a complete test migration run to determine the optimal memory configuration.
Disk I/O
The performance of export and import operations on the source and target DB is depending on a good disk input/output (I/O) performance. Therefore it might be necessary to postpone activities which create heavy disk I/O (such as backup jobs) during the migration run.
Sometimes it is not obvious, which activities create disk I/O and have a negative impact on the DMO migration performance.
In this case it might be useful to actively monitor the disk I/O during a test migration to pinpoint the timeframe of problematic activities.
Network
The network can also be a bottleneck, therefore it is recommended to monitor the throughput of the different network connections (from PAS to source DB, from PAS to target SAP HANA DB) during a migration run.
Theoretically this should not be a major issue with modern LAN networks. The recommend 10 Gbit LAN would already deliver an expected transfer rate of ~3500 GB / hour. Therefore a low throughput can be an indicator for an unfavorable setup for the migration (e.g data flow through two firewalls).
It also has to be considered, if parallel migrations of different systems or other activities that use network bandwidth, are planned.
Remove the bottlenecks
Depending on the results of your analysis there may be various ways to deal with the bottlenecks found.
If a more powerful machine is required for the R3load processes, it might be an option to run the SUM on a powerful Additional Application Server (AAS) instance with free resources.
In general, SUM and SUM with DMO may be executed not only on the Primary Application Server (PAS), but also on an Additioal Application Server (AAS). However, running SUM with DMO on an AAS is only supported if your system has a separate ASCS instance.
It might be even possible to use an SAP HANA Standby Node for this purpose, especially if the network connection to the SAP HANA database is the bottleneck.
Housekeeping
Especially when performing a SAP BW migration, the positive impact of housekeeping tasks like cleaning up the persistent staging area (PSA), the deletion of aggregation tables and compression of InfoCubes should not be underestimated.
For details regarding the SAP BW migration using DMO see the document:
SAP First Guidance – Using the new DMO to Migrate BW on HANA
But even with a standard DMO you should give some thought to housekeeping before starting the migration. For example, it might be an option for you to delete or archive old data that is not accessed frequently anymore (analogous to moving BW data to Near-Line Storage) before starting the DMO migration. This data does not need to be transferred, which reduces the migration runtime, and it does not need to be stored in-memory on the target HANA database.
Table Comparison
After you have optimized the DMO migration using the benchmarking tool, you are ready for the first test migration run.
You have now the option to let SUM compare the contents of tables on the target database with their respective content on the source database to make sure that everything has been migrated successfully.
We recommend to switch on the table comparison for all tables in the first test run only.
The reason is that the full table comparison via checksums takes a lot of time, usually as long as the table export itself.
If no errors are found, keep the table comparison off (“Do not compare table contents”) or compare only single, business critical tables in the productive DMO migration run.
This will minimize the Downtime in the productive run.
In fact, even when “Do not compare table contents” is selected, the SUM still compares the number of rows of the migrated tables on the target database with the number of rows on the source database after the migration of their content.
For further information regarding the DMO table comparison see DMO: table comparison and migration tools
Downtime Optimization
If the performance of the standard DMO is still not sufficient after all optimization potential has been utilized (usually a migration speed of up to ~500 GB/h can be reached) and the downtime needs to be significantly shorter, additional options to minimize the downtime are available.
Downtime optimized DMO
The Downtime optimized DMO further reduces the downtime by enabling the migration of selected application tables during the DMO uptime.
The report RSDMODBSIZE (available with SAP Note 2153242) determines the size of the largest tables in a SAP system and gives an estimation about the transfer time required for these tables in the DMO downtime.
Tables transferred with Downtime optimized DMO in the DMO uptime effectively reduce the downtime.
The report facilitates the decision if the usage of Downtime optimized DMO is suitable and generates a list of tables as input for SLT.
The following blog post describes this technology, prerequisites and how to register for pilot usage of the Downtime optimized DMO:
DMO: downtime optimization by migrating app tables during uptime (preview)
Note that the Downtime optimized DMO works for SAP Business Suite systems, but not for SAP BW.
BW Post Copy Automation including Delta Queue Cloning
To minimize the migration downtime of a productive SAP BW system, one of the recommended migration paths from SAP BW to SAP BW on SAP HANA comprises a system copy of your SAP BW system.
To keep things simple, SAP offers the Post-Copy Automation framework (PCA) as part of the SAP Landscape Virtualization Management which includes post-copy automation templates for SAP BW as well as an automated solution for delta queue cloning and synchronization, enabling the parallel operation of your existing production system.
In combination with the SUM DMO the production downtime of the migration from SAP BW to SAP BW on SAP HANA can be kept at a minimum. The usage of the delta queue cloning solution requires additional steps to be performed before the standard SUM DOM is started.
For further information about the downtime-minimized migration process of SAP BW using Post-Copy Automation with delta queue cloning see the following links:
TechEd 2014 – Migration to BW on HANA – Best Practice
SAP First Guidance – Using the DMO Option to Migrate BW on HANA
SAP First Guidance – BW Housekeeping and BW-PCA
Hi Stefan Mueller
Thanks for the detailed explanation, as compared to teched itm360.
However, instead of using classical migration, do you come across that DMO will be supporting anydb to HANA migration on HANA ready product version without the need of updating/upgrading sap?
Thanks,
Nicholas Chang
Hi Nicholas Chang,
thank you for your feedback.
DMO is an option of SUM and not a standalone tool. Therefore it can only be used for migration scenarios in combination with an SAP update/upgrade.
For a classical migration, you still have to use the Software Provisioning Manager.
Thanks,
Stefan
Hi Stefan,
However, we can still use DMO for mainly HDB migration purpose by just generate a stack.xml to update a single level for any software component, for eg, ST-PI?
Thanks,
Nicholas Chang
Hi Nicholas,
sure, as long as you have a valid stack.xml with at least one updated software component, you can use the SUM DMO. ST-PI is a good candidate since it is not too big and it is updated regularly.
Thanks,
Stefan
Thanks Stefan!
Hello
I get a Error in : MIGTOOL_PREPROCESSING/SUBMOD_MIG_SEL/SQLSELMIGTABLES_BENCH_TRIG
'No cached size for 'DDLOG''
Any Idea?
Best Regards
André
Hello André Hunziker,
to help you with this specific issue, could you please create a customer incident on component BC-UPG-TLS-TLA?
Thanks,
Stefan
Stefan,
Thanks for this well-written and clear detailed description. Is this same benchmarking technique available for system copies with SWPM, or is this only available for DMO operations? The basic strategy for the benchmarking seems like it would be applicable to any R3load-based export/import, and not just DMO.
Cheers,
Matt
Hi Matt,
The benchmarking tool is only intended for the DMO procedure - although some of the information provided might be relevant for all procedures, not everything can be transferred to the classical migration 1:1 and also not everything will be offered there (realtime monitoring etc.). Also, while DMO offers an integrated automatic optimization (such as on the table durations file), this would have to be figured out manually for the classical procedure, as described in the Best Practice Guide - Classical Migration of SAP NetWeaver AS ABAP to SAP HANA.
At least, this would be my current understanding, but maybe you can give it a try in your next migration run and share you experience here (what has been useful, what has been misleading...)?
Cheers,
Boris
Hello Stefan,
We are doing POC or ERP on HANA Migration, downtime is really a challenge.
We finished one run and I see write times to PIPE are too high over DB time for Export and Import to HANA time is also high. Do we see this as Network bottleneck?
MASKING file "MIGRATE_DT_00635_VBOX_EXP.LOG
(STAT) DATABASE times: 1040.938/215.345/5.467 19.8%/86.5%/53.1% real/usr/sys.
(STAT) PIPE times: 4208.011/33.717/4.823 80.2%/13.5%/46.9% real/usr/sys.
MASKING file "MIGRATE_DT_00635_VBOX_IMP.LOG
(STAT) DATABASE times: 4660.919/241.374/3.364 88.9%/93.0%/40.9% real/usr/sys.
(STAT) PIPE times: 584.524/18.280/4.859 11.1%/7.0%/59.1% real/usr/sys.
Thanks,
Bhushan
Bhushan, during your migration what is your % CPU and %RAM like on the OS level on both side?
--Steve--
CPU utilization on PAS where the DMO was running - Avg 60% Utilize
CPU Utilization on HANA Server was high - Avg 95%
CPU on Source DB was Avg 87% utilize
Memory on Source DB was utilize 98%
Thanks,
Bhushan
Is your DB running on the same box as your PAS? Usually when you do the import, it is using R3 load processes to load the Application data into the SAP HANA DB. and that is during downtime. The creation of the structure was during uptime. what DB are you running on Source?
--Steve--
DB running on separate Host it is DB2 database. On the DB Host the memory was completely utilized. However, the DB times from Export in compare to PIPE times are less.
MASKING file "MIGRATE_DT_00635_VBOX_EXP.LOG
(STAT) DATABASE times: 1040.938/215.345/5.467 19.8%/86.5%/53.1% real/usr/sys.
(STAT) PIPE times: 4208.011/33.717/4.823 80.2%/13.5%/46.9% real/usr/sys.
MASKING file "MIGRATE_DT_00635_VBOX_IMP.LOG
(STAT) DATABASE times: 4660.919/241.374/3.364 88.9%/93.0%/40.9% real/usr/sys.
(STAT) PIPE times: 584.524/18.280/4.859 11.1%/7.0%/59.1% real/usr/sys.
Thanks,
Bhushan
Bhusan, based on what you provided it seems like your problem is with the DT phase where the R3load is loading application data to the SAP HANA DB. What version of SUM-DMO are you running? Try increasing the number of R3Load processes on the fly and see what is happening and whether it makes any difference? You can also see in your UPGANA file to see where and what phase is causing the long load time.
--Steve--
Hi,
good document from my point of view, introducing useful hints as for example those related with checksum. By the way do you think the possibility to compare the checksums of exported and imported tables is useful? Is there any possibility to get a difference?
I Think we'll get differences in case an error occurs but in that case we'll get an error, why to add the checksum validation?
best regards
Hi,
the comparison of the checksums of the exported and imported tables is useful to validate the correct and successful migration of these tables.
Technically a difference should never occur after a successful run.
So this is primarily to validate the procedure itself on a customer system before starting the productive migration.
Best Regards,
Stefan
Thanks Stevan
My conclusion is that checking the number of exported vs. the imported entries is useless and it only consumes time.
Best regards
😎
Hi Antonio,
please don't underestimate the duration of the table comparison. Depending on the size of the tables in scope it can take several hours or even days. Therefore it's very unlikely in my opinion to take advantage of it during a production cutover.
On the other hand, the row counts should be rather quick and would be sufficient to give you a good indication on the success of your DMO run.
Therefore my conclusion regarding the use of checking the numbers is exactly the opposite to yours. 🙂
Regards,
Ronald
Thanks Ronald,
What could we expect to get wrong causing a diferent number of imported entries compared with the exported? I'm searching for causes or cases for that and I haven't found it. 😕
Regards
Antonio
Hi Antonio,
first of all, it's not about comparing exported with imported entries (i.e. retrieving figures from the corresponding log files) but performing row counts on tables in both the source and the target database.
The row count statistics (combined with the migration runtimes) give you a good indication on the table sizes as well as the throughput. Thus they make systems comparable to a certain degree. In addition, they are sometimes requested by the auditors as proof for a successful migration as they provide a good level of confidence within a reasonable time needed to collect them.
From my own project experience I can tell you that I was happy to have these figures in place when I had to explain the differences in a source and a target system during a production cutover. What happened? The consultant who was responsible for the productive migration had reused the WHR files that were created during a test run in order to save the time for table splitting on the go-live weekend. Unfortunately, the old WHERE clauses missed several key areas from the production system and thus only parts of the related tables were exported.
The only thing I don't know is the handling of pool and cluster tables in this context as they become transparent tables with the HANA migration. Unfortunately, section "4.5.1 Table Comparison" of the SUM DMO Guide doesn't go into details here. Maybe somebody can answer this question?
Kind regards, Ronald
Hi Stefan,
thanks for your good documentation. I have some questions and hope you can put some comments.
1. you mentioned in benchmarking tool, we can first define more r3load processes (10 * cores) to get enough packages from table splitting to be able to play around with number of parallel running r3load processes later on. and right before entering downtime we can reduce r3load process to 2 * cores.
My question is: during real DMO running, we how can achieve this? should we define a high number of "R3load processes (downtime)" and reduce it before entering downtime? And i am a little confused by your saying: "the r3load for "uptime are used for the preparation determin tables for export, the r3load for downtime are used for the export and import. so uptime and downtime are no indication for uptime or downtime"". I suppose this is only the case for benchmarking tool but not for real DMO run, right? during DMO run, R3load processes (uptime) indicates the process during MIGRATE_UT_RUN phase, while R3load processes (downtime) indicates the process during MIGRATE_DT_RUN phase.
2. i am not quite clear about what PIPE times stands for. for the export side, PIPE times are used for reading data from source db to pipe. So pipe times include mostly the time spent on data transfer from db server to PAS?
BR,
Johnny
Hi Johnny,
after you have found the optimal number of R3load processes with the benchmarking tool, you should stick to this number for your productive run. In a productive run we actually recommend not to change the number of R3load processes anymore.
Your understanding of the R3load processes (uptime) and R3load processes (downtime) is correct.
Uptime refers to the MIGRATE_UT_RUN phase and downtime refers to the MIGRATE_DT_RUN phase.
The pipe time is the time required to copy the data to the memory pipe (export side) and read the data from the memory pipe, including waiting times for data (import side).
The time spent on data transfer from DB server to PAS is part of the database time.
Best Regards,
Stefan
Hi Stefan,
thanks for your further clarification. Actually my first question is more about table splitting. Because i was told the number of configured R3load processes would have impact on table splitting. If we define more R3loads, SUM will consider this and maybe split more tables? and before downtime, we can dynamically decrease R3load. So i want to confirm this with you, is this true? 🙂
BR,
Johnny
Johnny, in the latest version of SUM-DMO the table splitting is dne automatically now based on how may R3loads you configured. Yes, giving more R3loads processes will speed up the copying of data over to the target. I hope this elps.
--Steve--
Hi Steve,
Great!! thanks for your update.
BR,
Johnny
Hallo Stefan,
Do Linux OS tuning would help DMO optimization?Linux tuning like tcp buffer,swap,io scheduler,storage block device,swappiness. What about changing from default page (4kb) HUGE PAGE ? What do you suggest regarding Ethernet card ?
regards,biral
Hello Biral,
actually all optimizations that support a higher throughput of mass data either on the export side or on the import side will help to optimize the DMO runtimes.
A 10 Gbit Ethernet card is always recommended. Please also make sure that the network speed is actually available (sometimes a single component in a network can reduce the whole network's performance). This can be checked with a simple file transfer between the source DB server and the PAS and between the PAS and the taget DB server or by using special test tools like iperf.
The expected transfer rate of a 10 Gbit LAN would be around 1000 MB/s.
We would be glad to hear from you, if you could already identify specific Linux configuration parameters that have a positive impact.
Regards,
Stefan
Hallo Stefan,
thank you.
Could you please share details in which PHASES R3load starts import and export of data?
Also in which PHASES will SQL, R3trans process will be used?
Any idea about which phases uses which process (ie.ABAP,SQL,R3load) during each phases of DMO run?
Regards
biral
Hello Biral,
the relevant phases for R3load import and export of data are EU_CLONE_MIG_UT_RUN (in DMO uptime) and EU_CLONE_MIG_DT_RUN (in DMO downtime).
In the benchmarking mode the relevant phase is: EU_CLONE_RUN.
The SQL processes are relevant for the row count, which is also part of the phase EU_CLONE_RUN.
The R3trans processes are not directly related to the DMO phases. They are especially relevant for the standard upgrade phases SHADOW_IMPORT* in the uptime and TABIM_UPG in the downtime.
Regards,
Stefan
Hi Stefan,
thanks for the good explanation. We made POC with ERP DMO. In the downtime phase all R3loads stops with
xxxadm 22575 30217 0 May19 ? 00:07:11 [R3load] <defunct>
xxxadm 22578 30217 0 May19 ? 00:04:55 [R3load] <defunct>
in this situation both databases was accessible also the CPU and memory was not over 90%. After stop and restart the SAPup the migration runs again. have you an idea why the system runs in this situation. before i start the SAPup I decrease the number of R3loads from 100 to 50 maybe that's the reason that it runs. But how can i avoid so an situation.
regards
Robert
Hi Robert,
the R3load processes in <defunc> state can be seen during regular operation due to the asynchronous handling of the R3load processes by the SAPup.
But, of course, it should not happen that all R3load processes are in <defunc> state for a longer period of time.
Since this is an unexpected behavior, I would recommend that you create create a customer incident on component BC-UPG-TLS-TLA and attach the following files, so that we can analyze this issue:
MIGRATE_DT_RUN.LOG or .SAV
EUMIGRATEDTRUN.LOG or .SAV
Also, please mention the exact SUM version and patch level in the incident.
Thanks,
Stefan
Hi Stefan,
thanks for your advise i opened an incident....
Another question I would open /SUM/abap/htdoc/MIGRATE_DT_DUR.XML with a browser to get a graphical representation of the runtimes, but i get only a blank screen i try it with firefox 38.7.0 also with IE11.0.9 and SAFARI9.1 and Chrome 49.0. Do you need a special addon ?
Hi Robert,
yes, unfortunately we had an issue with the combination of the xsl stylesheet and the durations xml file. This is meanwhile fixed. So if you download the latest SUM SP 16 patch level (currently 11) and put its UpgAnalysis.xsl next to the durations file, the display should work.
Regards, Boris
Hi Boris,
thanks for the advise i have SUM SP16 PL8 and you mean there is still there?
For testing i downloaded the SUM SP16 PL11 extracted and move the MIGRATE_DT_DUR.XML in the htdoc diretory. but i got the same result only a empty blue screen.
Have you any idea ? Or Working it not so ?
regards
Robert
SAP still fixing it 🙂
Hi Amt,
is that the official statement or your opinion ?
regards
Robert
Hi Robert,
my apologies: we thought that PL11 already contains the adapted stylesheet file, but this is not the case. I expect it to be part of PL12, and it will definitely be part of SUM SP 17 (expected to be available in June, but disclaimer applies).
Regards, Boris
Hi Robert,
SUM 1.0 SP 16 PL 12 is out as of today, and this includes an UpgAnalysis.xsl that will display the durations file in a browser correctly. This time I checked it on my own...
Regards, Boris
Hi Boris,
thanks a lot for the update.
I tried it out with some MIG_*_DUR.XML files created with SUM SP16 PL6. They can be displayed even with the corresponding UpgAnalysis.xsl but with the following limitations:
1. There is no title bar / labeling available that indicate the start and the end time of the run. Has this been omitted deliberately? See the output of the Migtime analysis tool for reference. This is valid for all UpgAnalysis.xsl versions, including the one from SP16 PL12.
2. The number of exported and imported records per table is indicated as "NaN" since my XML files contain only rowcount information. Has the syntax of the XML file been changed in the meantime? There were no similar issues with the UpgAnalysis.xsl from SP16 PL6.
Kind regards,
Ronald
Hi Stefan Mueller
Thank you for the article.
I have questions regarding this statement:
1) Where it is possible to find additional information/tech. documentation regarding running DMO / SUM on AAS? What section of DMO/SUM guide is it - was not able to find except this statement (in section 5 of SUM guide):
2) Can we execute DMO in parallel on PAS and AAS to increase throughput or we should choose 1 specific instance (the fastest one) to run DMO/SUM?
3) DMO on AAS is only supported if ASCS is separate - should we do ASCS split using SUM first prior running DMO on AAS? Will it help to follow the requirements?
Thank you,
Best regards,
Igor.
Hi Igor,
here are my comments regarding your questions:
1) Where it is possible to find additional information/tech. documentation regarding running DMO / SUM on AAS? What section of DMO/SUM guide is it - was not able to find except this statement (in section 5 of SUM guide).
I am not aware of any additional information regarding this topic. Maybe my colleagues can answer this question.
2) Can we execute DMO in parallel on PAS and AAS to increase throughput or we should choose 1 specific instance (the fastest one) to run DMO/SUM?
A parallel execution of DMO on PAS and AAS is not supported.
You need to choose 1 specific instance to run DMO/SUM.
3) DMO on AAS is only supported if ASCS is separate - should we do ASCS split using SUM first prior running DMO on AAS? Will it help to follow the requirements?
Yes, you need to split the ASCS beforehand (DMO guide: 2.6 ASCS Instance Split).
Best regards,
Stefan
Hi Stefan Mueller，
Thanks for your wonderful sharing！
I have a problem when doing service about analysis logs of R3load.
customer did DMO from db2 to hana,they set the numnber of r3load is uptime 800, downtime 999.
the following is a snap of MIGRATE_RUN*.LOG.
(RDI) INFO: /usr/sap/PE3/SUM/abap/migrate_dt/MIGRATE_DT_00694_CDCLS_EXP.STR has format version 2
/usr/sap/PE3/SUM/abap/exe/R3load: job completed
(STAT) DATABASE times: 324.809/8.432/2.595 0.4%/36.1%/59.9% real/usr/sys.
(STAT) PIPE times: 75588.921/14.952/1.735 99.6%/63.9%/40.1% real/usr/sys.
/usr/sap/PE3/SUM/abap/exe/R3load: END OF LOG: 20160812125918
(RDI) INFO: /usr/sap/PE3/SUM/abap/migrate_dt/MIGRATE_DT_00694_CDCLS_IMP.STR has format version 2
/usr/sap/PE3/SUM/abap/exe_2nd/R3load: job completed
(STAT) DATABASE times: 75861.036/4059.580/45.749 99.9%/99.7%/96.2% real/usr/sys.
(STAT) PIPE times: 41.076/13.534/1.786 0.1%/0.3%/3.8% real/usr/sys.
/usr/sap/PE3/SUM/abap/exe_2nd/R3load: END OF LOG: 20160812125910
I don't know why the PIPE time in EXP and IMP were so different.
and what caused so much time spending in PIPE (exp) and can I conclude that there is something wrong in Hana db ?
Looking forward to your reply.
Thanks,
Quintina
Quintina, I do not believe there is anything wrong with the HANA DB. I believe if there is any problem I believe it most probably on your Source side DB. Did you also check on the OS side resource usage? If you are running DB2 on Wins side, you can also change the resources on the fly and monitor your CPU and RAM to make sure you are running at Optimum on your target side.
Steve
Hello Stefan,
Do you happen to know where the total size comes from in EUMIGRATEDTRUN.LOG ?
My customer had a problem of migrating biggest tables ( one is 200GB, another is 80GB ), which contributed most of run-time.
Hence, the 2nd DMO-run was done by removing the data on the concerned tables, which definitely reduced the downtime significantly.
However, what interesting is that EUMIGRATEDTRUN.LOG shows the amount of data exported/imported not so different even though the data is truncated.
(but not sure how the data was removed ,should be either truncate or delete..)
====================================================================
<Before truncating the two tables >
ETQ399 Summary (export+import):
time elapsed 17:08:02, total size 660927 MB, 10.72 MB/sec (38.57 GB/hour).
<After truncating the two tables>
ETQ399 Summary (export+import):
time elapsed 3:39:42, total size 658746 MB, 49.97 MB/sec (179.90 GB/hour).
====================================================================
So again, it would be much appreciated if we know how the total size is calculated?
Best Regards,
Jonghoon
Hi Jonghoon,
Did db reorg executed after data truncation? you need to run a db reorg to defrag those empty pages.
Cheers,
Nicholas Chang
Thanks, Nicholas,
so you mean the figure(total size) comes from the database size rather than table size?
basically truncate cleans up the used pages but not like deletion.
And it seems DMO sorts tables by its size, and then makes buckets based on the table(maybe package) size.
If the size is based on the database size,I wonder how the throughput (like 40MB/sec etc) is calculated..
Regards,
Jonghoon
What's your source database? do you have a chance to look at your source db size after truncation?
Kim, why don't you archived off the old data before migration?
--Steve--
Hello All, thanks for your input.
All matters regarding data reduction are also in the customer's interest (in progress).
However, these are a little bit aside from the topic that my customer is seeking for the source of the figure in the log, later whether it can be utilized for a rough throughput measure or not.
Again, the question is what is the source of the total size indicated in EUMIGRATEDRUN.LOG..
For details, refer to the question on Sep 1.
Maybe the next time, the customer will have a change to compare it with DB size ( SQLServer, & table compression on )
Kim, did anyone boost up the CPU and RAM on your Source? If you assign more processes it will affect the run time even though you almost have the same amount of data.
--Steve--
Hi Stefan,
We are running DMO for ECC upgrade and hana migration, currently we are in phase "EU_CLONE_MIGRATE_DT_RUN" which is application Table Data migration is happening from ORACLE to HANA.
We have observed that HANA CPU's are always 100% occupied even if not many buckets are in progress. At (PAS+DB) host CPU utilization is not much.
We have 60 CPU cores at PAS side and 144 CPU cores at HANA side. What could be the reason that CPU utilization on HANA server is always 100%?
Thanks,
Ambarish
Hello Stephen,
We tried using UPGANA.xml file from previous DMO runs for ECC EHP6 upgrade and migration to EHP7 on HANA. SUM we used was SUM 1.0 SP17 PL8.
When we provided UPGANA.xml in download directory and entered path in sapup_add.par file in /usr/sap/<SID>/SUM/abap/bin/ however it was not accepted by SUM tool.
We received below error in phase – MAIN_SHDIMP/SUBMOD_MIG_PREPARE/EU_CLONE_MIG_UT_PRP as follows –
Illegal top level tag analysis in ‘/usr/sap/<SID>/Download_DIR/UPGANA.xml’ – expected ‘Clone durations’ .
Can you please help on this?
Thanks,
Ambarish
Hello,
Does anyone know if I need to complete SPUMG before I can use the bechmarking tool? I have completed all the other unicode pre-conversion checks.
Thanks
Hello Kevin,
Unfortunately, it is not possible to use the benchmarking tool on non-Unicode systems.
Regards,
Stefan
Hi Stefan,
Can you clarify your comment with further detail? I was able to use the benchmarking tool on SUM 1.0 SP20 PL2 with a HANA target. SPUMG was completed to the point of having already created the nametabs prior to starting the benchmarking tool execution.
Hello Mark,
The benchmarking tool does not contain the Unicode conversion functionality.
Therefore it does not make sense to use it on non-Unicode start systems since you would get wrong results (potentially too low numbers) anyhow.
Regards,
Stefan
Hi Stefan,
I have a question regarding Virtualization of CPU and your statement:
"Use 10 times the number of CPU cores available on the SUM host (usually the Primary Application Server) as the number of R3load processes here."
should we still use CPU core if we are using Power8 with SMT 8 so each core has 8 threads.
Regards,
Adel
Hallo Stephen,
We recently did migration of sap ewm system from Windows/SQL to windows/Hana using sum dmo 1.0 sp20.
There were no errors during downtime, and it took 8 hours to finish downtime phase.
When we compare time mentioned in upgana.xml it doesn't match with actual time. Upgana shows only 2 hours of downtime phase.
Eumigratedtrun.log does show correct time values.
Thanks,
Ambarish
Hallo Zusammen
wir haben auch ein Laufzeit Problem bei DMO - Migration Oracle - SAP HANA
Laufzeit 29Std
Source DB ist Oracle 12.1 3 TB Target SAP Hana Tenant 2.00.32
CPU max 30% 10%
Netzperformance getestet und ok
Anzahl der R3loads: 700
Während der Migration schaut es so aus, das die Export -Pipe Time riesig ist, und die Import DB Time auch. Die Target Hana DB langweilt sich die CPU. maximal 10% im Durchschnitt 2-3%
Beispiel:
EXP: Migrate-DT_00160_edi40-exp.log
A1 EDMP 008 Resource usage: DATABASE times: 105.205/3.198/3.107 0.0%/31.0%/90.4% eal/usr/sys
A1 EDMP 008 Resource usage: PIPE times: 18446744073683868.000/7.114/0.331 100.0%/69.0%/9.6% real/usr/sys
A1 EDMP 007 Memory usage: 33/26 MB buffer/disk (0 reused buffers)
IMP. Migrate-DT_00160_edi40-imp.log
A1 EDMP 008 Resource usage: DATABASE times: 18446744073683976.000/524.765/9.468 100.0%/98.6%/93.7% real/usr/sys
A1 EDMP 008 Resource usage: PIPE times: 4.999/7.220/0.641 0.0%/1.4%/6.3% real/usr/sys
--> bedeutet das die Target DB ist das Bottleneck?
Vielen Dank und Gruss
thomas Klein