DMO: comparing pipe and file mode for R3load
- Both the heterogenous system copy of Software Provisioning Manager (SWPM) and DMO of SUM support the migration to SAP HANA Database
- Both SWPM and SUM trigger R3load for the export from the source DB and the import into the SAP HANA DB
- But DMO has both R3loads running on the same host: on the primary application server (PAS, fka central instance CI)
(see DMO: technical background)
The “classical migration” (heterogenous system copy of SWPM) uses either the file mode or the socket mode.
Using the file mode, the data are exported, and written into a file, using compression. The file is transferred over the network:
Using the socket mode, the data are exported and transferred over the network:
Now with DMO, both R3load are running on the same host, and they communicate using the main memory of that host, using the pipe mode per default:
❗ Warning (Unix only)!
Do not use the statement grep* on the migrate_* directories to check for error messages. This generic grep statement may corrupt the named pipes used in R3load communication and cause errors in R3load processing. (Using “grep*.LOG” in these directories is not dangerous.)
Note
Even with DMO, it would be possible to switch to the file mode. The R3load exports the data into a temporary file of limited size. As soon as this file is processed and the data are imported into the target DB, the temporary file is deleted.
- Compression is used for creating the file
- For windows OS, the pipe mode is enabled starting with SUM SP13 (available since April 28th 2015)
- I have no scenario when you would want to switch from pipe to file mode.
Boris Rubarth
Product Manager, Software Logistics, SAP SE
[Update on June 1st 2015] File mode for DMO does use compression (this was initially wrongly stated in the blog).
[Update on March 15th 2016] Removed graphic for DMO file mode and shortened text; added warning on grep statement.
Thanks Boris,
Good info to explore. Couple of questions:
1. What is the default method opted by DMO
2. How to switch the modes from socket to file and vice-versa
3. While working with Socket mode do we need more memory (RAM) on the source application server to ensure optimal file transfer.
Please advise and thanks for your time.
Regards,
Srinivas K.
Hi Srinivas,
1) Default mode in DMO is pipe mode. For windows-OS, DMO chooses file mode per default
.
2) For DMO, socket mode is not possible (and not required as both R3loads are running on the same host). To use file mode instead of pipe mode for DMO, parameter clone/method has to be used as described above.
3) Memory requirements for socket mode (only available in SWPM) are not different to file mode.
Regards, Boris
Thanks Boris. 3rd point will help in better preparation of the Source instance.
Hi Boris,
Good one.
Thanks a lot.
Eddie
hello Boris,
with the DMO pipe mode, it uses the main memory right and not the shared memory, is that true and if so , is there any tweaking for memory parameters to be done before the procedure.
Thank you
Jonu Joy
Hello Jonu Joy,
not sure what you refer to with main memory .. anyhow it is none of both, as memory pipes are used, so it is based on the pipe concept. Instead of R3load writing to a file, the R3load process is writing data to a pipe (named pipe).
Regards, Boris
Thank you Boris, the consultant whom we are working with was asking to check the shared memory parameters (IPCS) for troubleshooting issue with named PIPES, which i thought was incorrect as named PIPES does not use shared memory, i am of the assumption that since the pipe file itself is of 0 bytes the contents which are written to the pipe would be stored on memory(RAM) until it is being read by the other end right.
Is there any tweaking that can be done for this named pipe operation to be perform better .
You are right in assuming that the memory pipes are NOT part of the Inter Process Communication (IPC) shared memory, so IPCS does not apply here.
There is no tweaking possible for the memory pipe technique. We have no indication from project experience that this is required.
Regards, Boris
Thx very much Boris
Hi Boris,
sorry, I don't know the pipe technology. You write "the R3load process is writing data to a pipe (named pipe)": this pipe is a RAM portion?
Thank you
Regards,
Walter
Hi Walter,
it is a standard unix technology, described e.g. in
http://en.wikipedia.org/wiki/Pipeline_(software)
Regards, Boris
thanks Boris!
Regards,
Walter
Hello Boris,
Very good content indeed!
i have a couple of questions:
1.) Since TABART is not considered for creation of R3load control files, what are the relevant files created ? If not TABART, what is considered for the creation of control files?
2.) In SWPM SAP<TABART<,STR SAP<TABART.CMD etc are created, How is the naming conventions of the files in DMO?
Regards,
Austin
Hello Austin,
thank you for the feedback. Concerning your questions.
Anyway you are not expected to examine the files, and definitely must not edit them, as the SUM will take care for everything, even in error cases.
Regards, Boris
Hi Austin,
You may refer to Phases behind DMO R3load parallel export/import during UPTIME and DOWNTIME to target HANA DB for little info on .cmd, .tsk and etc.
hope it helps.
Hi Boris,
could you please help me to understand if it's possible to use DMO in a "remote" scenario? I mean a scenario with source system PAS and source system DB in a specific LAN and the HANA DB server in another network.
I know that DMO is an in-place migration: both R3load process run on source system PAS and it doesn't create a permanent export (like normal SAP migration).
The first option could be: R3load that writes in HANA DB write data.. in another network, where HANA DB resides! I think this is not a useful solution!
Is there a way to leverage the advantages of DMO in this type of scenario?
Thank you
Regards,
Mark
Hi Mark,
indeed the WAN connectivity will be the crucial point. I do not know of any project that conducted this "remote" DMO migration successfully, this should rather be a case for the classical migration.
Regards, Boris
Hi Boris,
This might be a good scenario for using the file mode operation of DMO of SUM, because it may improve the data migration time (compression will mean much less network bandwidth). Where can I find any information on choosing to use and configuring file model for the DMO execution? (referring also to your blog post DMO: comparing pipe and file mode for R3load) - is this somethig that is simply selectable when starting? Presumably two app server hosts are needed for this.
Secondly, when executing the post-migration phases of a DMO of SUM, can this be run on an app server local to the HANA server rather than on the original PAS? Latency effects of running the post-migration steps from the original PAS (which is local to the source DB but far from the HANA DB) can be very bad.
I understand that the performance issue is the reason why using DMO for data entre migration is not supported (as per note 2257362) but it would still be useful to understand how the process is working so that we can optimise performance as much as possible.
Classical migration means a separate upgrade project is needed for any customers on BW <7.4 which is not necessarily feasible.
Thanks!
Hi,
using the file mode (instead of pipe mode) for data transfer between R3loads does not make sense because even for a data center move, both R3loads are running on the same host (on which the SUM was started) - it is not the data transfer between R3loads that is affected by network limitations.
Secondly, it is not possible to switch the host on which the SUM is executed during the procedure.
Regards, Boris
Hello Boris,
Is is possible to run DMO like DISTMON.
We run 10+ application servers and would like to utilize them all and distributing R2load and R3trans processes.
Kind Regards,
Muniraju
Hello Muniraju,
the DMO procedure of SUM will only allow R3load processes to run on the primary application server host on which the SUM is located.
Still for linux/unix platforms, memory pipes are used as mentioned in this blog, and these have a much smaller footprint on the host as no compression / decompression of files is required.
Kind regards, Boris
Hello Boris, During Upgrade of ECC5 to ECC6 Ehp7 with SUM DMO and CU/UC, I have the error below during the export/import of the tables systems on shadow instance to DB HANA only for for DDNTT & DDNTT_CONV_UC My CI is on AIX 6.1: export : ERROR write_to_pipe: Cannot write data to pipe off=0 len=1048576: Broken pipe ERROR write_to_pipe: Cannot write data to pipe off=0 len=20: Broken pipe import: ERROR read_from_pipe: Internal error: Invalid block magic=0x4c444454 nr=1 len=1048576 - expected 0x4c444649 After change the kernel of SUM directory with last R3load and DB Library, My Installaion Number is bad on Export file "INITIAL" and I have the error message with Pipe: ERROR write_to_pipe Cannot write data to pipe off=0 len=0 : broken pipe And in import log : ERROR: invalid migration key Could you please help to resolve this issue? Regard, Fabien
Hello Fabien,
Were you able to resolve the broken pipe issue? If so please let us know what you did to fix this?
ERROR write_to_pipe: Cannot write data to pipe off=65536 len=1046334: Broken pipe
Regards,
Avi
Hello,
an invalid migration key will be a problem for the import R3load, which will quit, and then the export R3load will raise an error "broken pipe".
Regards, Boris
Hello Boris,
Thanks for the quick response. So what's the possible solution ? Do we need to reset the SUM and restart it all together?
I have compared the migration key generated and the migration key that I have provided during the DMO procedure(screenshot)..they are same.
Regards,
Avinash
Hi Avinash,
refer to note 2143443 to re-enter correct migration key or else, check the solution below stated in latest DMO guide:
Hope it helps adn let us know how it goes.
Thanks,
Nicholas Chang
Hello Chang,
The issue is with the file system type created as Ext 4(VHANA on RHEL 6.5) provided by our hardware vendor which is not yet supported by SAP for HANA due to which the HANA DB was hanging.
SAP has advised us to redo everything with the supported file system type created for HANA DB.
The new run went fine with XFS files system type
Regards,
Avinash
Hi Boris Rubarth
Appreciate if you can clear my doubt on below:
What's the network usage on DMO downtime migration? For example, if 140 R3load configured, with 70 R3load running the import data to HDB, will this cause network bottleneck if it exceeded the network bandwidth?
Hope to hear from you soon,
Thanks,
Nicholas Chang
I have run DMO with 480 parallel R3load in past over LAN and achieved good results.
Did not notice network bottleneck.
However the document says more than 200 R3load processes don't give significant benefit.
I think you should be able to monitor network usage and increase R3loads on regular basis with the SUM url http://<hostname>.<domain>.<ext>:1128/lmsl/sumabap/<SID>/set/procpar
Regards,
Muniraju
Hi Muniraju & Boris Rubarth,
Basically, i just want to know how exactly the exported data in pipe get transfer, load and import to HDB. Is chunk of data in PIPE get imported via R3load thru network?
Thanks,
Nicholas Chang
Hi Nicholas,
Firstly R3load for the import is started with socket option, where it starts to listen on a port for data stream.
Then R3load export starts to pump the data into this listening sockets.
In case of DMO, both import and export runs on same machine, therefore the communication between R3load import and R3load export should happen locally.
Whereas R3load import should load the data into target HANA database over network using database connections.
Let me know if this answers your question.
Kind Regards,
Muniraju
Hi Nicholas,
A pair of R3loads (one for export and one for import) is started locally on the same server. The R3load for import, triggers the fast load and that's nothing but the inserts at HDB level.
Memory pipe are FIFO pipes. R3load import process will listen on this pipe and waits for data stream to come in. It will end when R3load export process sending the stream over this pipe is stopped (EOF reached).
Option 1 or 2 for me is no different.
R3load via its database drivers (kernel) and HDB client will load the data into the database. The end statement will be INSERT, UPDATE, etc.
And why do you think there is different network consumption with these scenarios.
Eventually all data has to transferred over network to target database. So end result will be the same network usage.
Let me know if you are still not clear.
Kind Regards,
Muniraju
Hi Muniraju,
For option 1, means "physical" data will transfer to target HDB via network.
eg: if 200 R3load configure, 100 R3load will doing the import. and there would be
100 "MIGRATE_DT_XXXXX_XXXXX.PIPE" data to be transferred.
Therefore, the more R3load configure, the more data to be transferred, the higher the netowrk consumption.
f that's the case, is it very likely if higher R3load configured, network bottleneck will happens if the bandwidth couldn't accomodate the concurrent data transferred?
If option 2, basically no "physical" data is transferred. Whereas, only the command, syntax (such as "inset into" & "update") are send over to target HDB for its to load and import data. In this case, no much consumption on network.
Thanks,
Nicholas Chang
Hello Nicholas,
Option 2 is not logical. Where will the data come from? The only source of data is the source databases. Data has to be migrated (read, transfer, write) from source database to target HANA.
For option 1, there will be network usage for sure. You can measure the usage at regular intervals to know the utilization.
The host running the R3loads and the HANA database servers are usually located in the same LAN.
They will usually have a 10-Gbps Ethernet. At 75% capacity, max speed of 875 MBPS can be achieved.
Bottlenecks can happen in several places during migrations.
Database should read the data (I/O bottleneck can happen)
into database memory buffer (memory buffer full, swapping - bottleneck can happen),
then into memory pipes (every R3load occupies some MB of RAM per process, not large though),
also number of R3load process does matter. Too many R3loads means CPU busy, CPU bottleneck can happen.
then transferred over network (network bottleneck can happen),
then target host's CPU, RAM, HANA database memory buffer, I/O.
Out of all these bottlenecks, even if even one happens, that will influence the entire export/import runtimes.
Source database are configured with memory buffers parameters for optimum day-today operational purposes (for example, oracle SGA).
These memory parameters influences the maximum throughput at which data can be exported. Once it reaches this optimum speed at which a given database can serve the data export, even increasing another 500 R3load export process does not increase the export speed.
Therefore these parameters should be tuned for optimum data export purposes during migrations. There is a SAP Note with recommended parameters to be set for migration purposes.
In my experience, target host is normally of newer technology, latest hardware with higher capacity in terms of CPU, RAM, I/O, etc.
The bottleneck will be at the source database and in case of very old hardware the CPU, RAM, storage I/O, network.
Therefore you can evaluate by doing few mock exports and identify the bottlenecks and work on alternatives to achieve maximum speed.
I normally do multiple mock exports, starting with low number of R3loads, increase step-by-step, keep measuring the size of exported data directory every 60 seconds. At one point increase R3load process don't increase the speed of export. Then I know the optimum R3loads to use.
Let me know if you still have questions.
Kind Regards,
Muniraju
Hi Muniraju,
Thanks for the explanation. yes, totally aware of others bottleneck.
Just want to be clear on how exactly data being transfer and import to target HDB.
Thanks again, Much appreciate!
Nicholas Chang
Hi Boris,
We recently were performing a ERP EHP6 Non-unicode to HANA Migration using DMO.
The first import finished fine.
We are using latest tools : SUM 16 SP2
This run completed just fine.
But when we used the migration repetition option by providing duration files for optimal table splits / handling mechanism and did a repeat of dryrun; It was observed that several packages of CDCLS table just did not get exported/imported at all and so CDPOS is empty in HANA side.
All migration tools used are latest.
Such packages gave an message (Error?) such as "Pipe does not yet exist, not starting import."
I have read your blogs where you mention this is just a warning. But I wonder if its just coincidence here that these all CDCLS table split packages did not get imported at all. Though .XMLs are generated and packages haven't failed either.
So we are wondering what could be reason for such data loss.
HANA DB gives logs as for e.g : -
Mergedog Mergedog.cpp(01134) : Cannot acquire table lock, automerge given-up: {IndexName: SAPWS2:_SYS_SPLIT_CDPOS~4
Any thoughts please ?
BR,
Sanjay Sahita
HI Sanjay, did you already open a customer incident to allow a detailed analysis of the issue? Regards, Ronald
Hi Ronald,
Yes I have already opened a SAP Incident; Awaiting some response from SAP on same.
Checks after phase MAIN_NEWBAS/RUN_RSCPINTS_POOL2ORG -ve ( 175189 / 2016 )
BR,
Sanjay
Hi Boris,
Do you have any response on this issue ?
I have already raised an incident at SAP. But still awaiting some concrete response.
BR,
Sanjay Sahita
Hi Sanjay,
thanks for opening an incident. Support is involved, and the colleagues have taken care.
BR, Boris
Hi Boris,
While performing DMO using SUM 1 SP17 we are getting the error during the phase MAIN_SHDRUN/SUBMOD_MIG_SMIGR/RUN_SMIGRCREATEDDL_740 it is failing with SELROWCOLSTORE.TQL No such file or directory /SUM/abap/var.
I could see that the file exists under the directory /SUM/abap/var not sure why the tool is not able to pick the file.
Source version: NW 7 EHP 2 SP13 on DB2 LUW 10.5 FP7
Target : NW 750 SP03 on HANA SP11 Rev112 on RHEL 6.7.
Any ideas pls
Regards,
Avinash
Have you got more than one application server? Perhaps the batch job was executed on a server where your filesystem was not mounted.
We have only one application server ASCS and PAS are running on the same host
Are you running DMO from an NFS ?
I have another question to add...I would appreciate any help in this regard..
What happens if there is packet loss during the DMO migration and using the pipe option? Will is stop with an error message or continue with the migration (and end up with corrupt data)?
If it stops with an error message, can we restart it from the point of failure or do we have to reset the whole migration and start again from scratch.
Hi Sunil,
always a good idea to read the guide ...
http://service.sap.com/~sapidb/011000358700000950402013E/
Section 4.3 "Special Features for DMO", part on "Failed Buckets Notification Flag".
Of course you will NOT end up with corrupt data ...
Regards, Boris