Has anyone else felt that data migration tools are tough to use?
The toolset I would like to manage a data migration is the following:
1. A data dump area in the SAP environment where the data from the source system is dumped, as is, everything as character strings (no length limitation). Basically, this should be a case of "just grab me the data and I'll dump it in SAP". No errors, no conversions, no cleansing, no specs, no extractors, no nothing. Just some selects in SQL to a file or the windows clipboard and an upload into a generic data store.
In previous projects, I have see tremendous effort being put into writing extractor specifications, building extractors, building upload tools and developing templates to then realize that something was forgotten, the spec wasn't quite right, and all objects in the chain have to be amended and rerun. I believe this is a hassle that can be avoided.
2. A data viewer/enhancer. An environment where you can look at the datasets you have loaded. You should be able to start applying data conversions (like silly date formats or whatever), validity checks and data fixes, filling additional fields using formulas and lookups (in standard SAP or in the generic data store) and filtering data out. The key thing is that this should be visual, i.e. you should see the affect on the data straight away telling you if you've missed a mapping or a formula isn't generating a sensible result.
There should also be a way to link datasets together. Like for example if you dump the headers and lines of documents as two separate datasets, you should be able to link the two datasets in a visual way. The system should check that each child record has a parent, and immediately show you which records don't have a parent with a sad face - like my old mac did when its disk crashed.
In previous projects, looking at the data was usually a case of opening lots of excel spreadsheets saved in network drives or sharepoint folders and doing VLOOKUPS all over the place. Often people would make changes and lose them again or start creating many versions of the data files. Such a real hassle.
3. Then you should have a nicely organized library of load modules: the infamous BAPIs and function modules but also possibly class methods and batch upload programs etc. This library should be easily searchable and well documented: when to use each one and all the checks and validations they contain, auth required etc...
Consistenly good SAP documentation of BAPIs is notoriously hard to come by but I'm happy to be proven wrong on this point...
4. A mapper/loader. This is the magic. So once you've looked over the source data and gotten rid of the stuff you don't care about and added the missing columns with the data SAP needs etc... you should be able to simply drag your data columns onto the input fields of the load module you've selected and hit a load button. Obviously you wanna save this mapping and give it a pretty name so you remember what it was all about and also transport it to QAS and PRD to execute there.
The most recent data migration tool from SAP seems to be the data migration cockpit (DMC) which runs partly on LTMOM (SAP GUI transaction where you define your migration 'projects') and LTMC or the equivalent Fiori App where you can download templates, upload data files and load the data into SAP using the project's BAPI. Certainly in recent years, this tool is very powerful and complete. But for me, it starts from a couple of false assumptions:
- You know up front which fields will be used in SAP, how they are populated (under all circumstances) by the load module you've chosen and you know what source data maps to it how. That is the assumption to steers you towards a solution based on generated templates and putting the workload on the source data side. "Fill my templates perfectly, or else..." The amount of understanding required from the source data consultant about SAP needs to be very high in this scenario.
- You don't need a programmer to do data migration. You don't need to understand the ins and outs of the BAPIs themselves. For simple data loads, yes, but most environments will have complex scenarios where you will need to understand the BAPI very well (especially its reliance on the right customizing) to get through the load. Debugging is definitely needed at times. And the new OO code in ABAP is much tougher to navigate than the old procedural code (I'm a big fan of OO - or should I say 'enormous' 😉 ).
- Everything can be loaded using BAPIs. Not always so...
