Data migration for Utilities: Combination of communication and technical skills
Data migration is always a challenging task, which, if approached with the needed commitment, can result in satisfaction of both customer and SAP consultant. However, there are many things to have in mind while going through the phases. Ill try to empasis some of them, in order to help the future data migrators acomplishing their task.
A successfull data migration project is a combination of technical and communication skills. Every data migration project is different but they all have several things in common. Ill try to explain different approaches, stress out the common issues and mistakes and try to write a guideline for the SAP consultants who are or will do the IS-U data migration.
EMIGALL: Get to know the tool
EMIGALL (IS-U Migration Workbench) is a powerfull tool which is well documented (check this link: IS-U Migration Documentation). As any other tool, it needs the knowledge of the person using it. Depending on your learning preferences, you can choose your own method of getting to know it. I would suggest reading the comprehensive PDF document (available from the service.sap.com, listed in the upper post) and trying it out on your test system.
There is no cookbook which will cover all and every needs. The setup provided in the SAP company in the EMIGALL tool, however, does cover a lot. Depending on your target system, needed and available data, you will need to keep it as-is or modify it to include everything you need. Custom (user) fields are supported and, again, well documented. Once you get familiar with the tool, you will be able to tell which data can and which can not be migrated.
EMIGALL is not the only available tool. You can adjust the well known LSMW tool for the IS-U migration purposes, and there are also several commercialy available tools that can help you with (or carry all of the) migration. I prefer using EMIGALL as it has a well developed Key and Structure Management, ability to add custom fields and the possibility to write rules for each field.
The prerequisits for using EMIGALL are prepared and configured system. EMIGALL can (and will) import master, transactional and history data (business partners, contract, accounts, documents, installations, meter reads, …). It will not import configuration settings nor similar data (such as prices, rate categories or industry sectors).
Analisys: Existing data – legacy system
Legacy data can basicaly be anything: from several different database engines to a well structured data warehouse. A common customer initial expectation is that data migration will solve their troubles just by existing. This isnt, and can not be true. At the very beginning of the data migration project it is important to let the users know what they will have to do and what they have to deliver to the SAP consultants and in which format.
This phase is usualy a point where the users will (again, depending on their legacy data storage method) recognize their data system weak points: incorrectly structured data, missing data, unlinked data and similar cases. If they dont, it would be wise to point that out for them.
Know the data: Data cleansing and purging
From my experience, its troublesome to talk the customers into settling their data before migration, but its also the method that gets the best results. Lots of situations can also be dealt with once the data is migrated to SAP, but it shouldnt be the right way of doing it.
It is hard and demanding to do several tasks at once, but this task will result in great gain if done. Some of the typical situations for the data to be cleaned or purged:
- Business partner created more than once – this is the most common situation in legacy systems not supporting more than one installation per business partner; You will also see several business partners created when impossible to enter more than one address;
- Installations that are inactive and have no financial data – this can be a result of a region that is no longer maintained by the company due to legal restructure, market change or any other reason; Such data can reside in legacy archives, but they dont have to be imported in the new system
- Addresses – there are so many issues that its even hard to cover them all. The most recognized situation cover entering addresses without using the city and street files, having the same address written in several different forms, entering wrong or impossible zip code, … There are many situations. In order to fully use the IS-U potential and possibilities, this has to be solved
Data conversion: Communication and education
For someone used to LSMW data transfers, EMIGALL can be freightening at first sight. Thats where the SAP consultant jumps in with explaining and examples. Migration objects and automatic structures (and the way of their representation in a text file) are understandable. What gets here as an issue is usualy the customers idea that they will export their tables into text files “and thats it”. Some migration object that are needed in your IS-U system configuration will not exist in the legacy system, some of them will be split in several places and some data will be perfectly structured.
This step is probably the most important step in connecting the two systems. Most of the errors that show up here are caused by ignoring the instructions or not understanding the given task. Make sure you take as much time as needed to correctly explain what exactly do you need, what is mandatory and what is optional, and which field means what in their future (SAP) system. If possible, try to give examples based on their legacy system (if you got the chance to see and understand it).
EMIGALL offers export of its structures and fields per migration object. Combining that with the exemplary text file can help in producing expected results. Always expect questions as there will be specific, non covered situations. Depending on how many those situations exist, you can decide wether to import them as a mass import or find another (perhaps manual) way of importing. Entering data manualy through transactions may, in some cases, be the best choice. Dont avoid it just because its not “fully automatic”.
When preparing data, dont forget that several fields have the selection lists. Make sure you prepare the possibilities in a separate file. This would be something like a look-up table for the people who are preparing the data. That way there will be one less chance to enter incorrect data.
Sometimes it helps to visualize things. One of the things you can visualize is the correlation between migration objects. You can do that in any graphical representation tools that you know how to use.
In the process of converting the data, several common issues arise:
- the date format can be in various formats – the prepared data should follow the destination (in our case EMIGALL) tool needs: YYYYMMDD
- numbers can be represented with coma or a dot as a decimal separator – EMIGALL works best if all numbers use dot as a decimal separator
- data can be in various code pages – make sure you use only one code page as a destination one; although it can be one of the many that SAP understands, you can consider using UTF-8 or unicode
One of the common made mistakes here is letting the IT personel itself prepare the data. They probably are the best tech people you will find, but they are not the ones who know all the business scenarios. This should be done by both the IT people (for their expertise) and the head operative people (for their business knowledge).
By using KSM (Key and Structure Management) you can easily link objects and let SAP deal with assigning new IDs. What you can suggest here is the key itself. Its always better to be able to easily identify what are you looking at. One of the suggested ways for creating keys would be using three letter migration object abbreviation and legacy key (for example, if the business partner code in legacy system was 100552, the key would be PAR100552). Then, when you see an error in the EMIGALL log, it will be easier to identify which record is faulty.
Once again, as I cant stress this out enogh: customers must understand given migration objects / automatic structures / fields and must be able to link them with their existing data. They also must know how will they create the data that currently does not exist in their existing system, if its needed to fill the migration objects. Time spent in explaining this is not lost at all.
Size: Amount of data
Knowing the amount of data will define your approach on the import itself. There are several methods you can use. Assuming having a lot of data, it would be best to plan the distributed mass import (check EMIGMASSRUN; this document might help you: EMIGALL – Scheduling Distributed Import Job (EMIGMASSRUN) ). In this case, you will need to plan the needed resources and the timeframe in which you will do it. In a more simpler environment, with less data, maybe it will be easier to do a sequential import of the migration objects. In either way, you need to know the correlation between the migration objects in order to know the import order.
Other systems: CRM
Its not rare that CRM and ERP systems are linked. Also, a link for the MM/SD/FI customers can be established/set. If it is the case, then they have to be considered in migration planning. The recommended way is to disconnect all links, and do the migration for the IS-U only, and then reestablish the link and trigger the replication of users to CRM. By correctly filling the automatic structures in migration objects, the ERP users will be created.
Testing: Preparation is needed
Assuming you have communicated your needs correctly, the customed understood it and prepared the data for you, you did all the needed configuration (links, rules, fixed values, …), doing import without testing would not be a good idea. Testing reveals errors which could result in faulty and incomplete data import. This can be done in a set of a data, which includes all the possible combinations or scenarios (you can do so in order to minimize the test data). EMIGALL itself will do a error log and display the errors which should be corrected prior the import. Sometimes its just the matter of code page, but sometimes it will require adding a new field to the automatic structure.
Once that is fixed, another tests should be carried on: the ones comprising of full load data and mass imports. By doing this you will get your system (and yourself/yourselves) prepared for the upcomming real import.
In this step you can test out what happens if you repeat the import (EMIGALL knows which data is already processed/imported) and how can you deal with the data thats not imported (can you change it and how? will you extract that data and put it in a new text file for further processing/import?). Its clever to check how many time will be needed for the import: EMIGALL will show the throughput in records per hour which is usefull for calculating the needed import time.
Importing: (Almost) the final step
Depending on the method you decided to apply (to keep it simple: EMIGALL or EMIGMASRUN), you need to plan the resources and timeframe for the tasks. Its hard to talk in absolute values (time needed for import, CPU and disk usage) as it heavly depends on the server configuration you have. EMIGALL really is fast and you can, most likely, carry a migration in less than a day.
Dont skip this step! Its something that should be done by both SAP consultants and customer team. There are several approaches in testing your data after the final import. Any way you do, the tests should include at least random checks of all types of data (several business partners, several devices, several contracts, …) and a exact test of financial data (customer open items, general ledger (reconciliation) account balance vs sum of customers (open) items). No matter how good the planning and the strategy is, there is always room for unplanned errors. Be aware of them and plan their solving. Usually, there are a few or none.
Knowing the migration tool, communicating the needs and changes to the customer, testing the data and planning the whole process are the keys to the successfull data migration. Its a process that should be taken seriously but not being freighten of it. Good luck!
Hi Marko.... Thanks for sharing this blog with us, it seems quite a good blog with on import runs and steps to be followed in migration.
Thanks for the detailed info.
Can EMIGALL be used for migrations outside the IS-U domain? Or is specific to IS-U?