Data migration by using Custom scripts

[Former Member]

Introduction

Whenever we move from legacy system to SAP system we need to migrate the existing data in legacy system like Business partners, Relationships and Transactions to the new system. I worked in a SAP CRM implementation project for a CPG major which was moving from SIEBEL system to SAP CRM system. I prepared this document to help people who have similar requirements and want to migrate data from legacy systems to SAP systems. This document will also help to avoid pitfalls where things can go wrong and affect the quality of data migrated.

Source and Target system

Firstly we need to identify Source and Target system for a particular object (Like Account, Order, Activity etc.). An object can have single source of data OR multiple sources of data.

For example in our project we had two kinds of customers:-

1.      Accounts: All accounts were maintained in ECC system along with SIEBEL system. So some data for a particular account would come directly from ECC via middleware for example address details. The data which is not maintained in ECC and present in SIEBEL system will be migrated by data migration program for example custom fields and marketing attributes. So this is case of two data sources ECC and SIEBEL.

2.      Prospects: All prospects were maintained in SIEBEL system and were not present in ECC. So data source for these was BORIS data mart where all SIEBEL data was maintained. So this is case of single data source.

Data mapping

·        We need to get the understanding of the object Like (Account, Order, Activity etc.) by conducting calls with client.

·        We need to get all the details of all the fields that will be there in that object and their usage.

·        Then we need to map all these fields with the standard fields present in SAP system.

·        There would be some fields that will not be present in the standard system. We will need to get custom fields developed for these fields.

·        We also need to get conversion logic if any applicable for a field. This will be applicable where usage of field OR its definition is different in the legacy system and SAP system. We will need to have this logic in our data migration program to properly maintain the data.

·        Finally we arrive at the data mapping sheet in the below format:

·        We will also need to convert the data where SAP CRM field is LOV (List of value) field. It is advised to get the data in the SAP CRM keys. If Business agrees to provide keys of LOV fields in the extract file then it is easier to get their description in the field. Otherwise we will have to have conversion logic in our program to get proper List of values in the New SAP system.

Data format

·        We can develop programs to handle different kind of formats. It’s better to finalize one format for all objects for convenience. In our project I convinced client to provide data in Pipe delimited flat files as these are easy to use, less in size as compared to excel files for the same amount of data.

·        We need to also finalize the sequence in which the data will be provided for an object so that the program will be developed accordingly. So one extract format is finalized in which all fields are mentioned in predefined sequence. For example account migration can have something like this

Data linkage

·        When the data is linked in the legacy system then we will need to store the legacy system IDs in SAP system to restore the linking in the new system. I will explain this by an example:

In our project Accounts had SIEBEL IDs and when we migrated in SAP CRM they had SAP CRM IDs. There were some transactions where these accounts were present in SIEBEL system. So we stored SIBEL IDs of accounts as external identification number. So when any transaction like activity came with SIEBEL ID of account then our program fetched the SAP CRM ID of that particular account with the help of SIEBEL Ids stored for every account in the system.

·        Configuration steps for creating external number in SAP CRM system for business Partners:-

Transaction code	SPRO
SAP CRM IMG menu	IMG-> SAP Customizing Implementation Guide-> Cross-Application Components-> SAP Business Partner-> Business Partner-> Basic Settings-> Identification Numbers

Define Identification Categories:

Define Identification Type:

·        For other objects like activities etc. we can have custom fields to store these legacy system IDs. Then these Legacy system IDs will help us in setting up the linkage of Legacy system in the new SAP system.

Data migration scripts

·        After getting clarity on field mapping for a particular object we prepare custom scripts by utilizing BAPIs, Standard function modules and coding.

·        There can be multiple scenario for an object like

1.      Object has already come via middleware from ECC and need to be updated with additional fields that are not present in ECC.

2.      Need to create and object in SAP CRM.

3.      Need to update and object which is already present but needed data refresh.

·        We develop custom programs to cater all scenarios that are required.

·        We can have option to run these programs from both application layer and presentation layer. But it is suggested to run programs from application layer where data is large and it can take significant time.

·        There would be many fields that will be same for all the records. These fields can be included in the initial screen of the program.

For example: We included Sales Org ID, Distribution Channel, Division, Customer group, Customer Pricing procedure and currency in the initial screen of the customer migration program as these were same for all the customers.

Errors that can be avoided for Data migration

Configurations and custom development

All the configurations and custom development should be moved in the new system before we start data migration for an object.

UTF-8 Format

All files should be in UTF-8 format to avoid errors from application server. We faced one issue while running the job from application server where special characters were not getting migrated successfully. For example in place of û, é etc. we were getting #. But when we saved the file with encoding value UTF-8 for it on the application server and ran the job the problem got resolved and we were successfully able to migrate the data.

System Check

All files will be run in two parts. First one will have few data 5-10 records and second one will have all the records. If everything goes fine in the first run we will run whole load. This will help us in knowing any system specific problem.

Mandatory fields

Every object has some mandatory fields for creation. Like for creation of Contact person we need to have Last name, Language and Country. If these fields are not present in the extract then we cannot create Contact person. So before running the program it is advised to check whether all the mandatory fields are present in the extract file to avoid failing of records later.

Keys for LOV fields

Extract file should have correct keys for LOV fields.

Territory Set up

All territories should be set up in the system before we start migration.

Coordination

There should be proper coordination with other teams to avoid inconsistency and ensuring proper migration of the data. For example

·        Integration Team: Data migration can be started only after middleware loads are complete

·        Solution Team: Data migration can be started only all custom development and configuration are complete

·        Basis Team: All programs are moved to the system in which we want to migrate the data.

Switching functionalities off and On

There are some functionalities that need to be off during data migration load to ensure proper data migration.

Example 1: In an activity if we are getting business partners in our extract then we want them to be migrated. We also have partner determination procedure configured in the system for this particular type of activity. Then there are chances that the Partner determination procedure will suggest some values which can differ from the values in the extract. So to avoid this situation we turn Partner determination procedure OFF during data migration and once the data migration is complete we switch it ON.

Example 2: We are migrating Historical order data. We have configuration for sending mails and fax once order is created in the system. But these all are historical orders and we do not wish to send mails and fax for these. So to avoid this we disable action profiles that control sending of mails and fax before data migration of orders and then enable that again after the data migration is done.

Data inconsistency

We need to ensure that the data is consistent for different objects. I will explain this by some examples

Example 1: In activity extract we get many business partners like Account, Contact person, employee responsible etc. If we get one employee responsible in the activity extract and the same in not present in the system OR it is present but not maintained as Employee.

Example 2: In order migration get many products. One of these products comes with EA (Each) Unit of measure. But EA is not maintained UOM for this product.

These kind of situation create data inconsistency in the system. We can avoid these type of errors by validating the data beforehand and sequencing the objects for migration properly.