SAP Data Migration Methodology
Goal of this document
This document provides you with a procedure to assist you organizing and performing the data transfer from the legacy system.
It describes a methodology for data migration I used successfully in different implementations. It is based upon my previous experiences. There is no warranty on its content or on the results. This guide gives you suggestions. It is up to you to take the hints and make up your own methodology.
Common Terminology and Abbreviations in Migration Projects:
Note: The terms SAP and R/3 are both use interchangeably to refer to SAP R/3 system.
Big Five: When referring to the Big Five, it means Material Master, Customer Master, Vendor Master, Bill Of Materials (BOM) and Routings.
Business Objects: To help in the analysis and transfer process, the data are not treated as tables or field contents but rather as objects in term of business operational. These are called Business Objects.
Business Object DC responsible: Responsible of the conversion process (Legacy data source and integrity, mapping, conversion rules, etc.) and for the respect of the planned schedule for his Business Object.
Business Object Owner: The one that owns the information in the everyday business. This is the person that will make the strategic choices on functional requirements for the business object and that will do the final validation of the converted data. Can be identified by finding “The highest hierarchical person who will be directly and mostly affected if the business object does not work”
Data Conversion & Data Migration: The data conversion process. “Data conversion” and “Data Migration” terms are used interchangeably in the document.
DC: Abbreviation for the data conversion process.
Domain: Functional domain within the project, like Finance, Sales, Production, etc.
Flat File: A file format used to import data into SAP. The flat file is a plain text file with a tab separator between fields. It can be easily generated from Excel or Access.
Intermediate file: An Excel, Access or other type of file, which is manually manipulated in a process between the LS extraction and the flat file generation.
LSMW: Legacy System Migration Workbench. It is a SAP tool for conversion that permits data loading using flat files extracted from the Legacy System.
Cross reference table or X-Ref table: A table that shows the relation between fields when one value is related to a parent field. For example, the “Sales Organization” will be set accordingly to the material type.
WBS: Work Breakdown Structure.
Implementing SAP is an important challenge, both in terms of resources (people, money, time) and in business process. A lot is at stake and, for most of you, failure is not an option you can afford. To put all odds on your side, you need a good methodology. One that will provide you with a realistic planning, a solid organization, a way to manage the process and control tools to detect and correct slippage before it becomes a problem.
Main steps of the conversion methodology:
Before you even start to work on specs, you must first get organized. Getting a good planning and organization structure take about two weeks for the first draft, which will leave you with some questions on project organization. Getting a complete and final planning will take at least one more week. Any unsolved issues on these will haunt you throughout the project, so finish this completely before stating any other step.
The data conversion requires functional and technical resources from most departments. These same resources will most probably be involved in other part of the project. For this reason, the risk of conflicting task is high and can quickly lead to a bottleneck where key peoples are overloaded. For this reason, you should consider the data conversion as a project within the project. This translates into the preparation of a complete conversion plan that will help you go through the process and will permit to foresee and solve the conflicting resources usage before the bottleneck ever occurs.
The main steps of the data conversion are:
Organization of the data conversion (Project manager & data conversion coordinator)
- Data conversion plan
- The WBS with workload estimates
- The calendar planning with resources loading
Going on with the Business Objects data conversion (The resource responsible of the Business Object DC)
- Data Purging and Cleansing
- Mapping and conversion rules
- Extract and Load Programs from rules
- Data and Rules Adaptation (adjusting rules and programs following testing)
- Load Unit Testing (unitary testing – small volume of manual data)
- Extract and Load Full size testing (data test and validation – large volume with real extracted data)
- Full data loading into ACCEPTANCE SYSTEM
- Full data loading into PRE PRODUCTION SYSTEM
- Validation of converted data and Key User + Business Objects Owner Signoff
- Full conversion into PRODUCTION SYSTEM and final Signoff
Data Conversion Plan
A Business object is a general category for data that defines something like material master, vendor master, stocks, orders, purchase requisitions or organizational units. The first step is identifying which business objects (Objects) are required in your SAP implementation.
There are three types of data involved in a SAP system: master data, transactional data, and historical data.
- Master Data. Application master data tends to be more static once defined. Most master data can be driven by the legacy applications. Examples include vendors, customers, charts of accounts, assets, bills of materials, material masters, info records, and so on.
- Transactional Data. Transactional data is current and outstanding transaction data that needs to be captured from the legacy system and defined to the SAP R/3 applications for business process completion. Examples include accounting documents, open purchase orders, open sales orders, back orders, and so on.
- Historical Data. Historical data needs to be brought over from the legacy system to the SAP R/3 System for reference purposes. Examples include closed purchase orders, closed sales orders, summary general ledger information, and so on.
Information to complete the conversion plan
- What Which business objects will be converted from the legacy system into SAP.
- Where Where are the data, which Legacy Systems are involved for the extraction.
- How much Estimate the number of records to be ultimately loaded into SAP.
- How There are two aspects to be considered :
- The way data is extracted from the Legacy System
- Automatically extracted from the Legacy system without manual intervention.
- Manually filled spreadsheet
- Combination of an automatic Legacy System extraction + Manual Entry into a spreadsheet
- The way data is injected into SAP :
- Automatic data transfer from a flat file into SAP
- Manually entering data with online transactions into SAP
- Combination of both
- The way data is extracted from the Legacy System
The data transfer method you choose will determine the types of resources you need. For example, you may need temporary employees for the manual data entry and programmers for writing your own extraction programs. You need to know both what data is in your legacy system and which SAP applications correspond to the business objects that will be transferred. One person does not have to know all of this, but the people who know this information should work closely together.
- Who Who is involve on each Business Object :
- Key user (Functional responsible of BO conversion : Rules, manual data corrections, test, validations)
- Responsible of data cleaning and purging in the Legacy System
- Responsible of the extraction
- Responsible of loading data in SAP
- Business Object Manager (Hierarchic owner who is responsible of day to day use and integrity of information and the one which will be signing off for data acceptance)
CONVERTING A BUSINESS OBJECT:
Data Purging and Cleansing
The purging and cleansing of the Legacy System will save you lot of time and effort in the following steps of the conversion. Start this as soon as possible and do as much as possible. This can be done without specific knowledge of SAP.
- Data Purging
Before transferring data from your legacy system, delete all the old and obsolete data. For example, you may delete all one-time customers or those for which there were no transaction in the last two years, also delete unused materials.
- Data Cleansing
This process corrects data inconsistencies and ensures the integrity of the existing data during the migration process. For example, there are often lots of inconsistencies in Customer and Vendor address fields. You will quickly find that SAP will not let you load any address fields unless you get them clean.
Mapping and Conversion Rules
The documentation of each business object will contain the Data conversion rules (or specification), which include:
- * Legacy sources and extraction procedures.
From which Legacy system(s) are we extracting the data and how. Document here specific steps that need to be taken.
- * Purging and cleansing rules
What are the cleaning steps to be taken and extraction filters to be used.
- * General Conversion rules
Guidelines to apply or rules that is used by many fields (thus avoiding to retype it and making updating easier as it is only in one place).
- * Fields Specific rules
Which SAP fields to use and how do we get the final value for each SAP field.
About the Rules
- General rules VS Field Rules
General rules are the one that does not yield directly to a field value. For example the way in which we differentiate the material types in the Legacy System is such a rule. Field rules are those that give a value for a specific field.
- Fields names
This is a crucial one. When discussing or writing notes, ALWAYS refer to a field in the form TABLE-FIELD. You will quickly realize that as the project go, different people will start using different names for the same field. As well they may start using the same name for different fields.
On top of this, some fields exist in different views in SAP master data. Sometime it is the same field that is shown at two places while other times it is really two different fields. The best way to know which case apply is to have the TABLE + FIELD information.
In Material Master, the field «Availability check» exists in the “MRP2” and the “Sales Gen” views. If you look at the TABLE-FIELD of each view you get :
MRP2 : MARC-MTVFP
Sales Gen : MARC-MTVFP
In both cases the TABLE-FIELD name is the same, so it is the same field.
In Customer Master, the field ” Terms of Payment’ exist in “Payment Transactions” and “Billing” views. If you look at the TABLE-FIELD of each view you get :
Payment Transactions : KNVV- ZTERM
Billing Views : KNB1- ZTERM
It is not the same field. In the payment view, the field is linked to the Company Code while for the Billing view it is linked to the Sales Organization (you find this by looking at the tables keys). So both of these fields can have different values
A special case for Material Master
Material Master involves all the domains and may require anywhere from 20 fields to a few hundreds depending on the complexity of your implementation. Some fields will be used by different domains while others will be used by only one domain but its value will have an impact on functionality used by another domain.
This is the most complex Business Object to document and, at the same time, it is the one you must start with in you conversion process.
1st step : Selection of the fields by each domain
- Get each domain with their consultants to go through the mapping file and look at the fields for each material type.
- The goal here is to see all the fields that are important and ask questions to understand them. This is done separately by each domain and documented in different mapping files.
- At these points we are not interested about where the values will come from and how will we get them. JUST GET THE MAPPING DONE and work on understanding what material master is.
- Each time a domain select a field for a specific material type, they must enter their domain type in the list. Here are some (theoretical) examples of mapping from MM, PP and SD
In Material Master, some fields can be entered / modified in different views. For example, the field “Goods receipt processing time in days (MARC-WEBAZ)” exists in views Purchasing, MRP2 and Quality management. When doing the rules and the load program, the same field can’t be in different views. To solve this, proceed as follow:
See with all implicated domains who are the lead for the field and decide in which view the field should be included.
Taking the example of the field “Goods receipt processing time in days (MARC-WEBAZ)”, it can be decided among the domains to put it in the Purchasing view (and nowhere else).
Material Master Conversion:
High Level Process Design
These are all the major views involved in Material Master Object:
- Basic Data
- Alternative UOM
- Inspection Text
- Sales Organization
- Sales General Plant
- Foreign Trade Import & Export
- APO Master Data
- MRP1 & 2
- Quality Management
Usually the Function specification Owners will do the recording method to capture all the fields on the above views and prepare the Mapping Logic.
The most complex design involves in Plant merging and Classification Merging. (Refer High Level Process Document)
CSG Split: Customer Segment Group
A member group of the type customer segment group is a collection of users, as defined by the Seller or merchant, who share a common interest.
For Eg: A mining Company has CSG’s like Cerro Matoso(CMSA),Met Coal (MTCO),Base Metals (BASE) etc.,
Based on CSG’s the data need to be split up before loading through LSMW for valuation.
The split logic can be done by Loop Component in Business Objects Data Services
- Get the distinct CSG’s in a dataflow.
- Assign a variable for the sequence number for the max value and loop from the initial.
Other Business Objects Conversion:
For the other BO, because they are simpler than Material Master and involve fewer people, we will start directly with the Conversion rules document. It is in this document that we will both, decide which fields we need and, in a second step, start working on the rules.
Here are some samples of BO conversion rules.
BOM conversion rules sample
Open Account Receivable conversion rules sample
Vendor Master conversion rules sample
Example of general rules
Note that SAP term “Security deposit” equal “Retention” in PRMS
Type of transaction
TYPE field in PRMS :
Partial PMT: “Pay.”
Credit Memo: “Cr M.”
Debit Memo: “Dr M.”
Invoice : “Inv.”
Non A/R cash: “Non AR”
Any other type is an error.
Validation to apply both at extraction and load.
Partial PMT………. must be negative in PRMS, if not ERROR
Credit Memo………must be negative in PRMS, if not ERROR
Debit Memo……….must be positive in PRMS, if not ERROR
Invoice……………..must be positive in PRMS, if not ERROR
Any other type is an ERROR.
LSM Load parameters
KTOPL – Chart of account : CA00
BUKRS – Company code: 0070
GSBER – Business Area : 0040
BUDAT – Posting Date : “05-31-02” or last day of last closed period.
OFFSET – Account (2) : REPRISECL
SKPERR – Skip err : X
Legacy System Migration Workbench (LSMW):
LSMW is used for migrating data from a legacy system to SAP system, or from one SAP system to another.
Apart from standard batch/direct input and recordings, BAPI and IDocs are available as additional import methods for processing the legacy data.
The LSMW comprises the following main steps:
- Read data (legacy data in spreadsheet tables and/or sequential files).
- Convert data (from the source into the target format).
- Import data (to the database used by the R/3 application.
But, before these steps, you need to perform following steps:
- Define source structure: structure of data in the source file.
- Define target structure: structure of SAP that receives data.
- Field mapping: Mapping between the source and target structure with conversions, if any.
- Specify file: location of the source file
Methods used for data migration like BDC, LSMW and Call Transaction
All the 3 methods are used to migrate data. Selection of these methods depends on the scenario, amount of data need to transfer. LSMW is a ready tool provided by SAP and you have to follow some 17 steps to migrate master data. While in BDCs Session method is the better choice because of some advantages over call transaction. But call transaction is also very useful to do immediate updation of small amount of data. (In call transaction developer has to handle errors).
SO Bottom line is make choice of these methods based of real time requirements.
These methods are chosen completely based on situation you are in. Direct input method is not available for all scenarios else, they are the simplest ones. In batch input method, you need to do recording for the transaction concerned. Similarly, IDoc, and BAPI are there, and use of these need to be decided based on the requirement.
Try to go through the some material on these four methods, and implement them. You will then have a fair idea about when to use which.
Difference between lsmw & bdc
BDC– It is Batch data communication. It’s used for data conversion from legacy system to SAP system. Only technical people can do it. Tcode is SHDB.
LSMW– It is legacy system migration workbench. Its also used for data conversion from legacy system to SAP system. But it is role of functional consultant.
There are 14 steps in LSMW. As soon as you complete the one step, automatically it will go to next step.
In general you can use LSMW. But if you want to transfer more than 40,000 data, then it is not possible in LSMW. That time you can take help of BDC
LSMW data migration for sales order VA01 / XD01 customer.