What is Unicode & why Unicode
In a computer system, one code page can be supported in clean manner. But due to globalizations, universal code page is required to support all characters of all languages.
Unicode is superset of existing character sets. This is an international encoding standard for use with different languages and scripts, by which each letter, digit, or symbol is assigned a unique numeric value that applies across different platforms and programs.
Unicode does have different encoding formats like UTF-8, UTF-16 & UTF -32. at SAP UTF-8 & UTF-16 is supported for encoding.
Unicode @Sap
Sunset of Non-Unicode Systems
"SAP NetWeaver 7.40 is the last release supporting non-Unicode"
(https://service.sap.com/~sapidb/012002523100009958832014E/ ). Additional details can be found at SAP Note 2033243 - End of non-Unicode Support: Release Details
so precisely SAP ECC 6.0 EHP7(based on NW 740) is last supported non-Unicode system (some database restrictions apply *)
in SAP there are three types of non-Unicode systems
Here focus of this blog on Unicode conversion from single code page Non-Unicode (NUC) SAP systems to Unicode system
How to find which code page installed in your system - You can get the information of installed code-pages from the table ‘TCPDB - Code page used in this database’ via se16. For the SCP system., there will be one code page installed e.g. 1100
Note: - If TCPDB contains more than one entry then the system is an MDMP system.
Single Code Page conversion is supported as of release SAP Web Application Server 6.20 and all higher releases.
Minimum Support Package level (SAP Note - 551344 - Unicode Conversion Documentation)
You must apply at least the following Support Packages for the Unicode Conversion:
SAPKB62065
SAPKB64023
SAPKB70017
SAPKB70106
However, as the conversion tools are still being improved, SAP strongly recommends you apply the newest available Support Package of your release.
Unicode Conversion Project Holistic view (SCP -- > Unicode)
Pre-Conversion
In early phases of project setup below points should be considered well in advance
Check the interfaces dependency on the code pages as well with system code pages.
e.g. if you have any third-party executables which are working with SAP kernels then those should work with Unicode Kernels.
In addition, there are tools or jobs which are generating /feeding some files are in non-Unicode format which must be looked during Unicode Conversion
The printers are getting used be compatible for Unicode device types
Below are additional hardware requirements for Unicode Systems
average numbers based upon parallel benchmarking of non-Unicode and Unicode customers systems
CPU: Average increase +10...30%
Main Memory: Average increase +40...50%
Network load: ~0%
Database size: Average increase at customers (sum of all sizes):
UTF-8 and CESU-8: -10% (for more than 90% of the installations the database size decreased because source DB had fragmentations)
UTF-16: +30...+40%
UCCHECK - ABAP Unicode Scan Tool
You can use transaction UCCHECK to examine a Unicode program set for syntax errors without having to set the program attribute "Unicode checks active" for every individual program. From the list of Unicode syntax errors, you can go directly to the affected programs and remove the errors. It is also possible to automatically create transport requests and set the Unicode program attribute for a program set.
Some application-specific checks, which draw your attention to program points that are not Unicode-compatible, are also integrated.
Sample input screen:
The default selection screen is fine to get an overview of affected objects. Note: by default, local object (package $TMP) are excluded from the selection. If there are local objects which are relevant for the customer, the corresponding option "Exclude $* Packages" should be checked off.
6. Reduce unnecessary data with the help of data volume management
Unicode Conversion runtime are complete depends on the volume of the data so it’s better to delete unnecessary data, archive historical data etc.
Below note will help you to prevent the growth of unnecessary data
2388483 - How-To: Data Management for Technical Tables
7. Understand the boundary conditions of the downtime
To decide the right approach to minimize the downtime this information is very necessary. Accordingly, you can decide the right approach e.g. parallel export /import, distribution monitor, DMO (database migration option – especially for HANA Migrations), special projects with SAP e.g. NZDT, Client transfer approach etc.
8. Scope of the project
Normally customer is choosing multiple maintenance activities like upgrade, HANA migration along with Unicode Conversion, DB upgrades, Platform changes, Data Center Relocation.
With additional scope it will additional activities during the project and which should be planned.
Customer should consider including Unicode Conversion with any maintenance as said above.
Note: - if customers insist to an upgrade to NW 7.50 or higher (e.g. S/4HANA), there will be two step approaches, as combination is not possible. As an alternative, NZDT (near zero downtime approach) may be used but this is project-based approach by SAP (693168 - Minimized Downtime Service (MDS))
9. Change management an impact of the parallel projects
With the additional scope of maintenance, the abap code will impact e.g. UCCHECK, SPDD, SPAU, HANA specific custom code impact, planned release project in parallel with Unicode Conversion project. Code freeze is very important during the project.
10.Always follow latest SAP documentations for additional preparations and best practices
Only Unicode Conversion - 551344 - Unicode Conversion Documentation
CU&UC - 928729 - Combined Upgrade & Unicode Conversion (CU&UC)
DMO/system copy - https://support.sap.com/en/tools/software-logistics-tools.html
Unicode Conversion
SPUMG - Database analysis tool for collecting words without language/code page information. SPUMG creates control information for the database conversion.
Before start of the SPUMG, you should check all necessary correction are available or not for SPUMG.
1142572 - Classification of notes of the Unicode conversion tools
1457258 - Correction instruction for the Additional Preparation Steps
SPUMG is consists several important steps.
In some of the preparations steps, it takes very long time e.g. report GRIX_DELETE_RW_EXTRACTS. Check this note applicable for your scenarios 1091152 - RW: Performance problems reading GRIX with ORACLE 10g
Note: before nametab creation these tables should be empty DDNTF_CONV_UC and DDNTT_CONV_UC.
This step is not mandatory for CU&UC, DMO execution because SUM (Software Update Manager) tool has the phase to generate nametab in uptime. But before SUM tool start you can create nametab using SPUMG to identifies issues prior. During upgrade phases, it will be created once again.
Do not forget – after SPUMG activities kick off, never import dictionary /DB objects manually, Transport, upgrades etc. this will invalidate the SPUMG activities at some extent.
Before entering the Unicode Conversion downtime steps
R3load is the tool which export the database and creates a new Unicode database. The SWPM (Software Provisioning Manager, formerly known as ‘sapinst’) is the tool for the performing the export / import process for only Unicode Conversion.
In CU&UC, this process will be performed after completion of the system upgrade along with some additional preparation steps.
For specially HANA migration, the Unicode Conversion can be incorporated in SUM tool where is the target is EHP7 (NW 740) + HANA.
How long does it take to convert a database?
Time required to convert the database will depend on many factors:
Procedures
High level overview of parallelism of the procedures
Which Procedure should be used?
Database Migration Option for Software Update Manger (DMO)
DMO simplifies the procedure of a migration to SAP HANA by combining the upgrade and the migration including Unicode conversion (target release only NW 740)
Benefits:
For more details refer blog https://blogs.sap.com/2013/11/29/database-migration-option-dmo-of-sum-introduction/
Important SAP Notes:
784118 - System Copy Tools for ABAP Systems
1322715 - Unicode FAQs
1319517 - Unicode Collection Note
1001383 - Distribution Monitor - Known problems and fixes
936441 - Oracle settings for R3load based system copy
1790232 - FAQ: Unicode - Technical FAQs
2033243 - End of non-Unicode Support: Release Details
Rahul Shetti.
In a computer system, one code page can be supported in clean manner. But due to globalizations, universal code page is required to support all characters of all languages.
Unicode is superset of existing character sets. This is an international encoding standard for use with different languages and scripts, by which each letter, digit, or symbol is assigned a unique numeric value that applies across different platforms and programs.
Unicode does have different encoding formats like UTF-8, UTF-16 & UTF -32. at SAP UTF-8 & UTF-16 is supported for encoding.
Unicode @Sap
Sunset of Non-Unicode Systems
"SAP NetWeaver 7.40 is the last release supporting non-Unicode"
(https://service.sap.com/~sapidb/012002523100009958832014E/ ). Additional details can be found at SAP Note 2033243 - End of non-Unicode Support: Release Details
so precisely SAP ECC 6.0 EHP7(based on NW 740) is last supported non-Unicode system (some database restrictions apply *)
in SAP there are three types of non-Unicode systems
- Blended Code Page System (Rel 3.0D)
- MDMP (Multi Display Multi Processing) (Rel 3.1I) – multiple code pages in one system. MDMP will no longer be (fully) supported from SAP ERP 6.0 onwards. For more details of MDMP support refer SAP Note 79991 - Multi-Language and Unicode support of SAP applications
- SCP (Single code page)
Here focus of this blog on Unicode conversion from single code page Non-Unicode (NUC) SAP systems to Unicode system
How to find which code page installed in your system - You can get the information of installed code-pages from the table ‘TCPDB - Code page used in this database’ via se16. For the SCP system., there will be one code page installed e.g. 1100
Note: - If TCPDB contains more than one entry then the system is an MDMP system.
Single Code Page conversion is supported as of release SAP Web Application Server 6.20 and all higher releases.
Minimum Support Package level (SAP Note - 551344 - Unicode Conversion Documentation)
You must apply at least the following Support Packages for the Unicode Conversion:
SAPKB62065
SAPKB64023
SAPKB70017
SAPKB70106
However, as the conversion tools are still being improved, SAP strongly recommends you apply the newest available Support Package of your release.
Unicode Conversion Project Holistic view (SCP -- > Unicode)
Pre-Conversion
In early phases of project setup below points should be considered well in advance
- Interface Unicode Compatibility
Check the interfaces dependency on the code pages as well with system code pages.
- All interfaces which are communicating through RFC’s can be handled with RFC UC Flag enablement in sm59
- For third party interfaces should be Unicode compatible once the system is migrated to UC
- Third Party tools are Unicode compatible?
e.g. if you have any third-party executables which are working with SAP kernels then those should work with Unicode Kernels.
In addition, there are tools or jobs which are generating /feeding some files are in non-Unicode format which must be looked during Unicode Conversion
- Printers compatibility
The printers are getting used be compatible for Unicode device types
- Hardware Requirements (SAP Note 1139642 - Hardware Requirements in Unicode Systems)
Below are additional hardware requirements for Unicode Systems
average numbers based upon parallel benchmarking of non-Unicode and Unicode customers systems
CPU: Average increase +10...30%
Main Memory: Average increase +40...50%
Network load: ~0%
Database size: Average increase at customers (sum of all sizes):
UTF-8 and CESU-8: -10% (for more than 90% of the installations the database size decreased because source DB had fragmentations)
UTF-16: +30...+40%
- UCCHECK (customer abap code is UC flagged or not?)
UCCHECK - ABAP Unicode Scan Tool
You can use transaction UCCHECK to examine a Unicode program set for syntax errors without having to set the program attribute "Unicode checks active" for every individual program. From the list of Unicode syntax errors, you can go directly to the affected programs and remove the errors. It is also possible to automatically create transport requests and set the Unicode program attribute for a program set.
Some application-specific checks, which draw your attention to program points that are not Unicode-compatible, are also integrated.
Sample input screen:
The default selection screen is fine to get an overview of affected objects. Note: by default, local object (package $TMP) are excluded from the selection. If there are local objects which are relevant for the customer, the corresponding option "Exclude $* Packages" should be checked off.
6. Reduce unnecessary data with the help of data volume management
Unicode Conversion runtime are complete depends on the volume of the data so it’s better to delete unnecessary data, archive historical data etc.
Below note will help you to prevent the growth of unnecessary data
2388483 - How-To: Data Management for Technical Tables
7. Understand the boundary conditions of the downtime
To decide the right approach to minimize the downtime this information is very necessary. Accordingly, you can decide the right approach e.g. parallel export /import, distribution monitor, DMO (database migration option – especially for HANA Migrations), special projects with SAP e.g. NZDT, Client transfer approach etc.
8. Scope of the project
Normally customer is choosing multiple maintenance activities like upgrade, HANA migration along with Unicode Conversion, DB upgrades, Platform changes, Data Center Relocation.
With additional scope it will additional activities during the project and which should be planned.
Customer should consider including Unicode Conversion with any maintenance as said above.
Note: - if customers insist to an upgrade to NW 7.50 or higher (e.g. S/4HANA), there will be two step approaches, as combination is not possible. As an alternative, NZDT (near zero downtime approach) may be used but this is project-based approach by SAP (693168 - Minimized Downtime Service (MDS))
9. Change management an impact of the parallel projects
With the additional scope of maintenance, the abap code will impact e.g. UCCHECK, SPDD, SPAU, HANA specific custom code impact, planned release project in parallel with Unicode Conversion project. Code freeze is very important during the project.
10.Always follow latest SAP documentations for additional preparations and best practices
Only Unicode Conversion - 551344 - Unicode Conversion Documentation
CU&UC - 928729 - Combined Upgrade & Unicode Conversion (CU&UC)
DMO/system copy - https://support.sap.com/en/tools/software-logistics-tools.html
Unicode Conversion
- Technical Preparations
- Consistency Check for cluster & Pool Tables (SAP Notes – 1348055, 1784377 & 1077403)
- Conversion of customer-defined code pages
- Delete Matchcode IDs (report TWTOOL01 & SAP Note 24860)
- Language Flag Maintenance (report RADNTLANG, SAP Note 480671)
- OTF Documents after the Unicode Conversion (report RSBCS_CONVERT_OTF_FOR_UNICODE - Note 1349413)
- Translation Environment (tcode SE63, Note – 1055820)
- Unicode Pre-conversion – SPUMG
SPUMG - Database analysis tool for collecting words without language/code page information. SPUMG creates control information for the database conversion.
Before start of the SPUMG, you should check all necessary correction are available or not for SPUMG.
1142572 - Classification of notes of the Unicode conversion tools
1457258 - Correction instruction for the Additional Preparation Steps
SPUMG is consists several important steps.
- Clean the old SPUMG execution if not needed
- Upload the exception list from the SAP Note 996990 - Exception List for transactions SPUMG and SPUM4
- Start the additional preparations steps (report UMG_ADD_PREP_STEP)
In some of the preparations steps, it takes very long time e.g. report GRIX_DELETE_RW_EXTRACTS. Check this note applicable for your scenarios 1091152 - RW: Performance problems reading GRIX with ORACLE 10g
- Language List - All languages which are productively used in your system are now inserted in the Language List as active languages in conjunction with the system code page
- Verify the SPUMG settings (you should change the Global Fallback Code Page from 1100 to 1160)
- Initialize the Worklist - Depending on the data volume the initialization job may take several minutes. The worklist of the Consistency Check (CC worklist) contains a list of all tables taken from the database’s nametab. This will be first steps of the Consistency Check.
- Consistency Check – this is the consistency check for the tables between dictionary and database. You can expect some inconsistencies issues for some tables. Unicode conversion guide has the detailed steps to how to correct
- Nametab Generation – for the Unicode conversion the nametab has to be created for the Unicode export. The report RADCUCNT will be scheduled through automatic steps from SPUMG.
Note: before nametab creation these tables should be empty DDNTF_CONV_UC and DDNTT_CONV_UC.
This step is not mandatory for CU&UC, DMO execution because SUM (Software Update Manager) tool has the phase to generate nametab in uptime. But before SUM tool start you can create nametab using SPUMG to identifies issues prior. During upgrade phases, it will be created once again.
Do not forget – after SPUMG activities kick off, never import dictionary /DB objects manually, Transport, upgrades etc. this will invalidate the SPUMG activities at some extent.
- Additional downtime steps – it will trigger Report UMG_ADD_DOWNTIME_STEP automatically.
Before entering the Unicode Conversion downtime steps
- Update the database statistics
- Update worklist for any new objects and complete the consistency checks
- Create delta nametab using report UMG_SHOW_UCTABS if required
- Unicode Conversion – Downtime
R3load is the tool which export the database and creates a new Unicode database. The SWPM (Software Provisioning Manager, formerly known as ‘sapinst’) is the tool for the performing the export / import process for only Unicode Conversion.
In CU&UC, this process will be performed after completion of the system upgrade along with some additional preparation steps.
For specially HANA migration, the Unicode Conversion can be incorporated in SUM tool where is the target is EHP7 (NW 740) + HANA.
How long does it take to convert a database?
Time required to convert the database will depend on many factors:
- Database size
- Number and speed of CPUs available
- Distribution of sizes of the tables
Procedures
- SWPM - sequential
- Parallel export /import using SWPM
- Distribution Monitor
High level overview of parallelism of the procedures
Which Procedure should be used?
Database Migration Option for Software Update Manger (DMO)
DMO simplifies the procedure of a migration to SAP HANA by combining the upgrade and the migration including Unicode conversion (target release only NW 740)
Benefits:
- Migration steps are simplified
- System update, Unicode Conversion (*), and database migration are combined in one tool
(*: only possible for a target based on 7.40 or lower) - Business downtime is reduced
- The source database remains consistent, so a fast fallback is possible
For more details refer blog https://blogs.sap.com/2013/11/29/database-migration-option-dmo-of-sum-introduction/
Important SAP Notes:
784118 - System Copy Tools for ABAP Systems
1322715 - Unicode FAQs
1319517 - Unicode Collection Note
1001383 - Distribution Monitor - Known problems and fixes
936441 - Oracle settings for R3load based system copy
1790232 - FAQ: Unicode - Technical FAQs
2033243 - End of non-Unicode Support: Release Details
Rahul Shetti.
- SAP Managed Tags:
1 Comment
