Garbage data issue while loading data from multiple sources to .csv file using SAP DS
In today’s world migrating or integrating the data from one system to another plays a vital role for any businesses.
While loading the data from SAP source to CSV file we have many tables which contain the data in various languages which means It includes information not only in western languages but also in other languages for example chinese, japanese etc.
While loading these kind of tables into CSV file, we will get improper data for other languages except English,The reason behind this is that when we load the data into CSV file format and open it using excel it uses windows-1252 encoding technique which is also known as ANSI (American National Standards Institute) encoding and is mostly compatible with English and other western languages.
In table TPART, the field VTEXT contains Chinese data at the source but when we load it in .csv file the garbage data is loaded.
To overcome this kind of data issue we need to tweak some settings in flat file editor in SAP DS.
- Code page: should be UTF-8 or UTF-16. (We have used UTF-8 since it is more flexible and efficient character encoding scheme that can represent data from all writing systems and is widely supported by modern softwares).
- Write BOM: should be Yes. (Enabling BOM is optional for code page UTF-8, but we enable it because some text editor especially older ones might not interpret UTF-8 without a BOM).
find the below links for more details on UTF-8 and file format properties.
What is UTF-8 https://www.freecodecamp.org/news/what-is-utf-8-character-encoding/
I hope you found this blog informative.