Can't convert Chinese characters with "sapdbmtk extract"

[Robin-Feng]

Hi There,

I got the wrong Chinese characters, after run this command: sapdbmtk extract.

The steps of re-process this issue:

1. Create a store procedure named "SP_By_Robin" in database "rrr", with Chinese characters.

2. Create a project with command: sapdbmtk create_project c:/rrr MSSQL HANA

3. Perform below command to setup this project:

  sapdbmtk config_set C:/rrr 110 "<gateway ip>"
  sapdbmtk config_set C:/rrr 111 "<gateway port>"
  sapdbmtk config_set C:/rrr 121 "rrr"
  sapdbmtk config_set C:/rrr 122 "dbo"
  sapdbmtk config_set C:/rrr 125 "GB18030"

4. Enable gateway

5. Run below command to extract from SQL:

sapdbmtk extract c:/rrr <MSSQL server> <MSSQL server ip> <port> <sa> <password>

6. See the result, the Chinese characters is wrong.

What should I do to fix this problem?

Refer to: This SAP Advanced SQL Migration > Tutorial

BR

Robin

[Javier-Martin]

               Hi Robin,

               I am able to reproduce in my environment with the information you have provided, thanks a lot for your help.

               These are the results from our investigations:

- As indicated in the manuals the migration tries to detect the encoding of the extracted data model but that is not always possible. When extracting from files it does it with utility like “file” and others, but sometimes it is not possible to get it. When extracting with the gateway it is even more complex and more taking into account that SQL Servera allows to use different charsets even at column level.

- Parameter #125 was created to to force some conversion and it is use for extract_offline, and that is the only solution for your current problem.

- Taking into account the repro file with GB2312  encoding you have sent our tests have shown the following when you extract in offline mode (extract_offline):

• If you don’t set any value for param #125 you get the output extracted file in UTF8 but the characters are wrong, this is because the migration tool is not able to determine the encoding of the file and no conversion from GB2312 to UTF is done.
• If param #125 is set to GB2312 then we force the tool converting from that to UTF8 and the extracted file encoding is UTF8 and the characters are ok.

So the only way for you to proceed to convert schemas with that encoding is to use text files with the extract_offline option setting param and our suggestion is to go on with this.

    This part handling charsets is sensitive inside the tool and after investigating this problem we are going to analyze it carefully and improve to provide a better behavior. We have your email so we will inform you about our progress and if you don’t mind in the near future we will probably provide a Beta version of the tool for you with fixes to validate our improvements.

Best Regards.

Javier Martin.

[Javier-Martin]

Hi Robin,

I don't have UE but as far as I remember there was a way to see and change the encoding of the file (something like "set encode" in the view section of the menu), can make a copy of the file a play with the copy, seeing and changing encode to the same encode the data stored in the DB has an see if you see the that data ok ?. My feeling is that those hard code texts are extracted well but appear in the wrong encode, you will probably have to modify them when moving to the target database.

Regards.

Javier.

[Robin-Feng]

Hi Javier,

I am not familiar with UE.
But I set encoding in VS code, this issue is still.

I suppose this problem is related with config id 125
"Encoding of input data, in case not detected automatically (id=125)"

And I follow the page 65 & page 138 in <SAPdbmtk-UserGuide.v3.5.2.1.pdf>, but I still can't fix this problem.

BR

Robin

[Javier-Martin]

Hi Robin, can you contact me directly via email (javier.martin@sap.com) and let' s try to solve it and then put a solution here ?. I will probably need some info from you to try to reproduce it in my test environment.

Thanks.

Javier.

[Javier-Martin]

Hi Robin,

I see you are opening the file with windows notepad and it may be due to the that client not showing the file with the right encoding. Can you try to see it with editors notepad++ or ultraedit and see what they show.

One more thing to check, in your project folder you have a subfolder called extracted, your procedure should be there too after extraction, can you check the proc in that folder (there you should the the source DB syntax) and see how it looks like ?.

Thanks.

Javier.

[Robin-Feng]

Hi Javier,

I updated this message, with re-process steps.

And this is the file look like in UE.

Robin