Character Encoding Handled the right way
In our daily business we have to handle different types of data-sources and data-sinks. Most of these data-source and data-sinks are UTF-8 aware already. But what if one of these is does not provide data in UTF-8 or does not understand it at all?
The PI/PO is only handling UTF-8-Data so how is it possible within the PI to convert the Encoding of a File/Message?
In one of our current scenarios we have to handle fixed-length files, which are read from a SFTP server.
(But this guide should also be valid for other scenarios)
The configuration is simple, but not obvious since the behaviour of the MessageTransformBean is a little bit strange 😉
Here is what you have to do in the Module-Chain of your channel if you want to convert a message from a supported encoding into UTF-8.
Processing Sequence
| Number | Module Name | Type | Module Key |
|---|---|---|---|
| 1 | AF_Modules/MessageTransformBean | Local Enterprise Bean | Codepage |
| 2 | AF_Modules/TextCodepageConversionBean | Local Enterprise Bean | Convert |
| 3 | AF_Modules/MessageTransformBean | Local Enterprise Bean | Transform |
| 4 | localejbs/CallSapAdapter | Local Enterprise Bean | entry |
Module Configuration
| Module Key | Parameter Name | Parameter Value |
|---|---|---|
| Codepage | Transform.ContentType | text/plain;charset=ISO-8859-1 |
| Convert | Conversion.charset | utf-8 |
| Transform | Transform.Class | com.sap.aii.messaging.adapter.Conversion |
| Transform | xml.conversionType | SimplePlain2XML |
| Transform | xml.documentName | Foo |
| Transform | xml.documentNamespace | http://example.com |
| Transform | xml.fieldFixedLength | 1,2,3 |
| Transform | xml.fieldNames | ID, Bar, Baz |
| Transform | xml.processFieldNames | fromConfiguration |
| Transform | xml.structureTitle | Bar |
The key in this configuration is to use two different instance of the MessageTransformBean. The first one only sets the codepage, and mime-type, which is then converted by the TextCodepageConversionBean. The sencond one parses the (now utf-8) text into xml.
These two steps are necessary since the class “com.sap.aii.messaging.adapter.Conversion” expects its data as utf-8 which is not right if the incomming codepage is not utf-8 but an other one.
Thanks Thorsten 😉
Thanks Thorsten.
I have implemented TextCodepageConversionBean in the receiver side before, where it converted utf-8 to specified charset after MTB.
I understand it is for the sender case here, but just wonder do we have to add an MTB to specify the incoming charset? in other words, will the TextCodepageConversionBean fail without an MTB to specify the incoming charset beforehand?
Hi,
the PI declares the charset always as UTF-8 so you have not specify it in the MTB. You only need place the TextCodepage-Bean after the MTB and specify the target charset.
So you do not need, or better, you should not specify a charset in the MTB in your case 😉 . Simply use the default UTF-8 charset, as defined by the PI/PO.
--
Kind Regrads
Thorsten
Thanks.
I tryed, and now it works 😆
Hi Alex,
I am getting below error in RECEIVER REST adapter audit logs, not sure what is causing this error. I believe it is some invalid character in the response. I am trying to do a JSON to XML conversion in response.
Information Server returned code: 200
MP: exception caught with cause java.lang.RuntimeException:
com.ctc.wstx.exc.WstxIOException: Invalid white space character (0x8) in text to output (in xml 1.1, could output as a character entity)
Exception caught by adapter framework: com.ctc.wstx.exc.WstxIOException: Invalid white space character (0x8) in text to output (in xml 1.1, could output as a character entity)
Transmitting the message using connection JPR failed, due to: com.sap.engine.interfaces.messaging.api.exception.MessagingException: java.lang.RuntimeException: com.ctc.wstx.exc.WstxIOException: Invalid white space character (0x8) in text to output (in xml 1.1, could output as a character entity)
As this message is getting stuck in receiver rest adapter on synchronous response side I believe I would either need to write a adapter module or change the code page to ISO Latin/ UTF -16 ?
I tried to put the data format of response json to cp1252 but still same error.
Thx
Ravijeet
If I am handling a xml of boeing aircraft having 100000 bom components,will I provide xml.fieldNames = "....".
Who is this insane person who has made this module to let others weep.
Tested with SFTP Sender Adapter (FCC) and is not working. conversion from ANSI to utf-8 fails in some characters..
PO 7.5
Hi ILIE ALEMAN,
maybe SAP Note 2544233 - Support for non UTF-8 and special characters in SFTP Adapter can help you.
Kind regards
Jo
In my case it was an issue about the adapter version, depending on that some of the modules are not working well. The solution was to add advanced parameter: encodingFormat and value ISO-1559-1 to the channel, but this also depends on the adapter patch level.
Regards
Hi Ilie,
I have the same problem. Where did you put the encodingFormat in Advanced?