Invalid characters are a common issue impacting loads in BW environment, and yet there is still a lot of confusion about what is invalid and how to effectively handle them.
On a UNICODE system with RSKC setting ALL_CAPITAL_PLUS_HEX there are 4 scenarios causing invalid characters:
1 – lowercase letters if lowercase is not flagged in the InfoObject
2 – ! as the first character in the string
3 – # as the only character in the string
4 – non-displayable HEX characters 00-1F (http://web.itu.edu.tr/~sgunduz/courses/mikroisl/ascii.html)
When it happens infrequently and these characters are not valid in the source system, a valid approach can be to manually remove the invalid characters in the PSA and continue the load. Subsequently, the team responsible for the source data can be updated in order for them to remove the invalid characters to prevent it from impacting future deltas or full loads to BW.
If the business is not motivated or required to remove the invalid characters in the source, and/or it happens repeatedly for a field then a good solution to prevent the interruptions to loads is to automatically filter out the invalid characters in the BW load itself.
This can be achieved using a function module called from a transformation field routine. Below is a template that will cover these scenarios:
FUNCTION zbw_invalid_character.
*”———————————————————————-
*”*”Local Interface:
*” IMPORTING
*” REFERENCE(I_IOBJ) TYPE RSCHABASNM OPTIONAL
*” CHANGING
*” REFERENCE(RESULT) TYPE ANY
*” EXCEPTIONS
*” INVALID_DATA_TYPE
*”———————————————————————-
FIELD-SYMBOLS: <ic> TYPE x,
<tc> TYPE c.
DATA: l_flag TYPE c,
ch1(64) TYPE x VALUE
‘000020000100200002002000030020000400200005002000060020000700200008002000090020000A0020000B0020000C0020000D0020000E0020000F002000’,
ch2(64) TYPE x VALUE
‘100020001100200012002000130020001400200015002000160020001700200018002000190020001A0020001B0020001C0020001D0020001E0020001F002000’.
CHECK result IS NOT INITIAL.
* If basic characteristic does not allow lowercase letters, perform UPPERCASE
IF I_IOBJ IS NOT INITIAL.
SELECT SINGLE b~lowercase INTO l_flag FROM rsdcha AS a INNER JOIN rsdchabas AS b
ON a~chabasnm = b~chabasnm
WHERE a~chanm = i_iobj AND
a~objvers = ‘A’ AND
b~objvers = ‘A’.
IF sy–subrc = 0 AND l_flag IS INITIAL.
TRANSLATE result TO UPPER CASE.
ENDIF.
ENDIF.
* Replace invalid non-display hexadecimal values 0000-1F00 with space
ASSIGN ch1 TO <ic> CASTING.
ASSIGN <ic> TO <tc> CASTING.
TRANSLATE result USING <tc>.
ASSIGN ch2 TO <ic> CASTING.
ASSIGN <ic> TO <tc> CASTING.
TRANSLATE result USING <tc>.
* Remove any leading spaces
SHIFT result LEFT DELETING LEADING space.
* Replace leading ! with #
IF result(1) = ‘!’ .
result(1) = ‘#’.
ENDIF.
* Eliminate the # sign if it is the only value in the string
IF result = ‘#’.
CLEAR result.
ENDIF.
ENDFUNCTION.
This can be called from the field routine as follows:
RESULT = SOURCE_FIELDS–alloc_nmbr.
CALL FUNCTION ‘ZBW_INVALID_CHARACTER’
EXPORTING
i_iobj = ‘0ALLOC_NMBR’
CHANGING
result = RESULT
EXCEPTIONS
invalid_data_type = 1
OTHERS = 2.
Thankyou Thomas for this Blog. Could you please explain why is the HEX string, separated by spaces 2000. Does that make Space an invalid Char?
Also if an IO is getting data with trailing spaces, I used Condense, but still I get invalid char error. Note I do not have Lowercase enabled for this IO.
Hi Srinivas,
Each HEX invalid character is followed by the value it should be translated to - in this case space (2000).
I would not expect trailing spaces on its own to cause an issue.
Kind Regards,
Tom
Dear Thomas Weinlick,
Nice blog, i have the below suggestions
1.The code
ASSIGN ch1 TO <ic> CASTING.
ASSIGN <ic> TO <tc> CASTING.
can be replaced as
ASSIGN ch1 TO <tc> CASTING. (because both CH1 and <ic> are of type X hence casting not needed)
2. similarly
ASSIGN TO <ic> CASTING.
ASSIGN <ic> TO <tc> CASTING.
can be replaced as
ASSIGN ch2 TO <tc> CASTING. (because both CH2 and <ic> are of type X hence casting not needed)
3. The variable ch1 contains
# # # # # # # # # # # # # # # #
as i understand the aim is to replace # with blank using "transalate" statement,so the below one should be sufficient
#
(# followed by space, so that the transalate will replace # with blank space)
Also for the code to work fine, we need to have a unicode system and RSKC must contain ALL_CAPITAL_PLUS_HEX, is that correct ?
Hi Thomas,
I like the way you solved this issues.
But are you sure that these HEX Strings in the DATA section are correct?
It didn't work for me.
But when i changed is to:
ch1(64) TYPE x VALUE '00000020000100200002002000030020000400200005002000060020000700200008002000090020000A0020000B0020000C0020000D0020000E0020000F0020',
ch2(64) TYPE x VALUE '00100020001100200012002000130020001400200015002000160020001700200018002000190020001A0020001B0020001C0020001D0020001E0020001F0020'.
Correct me when I'm wrong. But Space e.g. seems to be HEX 0020 and not 2000?!
Frank