Skip to Content

Invalid Characters in Characteristic Values of Infoobjects

It’s very well known that the most errors during data load occur due to invalid characters in data inflow.

The previous my blog was dedicated to invalid characters in the texts of infoobjects. Hereinafter, all is to be said is applied to invalid characters in characteristic values only. Though, provided here information is valid for non-unicode systems, I believe that the big part of it might be applied to Unicode systems also.

It’s amazing that even some very experienced people share some delusions about invalid characters. So, here is a list of myths related to invalid characters in characteristic values.

Myth 3: The SPACE symbol is a special one and has to be added to allowed characters in RSKC (if we want to have the SPACE in characteristic values)

Not all characters are allowed in characteristic values of infoobjects. There is a predefined set of allowed characters. All notes, HowTos and help documents state that this set include the following characters (see, for example, the OSS Note #173241 – “Allowed characters in the BW System”):


!”%&'()*+,-/:;<=>?_0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZ

As you can see, there is no SPACE symbol. And I remember that there were even threads in BI forums related to inserting the SPACE in RSKC by pressing ALT-0160 or ALT-255.

Actually, the default set of allowed characters DOES INCLUDE the SPACE symbol. You can see it by yourself:

Execute SE37, type in RSKC_ALLOWED_CHAR_GET.
Double click in the statement

E_DEFAULT_CHAR = G_C_ALLOWED_CHAR

on the right part.
You’ll be brought to LRSKCTOP include.

There you can see the default allowed chars, and the SPACE is the first one:

!https://weblogs.sdn.sap.com/weblogs/images/32317/w01.JPG|height=154|alt=image|width=567|src=https://weblogs.sdn.sap.com/weblogs/images/32317/w01.JPG|border=0!

There is also a list of clearly forbidden characters: from hex code 00 to 1F.

Additionally, you may just make a little experiment – typing in some characters with spaces in characteristic value of infoobject of type CHAR:

image

You’ll see that SPACE symbol is allowed without any special settings in RSKC.

Myth 4: Presence of # sign in RSKC allows all invalid characters

Since invalid characters are often represented in BW as # sign, many beginners believe in this myth.

It’s a delusion. The system uses # character for all hexadecimal values, for which a code page does not know the appropriate character to use. This is the case for invalid characters, and hence, this sign may represent ANY invalid characters in BW messages.

The presence of this sign in RSKC will not prevent failure of data load with invalid characters. It only will allow presenting the # sign itself (hexadecimal code 23) if it is not alone in the value.

Myth 5: ALL_CAPITAL setting in RSKC along with the other special characters solves problems with invalid characters

The example:

image

It’s wrong. Put NOTHING else than “ALL_CAPITAL” in transaction RSKC, otherwise the system won’t recognize it. Only additional special characters set will be recognized. ALL_CAPITAL will be treated simply as letters A, L, _, C, P, I and T to be allowed. Though, these symbols are already permitted by default.

Myth 6: ALL_CAPITAL setting in RSKC allows all (special) characters

As you can see in the picture to Myth 3, characters with hex codes from 00 to 1F are forbidden. These very characters cause the most of problems during data load.
Remember that pressing the keys like TAB, ENTER, arrows, BACKSPACE creates exactly these not permitted codes. Hence, some validation of incoming to BW data must be made if data is created in an application where a user types in information and the application doesn’t check it.

By calling some ABAP functional modules returning l_userdef_char (user defined allowed characters) you’ll not get a set of permitted symbols in case of ALL_CAPITAL setting. You’ll simply get this very ‘ALL_CAPITAL’ string.

‘ALL_CAPITAL’ is interpreted by the other modules that check languages installed in the system.

ALL_CAPITAL_option, as it is said in the How To “Permitted Characters in Characteristic Values” allows you to use all the characters that are capitals in the local language (the language in which the batch processes run). The result is that the loading process becomes language dependent.”

Two questions arise immediately:


1. Is it possible to use lowercase letters with ALL_CAPITAL option?

There is a common belief that with this option only uppercase letters are allowed. That’s not true! If you check ‘Lowercase Letters’ for an infoobject, you can load data with lowercase letters too! Even for foreign languages installed in the system.

2. Since except ‘ALL_CAPITAL’ string you cannot set another characters in RSKC, how to allow the other special characters like #, ~ etc. (not hex 00-1F codes)?

You do not need this. ALL_CAPITAL option takes care about such symbols too!

So, as we see, ALL_CAPITAL is very powerful option and allows the maximum number of signs in BW.

It cannot manage only a small set of exceptions. For completeness of information it makes sense to provide here a quotation from the mentioned How to:

The following characteristic values are not permitted and lead to errors in the system:

  • Values that consist only of the character #. This is because blank entries in variables are marked with # (no entry = space).
  • Values that begin with the character !. The system deletes these values.
  • Control characters with the hex-display 00 to 1F (valid as of BW 3.0A if small letters were previously allowed).
  • Characters are not allowed if they are represented by a hex code that has a small letter in one of the installed languages. You could, for example, only allow the capital Ö if no language is installed where the corresponding hex code is a small letter. In this example, it would not be possible to have Russian installed, because the hex code for capital Ö means a small sch in this example.

    Special currency indicators

    The symbols for the US Dollar ($), British Pound (£), Japanese Yen (¥) and the Euro are not allowed by default.

    The symbol for the US Dollar ($) is allowed in BW customizing. The other special currency indicators lead to errors in the system.

    In the InfoObject maintenance screens or in the transfer rules, you need to create a transfer routine for any currency fields and other characteristic values that have special currency indicators. This transfer routine converts the invalid characters into valid characters or character sets. This conversion is mandatory for currency fields. The currency codes, into which you need to change the currencies, are stored in BW customizing under General Settings -> Currencies -> Check Currency Codes (or ISO4217). The key must agree with the keys in the currency tables.

    Most of the special currency indicators can be assigned to three-character currency codes. For example, the $ dollar symbol is converted into USD for US dollars or AUD for Australian dollars.

    How to catch and eliminate invalid characters?

    As we can see, the most errors during data load belong to the first three categories provided in the quotation.

    How to deal with these invalid characters? The best option is to write an ABAP code and place it into a routine in transfer rules. Here, in the forum I saw many examples of such a code.

    I somewhat like the one posted in this thread:
    Re: Invalid Characteristic data

    Though, I don’t agree with the statement that “for characteristics to be used for navigation – lower case is not permitted”.

    I’ve made an experiment:

    • created 3 infoobjects (type CHAR) with lowercase letters flag checked,
    • made one of the infoobjects as navigational attribute, and another one as a compounding attribute,
    • loaded master data with lowercase letters for this basis infoobject,
    • created an ODS (BEx relevant) and a cube with this infoobject,
    • loaded transaction data into infoproviders,
    • created 2 queries, for the ODS and the cube.

    Queries showed results without problems. Drill down by navigational attribute also worked fine. I had ‘ALL_CAPITAL’ setting in RSKC.

    I think that the main prerequisite for successful data load is a load of master data before transactional ones.

    So, in case of loading data in infoobjects with lowercase letters flag checked, there is no necessity to translate everything to upper case.

    Unfortunately, as far as I remember, all the code examples for removing invalid characters posted in the forums deal with any settings in RSKC, EXCEPT ALL_CAPITAL.

    One More Code

    The code below deals with any settings in RSKC. It doesn’t compare incoming text with permitted characters. It simply replaces by space the following invalid characters:

    • forbidden symbols with hex codes 00 – 1F;
    • strings started with ! sign;
    • strings with the only # sign.

    The code assumes that the field for which this routine is created is of CHAR type. Moreover, if lowercase letters may come from the source system, flag ‘Lowercase Letters’ should be checked for the infoobject.

    Replace ZZZ by the name of the infoobject in the transfer structure.

    
    

    FIELD-SYMBOLS: &ltic&gt TYPE x,                &lttc&gt TYPE c.

    DATA: ch1(32) TYPE x VALUE ‘00200120022003200420052006200720082009200A200B200C200D200E200F20’, ch2(32) TYPE x VALUE ‘10201120122013201420152016201720182019201A201B201C201D201E201F20’.

    RESULT = TRAN_STRUCTURE-ZZZ.

    • The only # sign is not permitted

    IF STRLEN( RESULT ) = 1.   IF RESULT(1) = ‘#’.     RESULT(1) = ‘ ‘.   ENDIF. ENDIF.

    • Exclamation mark is not permitted as a first symbol of the field
    • content

    IF RESULT(1) = ‘!’.   RESULT(1) = ‘ ‘. ENDIF.

    • Replace Invalid Characters by SPACE

    ASSIGN ch1 TO .

    • returncode <> 0 means skip this record

      RETURNCODE = 0.

    • abort <> 0 means skip whole data package !!!

      ABORT = 0.

    Unfortunately, the code will not work in Unicode system.

    Some limited solution may be created by using method CL_ABAP_CHAR_UTILITIES=>GET_SIMPLE_SPACES_FOR_CUR_CP for getting Unicode presentation of TAB, LF and CR.

    Hopefully, some ABAPers will come up with the solution for Unicode systems also.

    Resume

    The golden rule in data load in BW:

    • Use ‘ALL_CAPITAL’ option in RSKC.
    • Load master data first.

      If master data SIDs were created successfully, then problems with ODS activation and loads into cubes may arise due to:Lowercase letters. Solution: check the ‘Lowercase Letters’ for appropriate infoobject or apply
      TRANSLATE RESULT TO UPPER CASE.
      statement in a routine in transfer rules (it might be done in a formula too) if you don’t care that all texts will be in upper case only.

    Forbidden characters listed earlier. Solution: apply a routine either posted in forums or provided here.

    To report this post you need to login first.

    30 Comments

    You must be Logged on to comment or reply to a post.

    1. Dirk Herzog
      Something you really need in any system. Here is a completely compatible variant that also checks if the InfoObject allows lower case letters.

      DATA: l_d_length like sy-index,
      l_d_char type c,
      l_d_index type sy-index.
      l_d_length = strlen( result ).
      result = COMM_STRUCTURE-/BIC/ZZZ.
      DO l_d_length times.
      l_d_index = sy-index – 1.
      l_d_char = result+l_d_index(1).
      CALL FUNCTION ‘RSKC_CHAVL_OF_IOBJ_CHECK’
      IMPORTING
      I_CHAVL = l_d_char
      I_IOBJNM = ‘ZZZ’
      *” I_S_COB_PRO =
      *” I_T_COB_PRO_CMP =
      EXCEPTIONS
      CHAVL_NOT_ALLOWED = 1.
      IF SY-SUBRC <> 0.
      result+l_d_index(1) = ‘ ‘.
      ENDIF.
      ENDDO.

      Simply leave all the logic to SAP, don’t care about Unicode, etc. The only thing that is (a bit) wrong is that my code eliminates inner # even if it would pass otherwise. Not a big thing to add.

      Best regards
      Dirk

      (0) 
      1. Hi Dirk,

        It’s simply superb!

        Finally, we have a code that eliminates invalid characters!

        Thank you so much for your support and valuable addition!

        Best regards,
        Eugene

        (0) 
          1. namuduri VISHALAKSHMI
            Hi,

            i want to get this data into the cube….

            ST-FÉLICIEN-QC-CA
            QUÉBEC-QC-CA

            even i have tried RSKC but no use any conversion roune should be written for these symbols usually we get it in insert symbol in word so ……….pls send me a mail visalakshi.sapbw@gmail.com

            it’s very urgent

            thanks in advance

            vishali…

            (0) 
        1. BI Learner BI Learner
          Hi Eugene
          I am new to BI..but still I am working in BI..Could you pls help me with regard to the Invalid character..??
          Before that I want to know …NORMALY
          1)SOURCE FIELD LEN CAN BE DIFFERENT FRM OUR TARGET FIELD LEN
          2)i have an invalid character error while I am activating the Request..
          So if I use ur code..is that I can resolve for the following error..
          Error: Value ‘Not Applicable’ (hex. ‘4E6F74204170706C696361626C65’) of characteristic ZANS_LAB contains invalid characters
          Error: Value ‘Eda Lam’ (hex. ‘456461204C616D’) of characteristic ZPENAME contains invalid characters
          Could u pls revert me ASAP..
          Thanx in advance
          (0) 
      2. BI Learner BI Learner
        Hi Dirk
        I am new to BI..but still I am working in BI..Could you pls help me with regard to the Invalid character..??
        Before that I want to know …NORMALY
        1)SOURCE FIELD LEN CAN BE DIFFERENT FRM OUR TARGET FIELD LEN
        2)i have an invalid character error while I am activating the Request..
        So if I use ur code..is that I can resolve for the following error..          
        Error:     Value ‘Not Applicable’ (hex. ‘4E6F74204170706C696361626C65’) of characteristic ZANS_LAB contains invalid characters     
        Error:     Value ‘Eda Lam’ (hex. ‘456461204C616D’) of characteristic ZPENAME contains invalid characters     
        Could u pls revert me ASAP..
        Thanx in advance
        (0) 
      3. Iris Siebrands
        Hi Dirk,

        thanks for that really comfortable solution! Since I’ve been a bit busy with simply trying to find this essential peace of information – 2 years after it had been posted – I’d very much appreciate if SAP could include this central and basic information e.g. in the standard online SAP-library – for example where the general settings for characteristics are described.

        Thanks to all of you for your troubles to help the SAP developer community …  πŸ˜‰

        Iris

        (0) 
    2. Ian Curtis
      Hi Eugene,

      I have been using this setting in RSKC since November 2000 in all BW installations, but it is good to know that all my recommendations of using it were true. I now know more about it and can really justify it’s inclusion in RSKC.

      Thanks

      Ian

      (0) 
      1. Hi Ian,

        Do you mean ALL_CAPITAL?

        The most of information provided in the blog is of experimental/experience nature (in the docs and help you may find just some peaces of truth). I don’t exclude that some of the statements might wrong in some particular circumstances.

        So, I welcome any feedback on the matters (either confirming or the opposite).

        Best regards,
        Eugene

        (0) 
    3. Pedro Pais
      I’ve tried to use Dirk’s code in a transfer rule and had to make a small correction so I thought I’d let you know.

      Instead of entering directly the infoobject’s name to the function module I had to declare a 30 character string, assign it with the infoobject’s name and use that string when calling the function.

      Thank you for your help.

      (0) 
      1. Xiuhai Yue

        Fix ‘0200’ bug: new code:<br/>–


        <br/>FIELD-SYMBOLS: <ic> TYPE x.<br/><br/>DATA:<br/>ch1(64) TYPE x VALUE<br/>’00000020000100200002002000030020000400200005002000060020000700200008002000090020000A0020000B0020000C0020000D0020000E0020000F0020′,<br/>ch2(64) TYPE x VALUE<br/>’00100020001100200012002000130020001400200015002000160020001700200018002000190020001A0020001B0020001C0020001D0020001E0020001F0020′.<br/><br/><br/>DATA:<br/>   x8(4) type x,<br/>   x0(2) type x VALUE ‘0020’,<br/>   x0200(2) type x value ‘0200’.<br/><br/>RESULT = TRAN_STRUCTURE-ZZZ.<br/><br/>* # sign<br/>IF STRLEN( RESULT ) = 1.<br/> IF RESULT(1) = ‘#’.<br/>   RESULT(1) = ‘ ‘.<br/> ENDIF.<br/>ENDIF.<br/><br/>* ! sign<br/>IF RESULT(1) = ‘!’.<br/>  RESULT(1) = ‘ ‘.<br/>ENDIF.<br/><br/>* 0000-001F

        (0) 
      2. Sherman Ames
        I tried the above code to remove invalid data and it did not work.

        #N#6.75″ X 2″, 48VDC, THERMISTOR, REMOTE

        The above character # is 0001 and 0002.

        (0) 
    4. Dubravko Cunovic
      Shouldn’t it be included in some control on R3 system – that user can not enter invalid characters in texts? It will solve 90% of problems like that!

      Thanks for great blog

      (0) 
      1. HS Kok

        Unfortunately a lot of SAP BI/BI consultants are only concerned with fixing the problem at the surface level, instead of addressing it at the root cause.

         

        Invalid characters are usually caused by bad entry of data in the source system (ECC, or non-SAP sources). These must be fixed in the source system, instead of patching the character values in BI, because:

         

        1) Data mismatch: You won’t be able to match up the values between BW and the source systems, if you start patching up the illegal characters.

         

        (You might as well be transforming every single $100-sales order into $1,000 sales order to mark up your reported earnings!)

         

        2) Audit: As mentioned in #1, this will be a cause of concern.

         

        3) Future support: If the data from the source system is loaded into multiple DSOs or cubes, are you going to add the function module and call it in each and every transformation involving the affected InfoObject/field?

         

        #3 is usually the reason why most consultants are not bothered by this issue – because they never stick around the client’s company long enough to see things in the long-term (support) perspective.

        (0) 
    5. D Nowley
      These days I just search for help on Blog and not in forums.Thanks Eugene ! Your blogs have been of great use earlier as well ! πŸ™‚
      (0) 
    6. Asrar Ahamed
      Hi Eugene,
      Thanks for your wonderful blog which cleared so many doubts. But i am still facing one issue even after enablling ALL_CAPITAL in RSKC. In ECC some texts are maintained independent of languages as this represent the name of a person(Data source 0RESPNO_TEXT). The text description includes special characters like ŠšΕΎè..
      These are not working in BW. I am getting # in the place all these characters.
      Please help me ho to resolve this issue.

      Thanks & Regards,
      Asrar Ahamed MA

      (0) 
    7. Arun Varadarajan
      Eugene,
      I have used the code given by Dirk and some local changes in my system.. however I have a unique problem .. I have a file that has Chinese , Korean , Japanese etc and I am having problems in loading this file. Do I have to :
      2. Log into the system in Chinese or Japanese and load the file ( The file comes in to the app server and I would eventually want to load the same through a process chain )
      3. Is there anything I can do in terms of settings to be able to load this file while logged into the system in good old English..?

      Thanks ,
      Arun Varadarajan

      (0) 
        1. Arun Varadarajan
          Thanks Eugene for the response…

          We are trying to see if installing the language packs will help. Once this is working , I should be able to put together a nice blog on the same…

          (0) 

    Leave a Reply