Skip to Content
Technical Articles
Author's profile photo Sandra Rossi

How ChatGPT answer made me lose my time

Someone had posted a blog post last month but it was removed (I don’t know why), but I had spent lot of time to verify the result of ChatGPT, so I think it’s useful to post my experience (again).

ChatGPT was queried to converting this algorithm to group anagrams to ABAP:

class Solution {
    public List<List<String>> groupAnagrams(String[] strs) {
        if (strs.length == 0) return new ArrayList();
        Map<String, List> ans = new HashMap<String, List>();
        int[] count = new int[26];
        for (String s : strs) {
            Arrays.fill(count, 0);
            for (char c : s.toCharArray()) count[c - 'a']++;

            StringBuilder sb = new StringBuilder("");
            for (int i = 0; i < 26; i++) {
                sb.append('#');
                sb.append(count[i]);
            }
            String key = sb.toString();
            if (!ans.containsKey(key)) ans.put(key, new ArrayList());
            ans.get(key).add(s);
        }
        return new ArrayList(ans.values());
    }
}

 

This was the code generated by ChatGPT:

CLASS solution DEFINITION.
  PUBLIC SECTION.
    METHODS: group_anagrams IMPORTING strs TYPE string_table
                       RETURNING VALUE(rv_anagrams) TYPE t_t_string_table.
ENDCLASS.

CLASS solution IMPLEMENTATION.
  METHOD group_anagrams.
    DATA: ans TYPE t_t_string_table,
          ca TYPE string,
          key TYPE string.
    IF lines( strs ) = 0.
      rv_anagrams = VALUE #( ).
      RETURN.
    ENDIF.
    CREATE DATA ans TYPE HASHED TABLE OF t_string_table
                      WITH UNIQUE KEY string.
    LOOP AT strs INTO ca.
      SORT ca BY character.
      key = ca.
      IF NOT ans IS INITIAL.
        INSERT VALUE #( ca ) INTO TABLE ans-values INDEX key.
      ELSE.
        INSERT VALUE #( ( ca ) ) INTO TABLE ans INDEX key.
      ENDIF.
    ENDLOOP.
    rv_anagrams = ans-values.
  ENDMETHOD.
ENDCLASS.

 

Isn’t it nice?

It’s impressive, but it doesn’t compile and the algorithm is plain wrong, lots of approximations. Maybe soon we’ll have AI connected to Java and ABAP knowledge base, but for now we get something apparently good, but needs a lot of rework. Let me explain.

 

I tried to complete the code to make it work. Before that, I created this ABAP test class which corresponds to the expected result of the algorithm (of the Java code above) and then I converted the algorithm from the Java code line by line, and made sure that the test succeeds:

CLASS solution DEFINITION.
  PUBLIC SECTION.
    TYPES: t_t_string_table TYPE STANDARD TABLE OF string_table WITH DEFAULT KEY.
    METHODS: group_anagrams IMPORTING strs               TYPE string_table
                            RETURNING VALUE(rv_anagrams) TYPE t_t_string_table.
ENDCLASS.

CLASS solution IMPLEMENTATION.
  METHOD group_anagrams.
    TYPES: ty_count TYPE STANDARD TABLE OF i WITH EMPTY KEY,
           BEGIN OF ty_ans_line,
             mapkey   TYPE string,
             mapvalue TYPE string_table,
           END OF ty_ans_line,
           ty_ans TYPE HASHED TABLE OF ty_ans_line WITH UNIQUE KEY mapkey.
    IF lines( strs ) = 0. rv_anagrams = VALUE #( ). RETURN. ENDIF.
    DATA(ans) = VALUE ty_ans( ).
    DATA(count) = VALUE ty_count( FOR x = 0 WHILE x < 26 ( ) ).
    LOOP AT strs INTO DATA(s).
      DO lines( count ) TIMES. count[ sy-index ] = 0. ENDDO.
      LOOP AT VALUE string_table( FOR off = 0 WHILE off < strlen( s ) ( substring( val = s off = off len = 1 ) ) ) INTO DATA(c).
        DATA(index) = cl_abap_conv_out_ce=>uccpi( c ) - cl_abap_conv_out_ce=>uccpi( 'a' ) + 1.
        count[ index ] = count[ index ] + 1.
      ENDLOOP.
      DATA(sb) = VALUE string_table( ).
      DO 26 TIMES.
        APPEND `#` TO sb.
        APPEND |{ count[ sy-index ] }| TO sb.
      ENDDO.
      DATA(key) = concat_lines_of( table = sb ).
      IF NOT line_exists( ans[ mapkey = key ] ). INSERT VALUE #( mapkey = key ) INTO TABLE ans. ENDIF.
      ASSIGN ans[ mapkey = key ]-mapvalue TO FIELD-SYMBOL(<mapvalue>).
      INSERT s INTO TABLE <mapvalue>.
    ENDLOOP.
    rv_anagrams = VALUE #( FOR <ans_line> IN ans ( <ans_line>-mapvalue ) ).
  ENDMETHOD.
ENDCLASS.



CLASS ltc_main DEFINITION FOR TESTING DURATION SHORT RISK LEVEL HARMLESS.
  PRIVATE SECTION.
    METHODS test FOR TESTING.
ENDCLASS.
CLASS ltc_main IMPLEMENTATION.
  METHOD test.
    DATA(words) = VALUE string_table( 
        ( `are` ) ( `bat` ) ( `ear` ) ( `code` ) ( `tab` ) ( `era` ) ).
    DATA(answer) = NEW solution( )->group_anagrams( words ).
    cl_abap_unit_assert=>assert_equals( act = answer exp = VALUE solution=>t_t_string_table(
        ( VALUE #( ( `are` ) ( `ear` ) ( `era` ) ) )
        ( VALUE #( ( `bat` ) ( `tab` ) ) )
        ( VALUE #( ( `code` ) ) ) ) ).
  ENDMETHOD.
ENDCLASS.

 

Then I wanted to see how to correct the version of ChatGPT with the fewest changes possible. That’s the result and below is the screenshot of Diff:

CLASS solution DEFINITION.
  PUBLIC SECTION.
    TYPES: t_t_string_table TYPE STANDARD TABLE OF string_table WITH DEFAULT KEY.
    METHODS: group_anagrams IMPORTING strs TYPE string_table
                       RETURNING VALUE(rv_anagrams) TYPE t_t_string_table.
ENDCLASS.

CLASS solution IMPLEMENTATION.
  METHOD group_anagrams.
    TYPES: BEGIN OF t_string_table,
             string   TYPE string,
             mapvalue TYPE string_table,
           END OF t_string_table,
           t_string_table_2 TYPE HASHED TABLE OF t_string_table WITH UNIQUE KEY string.
    DATA: ans TYPE REF TO t_string_table_2,
          key TYPE string.
    IF lines( strs ) = 0.
      rv_anagrams = VALUE #( ).
      RETURN.
    ENDIF.
    CREATE DATA ans TYPE HASHED TABLE OF t_string_table
                      WITH UNIQUE KEY string.
    LOOP AT strs INTO DATA(s).
      DATA(ca) = VALUE string_table( FOR off = 0 WHILE off < strlen( s ) ( substring( val = s off = off len = 1 ) ) ).
      SORT ca BY table_line.
      key = concat_lines_of( table = ca ).
      IF NOT line_exists( ans->*[ string = key ] ).
        INSERT VALUE #( string = key ) INTO TABLE ans->*.
      ENDIF.
      ASSIGN ans->*[ string = key ]-mapvalue TO FIELD-SYMBOL(<mapvalue>).
      INSERT s INTO TABLE <mapvalue>.
    ENDLOOP.
    rv_anagrams = VALUE #( FOR <ans_line> IN ans->* ( <ans_line>-mapvalue ) ).
  ENDMETHOD.
ENDCLASS.

Diff%20between%20ChatGPT%20code%20and%20code%20fixed%20with%20minimal%20changes

Diff between ChatGPT code and code fixed with minimal changes

What it took to me to do these fixes:

  • I had to understand the algorithm (from the java code for instance).
  • It was impossible for me to FIRST understand the crazy code generated by ChatGPT.
  • After that, I had to understand which parts of ChatGPT code correspond to which part of the algorithm, and I could find the minimal changes.

Conclusion: it took me 3x more time to find how to fix the program with minimal changes rather than writing the algorithm from scratch.

(3x is an estimation by me)

But I guess ChatGPT could be trained with ABAP code equivalent to Java code and maybe that would result in more reusable code in the future…

Possibly, ChatGPT could be used just to generate simple logic. Let’s see if anybody can propose real use cases, and not just “oh look how the code looks good” without checking “what it takes to fix the code”.

 

Assigned Tags

      8 Comments
      You must be Logged on to comment or reply to a post.
      Author's profile photo Ryan Crosby
      Ryan Crosby

      Lots of folks are sold on ChatGPT, but I'll save my money for more useful things and do it myself.  I suppose it could give you a useful answer if it "understood" what it was generating, and for that matter were not just a large language model based off scraping all the ABAP content it can from the internet (including the assumption that that content is also accurate).  How about the mountains of ABAP that do not exist anywhere in the internet realm because someone hasn't bothered to blog on it or ask a question about it?  How is it going to manage to work in that space when no content exists from which to derive an appropriate answer?

      Btw, I think the blog that you were referring to is up again because I feel like I saw it yesterday or the day before.

      Author's profile photo Sandra Rossi
      Sandra Rossi
      Blog Post Author

      It was this blog post: Page not found | SAP Blogs

      Author's profile photo Ryan Crosby
      Ryan Crosby

      It isn't this one - https://blogs.sap.com/2023/02/15/chatgpt-for-sap-developments-threat-or-an-opportunity/?url_id=text-global-profile-inbox-bp-new-in-tag-followed

      Author's profile photo Sandra Rossi
      Sandra Rossi
      Blog Post Author

      It's not this blog post. The blog post you mention, I commented it yesterday, that made me look for the old one I mentioned and saw it was missing, then I created my own blog post so that to argument my comment.

      Author's profile photo Sean Jorden
      Sean Jorden

      I, too could fill a blog with the incorrect and misleading answers ChatGPT has given me. At least for SAP technology questions, it has a long long ways to go.

      Now, if SAP hosted their own version of it, which has been properly trained by SAP experts and fed internal consulting and training documentation, plus the entire contents of SAP notes, help and KBA .. that might give us a better answer.

      SAP questions are often bigger than a simple snippet of code, and development in SAP involves putting together a LOT of different pieces.

      Plus, you know, SAP terminology can't even be readily understood by human beings 🙂

       

      Author's profile photo Matthew Billingham
      Matthew Billingham

      Fundamentally no AI can be trusted to produce good code. It's called "The Halting Problem" and was proved unsolvable by Gödel.

      It is absolutely impossible to prove that any program is without bugs.

      Given that generated code is produced by a program, it's "The Halting Problem" squared.

      Author's profile photo Sandra Rossi
      Sandra Rossi
      Blog Post Author

      Sorry but I think I am far from the required level for understanding that 😀

      Although I tried... Halting problem - Wikipedia

      Author's profile photo Matthew Billingham
      Matthew Billingham

      It's impossible write a program A that takes another program B as input and determines whether B goes into an eternal loop or not. This was proven by Gödel. A consequence of this is that it's impossible to write a program that will catch all bugs in another program. (This is why I knew when the Post Office and Fujitsu claimed their Horizon program could not be at fault, they were either lying or ignorant).

      With these AI tools, we've got a program (the AI) generating code. We know the AI may have all kinds of bugs in it - probably more likely since no one knows how it comes to its conclusions - so it seems to me that AI can only generally produce bad code.

      The number of programs with good code will be aleph-0 - the same as the number of integers. The number of programs with bad code will be aleph-1 - the same as the number of real numbers. I.e. the AI will produce vastly more bad code than good code.

      Humans are usually considered to operate at a level beyond what programs can achieve. If this is true, AI can never replace humans as programmers.