Back to the Future 2018 – Reloaded
Massive custom programs tend to grow and grow and mutate and become more and more fragile until they get to the stage where you just cannot change any part of them without the whole thing falling to bits like a Vampire in sunlight. How can you avoid this?
Table of Contents
- Introduction – The Wilderness Years
- Part 1 – Inheritance – The Sins of the Fathers
- Part 2 – Unit Testing – Poltergeist: The Legacy
- Part 3 – Rewriting Programs – On my way from Misery to Happiness Today
Introduction – The Wilderness Years
I work for a company which has operations in a very large number of countries. Not all are on SAP, but one day everyone will have that exquisite pleasure.
Back in 2009 it was decided that the set of programs best suited to form a basis for the entire world were to be found in the Australian system. Thus I travelled to the UK and then Germany to copy over all the Australian programs into the European system (a) for the UK initially and (b) to serve as a template for all subsequent SAP rollouts in all the other countries.
All well and good, but after I copied over the programs (which did not take very long at all, mainly thanks to SAPLINK) there then followed a two year period of changing them somewhat radically to cater for UK specific requirements.
Ironically before I even copied over one object between the systems – nine months before the event in fact at the kick-off meeting – I did a presentation in which I suggested that OO programming would be the way to go here, using subclasses to isolate country specific behaviour. That way you could keep the core program aligned between the various countries – only the UK and Australia initially.
Sadly that was not to be. Due to time constraints, and possibly no-one understanding what in the world I was on about when I started spouting off about inheritance and the like, I copied and changed the programs and we ended up with two very different versions of the same programs in two different SAP system.
In May 2012 I returned to Australia, and as I knew that back in Europe the SAP rollout would progress with a new country added every so often. Even back then I could see a big problem looming on the Horizon and so I wrote the following blog (from a bar in Moscow) explaining my concerns to the SCN community as a whole and asking for suggestions:-
Since that day I read all the OO books I could get my hands on – Clean Code by Robert Martin, Head First design Patterns, and so on, until I got my head around the concepts involved.
I wrote blogs on the SCN about everything I was discovering, and how to implement the Java examples in ABAP. I got to the stage where I was invited to write a book for SAP Press, presented at SAP TECHED and became an SAP Mentor.
During that period, on the other side of the world, the huge programs were gradually getting more and more logic added for each subsequent country, bloating and filling with more and more conditional logic and getting more and more fragile. Eventually things were going to fall off a cliff.
“Oh Well” I thought “there is nothing much I can do about this. I have outlined the problem and the solution since day one, no-one seems interested, and most likely they never will be”.
Through the Looking Glass
Then one day last year everything changed. I woke in the morning and I stepped outside, and I took a deep breath, and I got real high, and I screamed at the top of my voice “WHAT’S GOING ON?”
Now I was being asked by my managers to go round to all the programmers in our worldwide organisation and explain “my” theories (i.e. the ones I repeated from Robert Martin, the Gang of Four, Michael Feathers and so son) about inheritance and the other SOLID principles and how they could be used to solve the problems at hand.
Moreover I am allowed to work with the crème de la crème of our ABAP programmers (everyone but Yul Brynner) to re-write two of the most important business critical programs according to “my” radical new approach.
So here I am in 2018, back in Germany to do just that. It is literally a dream come true. What I am doing is to in effect put my money where my big fat mouth is, and prove that everything I have been spouting for the last nine years to anyone who will listen – on the internet in blogs, at SAP conferences via speeches and in my books – is actually true and will actually make matters violently better.
I am going to look like a complete idiot if I fail here. However since I have a (hopefully) sound grasp of the SOLID principles and my fellow programmers understand and are not resisting and – most importantly – senior management are all for this, I am feeling very upbeat about all this at the moment.
In the blog above someone said in the comments “I love it when someone goes on a journey and takes us with them”. This is what I am trying to achieve here.
I recall a “Hagar the Horrible” cartoon where he is about to lead his army on a journey of conquest and shouts out “Remember! A journey of a thousand miles starts with a single step!” Then he steps forward and falls over. Lucky Eddy says “This is going to be one hell of a journey!”
In the same way I have been laying on the floor for nine years trying to get back up and start on the journey and only really set out on it for real yesterday. And yes – I would like to take you with me!
What follows (in two or more instalments) is essentially a much updated version of the original blog, outlining the problem and how to fix it, and my initial thoughts on how to proceed, so I can ask for feedback from the (admittedly reduced) SCN community.
Warning: Since the whole exercise has been a non-stop progression from that one original blog, wishing I could do something, to finally actually be able to do it in real life, a lot of the material in this blog touches on concepts I have mentioned before in one or more prior blogs. However most of those concepts I am more than prepared to keep mentioning again and again like a broken record forever until I get some sort of evidence that the majority (or even a minority) of the ABAP community has embraced them.
Part 1 – Inheritance – The Sins of the Fathers
To recap then, what problem is it that I am trying to solve?
01 Business Problem
We have several core programs, which will be getting a constant stream of requests from different countries to make changes, fix bugs, and add functionality.
Moreover, the number of countries will steadily increase over time.
The obvious problem is how can we accommodate all those change requests in the fastest way possible, yet at the same time minimise the risk of breaking production code that already works perfectly. To put that another way, the code that is most vital to the company needs to be changed all the time, forever, with all the risk that entails.
One way around the problem would be to say this program is so vital to the business we are not going to take the risk of an outage, so we will not change it ever, it is CLOSED for modification.
That is not realistic, but as it turns out this problem has been encountered – and solved – before with the goal of making such programs both OPEN and CLOSED at the same time, rather like Schrodinger’s cat.
The Open Closed Principle proposes a seemingly impossible concept – how to make an existing program do something new and improved, something different, without changing it. That seems impossible at first glance but in fact there are many ways to achieve this parlour trick.
The Open Closed Principle is the O in the SOLID principles.
If you want to pitch this idea to someone without them laughing in your face, one way to tone it down is to describe the basic idea as avoiding changing working code whenever possible.
Literally Re-Inventing the Wheel
Amazingly as it may seem a lot of people at work seem to think I invented these SOLID type ideas all on my own and refer to this as “Pauls’ New Programming Framework”. Well it is new to them, but in real life these five principles were codified by “Uncle Bob” in the C++ review back in 2000 (about the same time SAP introduced object orientated ABAP in 4.6C) but the individual theories stretch back much farther than that e.g. the “Open/Closed” principle was first mentioned by Bertrand Myer in 1988, when I was half way through university. This is why I always laugh when people at work call this the “new” programming framework, it is new to the ABAP world but not to the 95% of programmers who program in other languages.
In any event, the slide above about the OCP says you should “program to interfaces”. Interfaces are a very difficult concept for a procedural programmer to get their head around, but as interfaces are the “fifth pillar” of object orientated programming it is vital to get to know them. It does not help that the term “interface” has many different meanings in IT.
Put simply an interface (in OO world) describes to the public a set of functionality and data that a class implementing the interface promises to provide. In my example I am going to promise that my program can carry out a particular business process.
I have used the above diagram before right at the start of my blogs series in 2012, but it is as true today as it ever was. For once the term “template” has the exact same meaning in the IT world as it does in the business world – in SAP implementations you will often hear the term “global template” meaning the default SAP system a new country gets before adjusting customising and Z code for regional variations due to “legal laws” (which are often made up on the spot by the countries to justify the way they have always done things) and market conditions.
In the IT world the so called “Template” pattern was invented in 1994 by the so called “Gang of Four”.
In the diagram above the area in the circle is the highest level of abstraction agreed upon for all countries in the group. Local variations live outside the circle.
Under this model the functionality an application offers is defined centrally via a so called “interface” and there is one main class for shared functionality, and subclasses for country specific variations.
SAP uses this approach for user exits in standard SAP code via so called “Filter BADIS”. For Z code we can do this directly using our own interface/classes.
The next question is how to structure the class hierarchy, or to put it another way decide what code goes where. Once again we have a theory to help us, let us look at the theory and how it can help us with the actual business problem at hand.
04 Single Responsibility Theory
The “Single Responsibility Principle” (the S in SOLID) was invented by best-selling IT author Robert C. Martin in 2003, though he would probably deny that and say he was just making official a concept that had already been around for ages.
You can tell when this principal is applied when a class/method has only one reason to change. If two possible changes occur that can only be achieved by changing the class, the design needs to be changed.
An example could be a subclass that deals with the rules for one country only e.g. Spain. For example the rain in Spain falls mainly on the plain, and it may well fall in different places in other countries.
05 Current Approach
This is the traditional way to account for country specific differences.
You have many blocks of almost identical code, within a single SAP “object” (Executable program in this case).
When any country changes you have to change the monolithic program. This means the program has many reasons to change.
The same is true of any business line change that spans countries. Here it is worse, as you have to change each countries block of code, and you might miss one.
Moreover as each country has a constant stream of changes the monolithic program is constantly locked as a whole. Either you do one countries change at a time, creating a huge backlog of requests waiting in an endless queue for the post office counter, or you do them all at once, and then when something goes wrong after the change hits production, try and guess which of the five hundred changes caused the problem.
Neither option is much fun. Even worse, as everything is woven together tightly like a big ball of wool that has had a run-in with a tangling factory, a change to Polish concrete can stuff up the processing of aggregates in Brazil.
So let us attack the problem from another angle – is there a way we can untangle this and have each piece of country specific or business line specific logic sitting in its own SAP artifact, being capable of changed on its own without affecting all its friends? This is where OO principles come to the rescue.
06 Inheritance Tree
This shows the OO principle of “inheritance” where you split code up such that each part has only one reason to change. Note that the outside world (calling program) does not need to know or care about all the complicated logic under the surface.
I keep harping on about the fact everyone at works thinks this is a radical new thing. In fact Object Orientated Programming was invented round about 1960. That was before I was born and by definition anything invented before I was born I do not consider new.
A USA comedian one phrased it thus – “You can’t be scared of something that has been around your whole life. Young people today have no fear of smartphones and the internet, rather like I have no fear of a refrigerator”.
Anyway, OO became the dominant programming approach worldwide around the 1990s. These days 95%+ of code written is OO in nature, due to the internet and mobile devices.
SAP embraced OO in the year 2000, and since that point the recommendation from SAP has to be to use this programming model. They even renamed the language from ABAP to “ABAP Objects”. Here we are, almost 20 years later, and the bulk of ABAP programmers remain firmly welded to procedural programming, with some people not even bothering with procedural programming and having all the code in one huge block after START-OF-SELECTION.
In the designed approach the calling program only knows the “interface” or template i.e. what functions that the called subclass provides. The details of the exact subclass being used will be unknown to the calling program.
This will be done by a technique known as the “Factory Pattern”
07 Factory Pattern 01
In a restaurant the customer orders what they want from the menu. They do not need to know the actual chef that makes the meal, only that the chef choose can do the job. The head chef decides which actual chef can best do the job.
In the same way the artist formerly known as “huge monolithic application” passes to the “factory” the fact it is dealing with a Polish concrete plant. The factory passes back the subclass that is best equipped to handle the business logic of that combination.
08 Factory Pattern 02
This IS something I invented myself.
The idea is when a new subclass is created – for a new country for example – you do not have to change either the calling program or the factory method that returns the correct subclass, thus adhering to the “open closed” principle.
I explain my initial thoughts in the following blog:-
Naturally I am never happy with anything so I have changed the design several times since then, but the basic principle is the same. The information about what responsibility the class looks after is inside the class itself, and if you pass in the correct “context” information e.g. what country the calling program is dealing with for example, and what business line, each class can decide for itself whether it is up to the job.
In my original example (in the blog above) I had the context being a table of name / value pairs, and all the logic as to whether the subclass was suitable for the provided context was done using ABAP code inside the class. In the end I decided that, whilst there is a still a place for the code based derivation of suitability, for simpler cases (which you would hope would be in the majority) where a class looks after one obvious thing (lie a sales organisation) that can be indicated by declaring a static constant in the class e.g. WELSH_SALES_ORGANISATION typed as VKORG with a value of 123.
The other change is to change the context from being tabular to being a structure so you can explicitly declare the types of the data you need the calling program to pass in.
The interface (which the subclass implements) declares a “context” which is a list of fields it requires.
The subclasses have static constants to indicate what their responsibility is.
The calling program fills the context structure and passes it to the factory which then decides the correct subclasses to get instantiated.
The ZCL_AF_FACTORY class is 100% generic, and can thus be used by any calling program, for interfaces and classes that use this context approach.
In the example above with the CASE statements I did some analysis to see what exactly it was that varied between the countries. It turned out to be steps in a business process, the process being executed when the user selected a delivery and pressed as certain button.
The exact steps in the process varied by country. Moreover there were some steps that were only executed for certain business lines, and some steps only for a combination of country and business line.
10 Process Steps
We want the code for each of the steps above e.g. “ATTORNEY” to only be in one place i.e. in one subclass. The instant you have the exact same code copied to two different classes then something is rotten in the state of Denmark.
So you have the base class where the generic steps live, subclasses which inherit from the base class and look after a country and subclasses which inherit from the base class and look after a business line. Note both sorts are only one level down the inheritance tree, directly below the base class, and the nature of their specialisation is defined by the static constant context variable they declare be it COUNTRY or BUSINESS_LINE. Most companies would use standard SAP field SPART to indicate the business line. If a subclass looked after Spanish Concrete it could either inherit from the SPAIN subclass (thus inheriting the country specific constant) and declare a business line specific constant, or vice versa.
A-ha! I hear you scream – if the code for each step is in its own subclass and if your factory is only going to return ONE subclass then how can the process at runtime execute both country specific, business line specific, and country/business line specific steps, if the code for each is in a different subclass?
Tree’s a Crowd
At the end of my “Chocolate Factory” blog mentioned above I ask that exact question – what if I need more than one subclass to get the job done? There is no multiple inheritance in ABAP.
Once again, a long long time ago in a galaxy far far away, the “Object Mentors” encountered this problem and solved it – hence we have the “decorator” pattern.
Here is a blog I wrote about implementing that pattern in ABAP, several years ago
You can think of the decorator pattern exactly like a Christmas tree – the base class is the tree itself, and the subclasses are like tinsel and glass balls, and plastic birds, and stuffed kangaroos (which we hang on the trees in Australia) and rubber chickens (which I have also seen hung on Christmas trees). You can have no decorations at all on the tree, or one or more of the decorations I described above, or all of them.
In every case what remains is fundamentally a tree, fulfilling the “Liskov Substitution Principle” as when you call the method LOOK_AT_TREE the behaviour is the same – you look at the tree – whilst the result is different depending on how many (if any) subclasses you have decorated te tree with.
At this point I cannot help but think of the famous pop song “I wear my Subclasses at Night” which seems appropriate to the decorator pattern.
So it is simply (or not so simply) a question of working out all the classes that implement the interface of your desired result instance, and then adding all the subclasses for those classes, and looping through the whole lot looking for subclasses that fulfil what the context data is looking for, and decorating the base class with all valid subclasses. The calling program THINKS it has only got one specific subclass back, but how wrong it is. What a fool that calling program is – it is such a fool that it gets pitied by Mister T.
Let’s pump up the volume, pump it up, pump it up, and consider a more complex business requirement. What if there is a step in the business process which is only executed for a group of countries e.g. the EU, or maybe for only two countries, or maybe only for three countries and only then if a certain business line is involved, and only then if it is a Tuesday and you have seen Lenny Henry during the day and the rooster has crowed three times and the hen has laid a dozen addled eggs? What then I hear you ask? It is a common enough business scenario.
In that scenario we go back to my original idea and have some complicated ABAP logic in the subclass to analyse the context data and execute the complex business logic above to see if the subclass should be used or not. The subclass could have a marker (another static constant, or the subclass could implement a marker interface, or have a specially named method) to indicate it needs special treatment and then the factory could do a dynamic call to that method.
Swing Out Sister: Break Out!
How all this works at run time is that we have the basic business process (the generic steps that all countries and business lines execute) in the base class, and each stpe is surrounded by “break out” methods, or extension points, which are the equivalent of user exits, or filter BADIs. This immediately begs the question – why not use filter BADis then? It is because of the (admittedly rare) situation where I need bonkers complicated logic to determine if a subclass can do the job or not. In addition – and this is just a personal opinion – there seems to be to be an inordinate amount of steps involved in setting up a filter BADi in one of your custom programs.
11 Template in Base Class
In the screen shot above the methods starting with SL_EXIT are redefined by the subclasses and via the decorator pattern more than one subclass can add its two cents worth to the processing of the exit. When running this in debug mode you see the full power of polymorphism in action – during runtime the name of the exact class type executing the code keeps changing.
The b side of this is one of the disadvantages of OO over procedural programming – at design time you cannot double click on a method call and see what is going to happen at run time.
You could of course do the same sort of thing using filter BADIs and if you do, then I am not going to lose any sleep over it. It may well run faster as well due to the BADI call being in the Kernel Sanders.
As always the exact mechanism is not that important and not actually worth arguing about – it is the result that is important, in this case there were two things I was looking for:-
- Code only lives in one place (the class with the single responsibility) thus adhering to the DRY principle
- When a new subclass is created I do not have to change either the factory or the calling program (thus fulfilling the “open-closed” principle).
With the approach I outline above (or something similar) then I have found what I am looking for, which puts me one up on U2.
Oh no! Just when I was getting going I have been captured by arch-villain Casanova Frankenstein who was thrown me into a vat of sulphuric acid filled with acid resistant piranhas and released a swarm of killer bees in case I try to jump out of the acid and he has an atom bomb timed to go off in ten minutes time surrounded by electrified barbed wire guarded by starving lions with laser beams coming out of their heads.
So … stay tuned for the next exciting episode in the series… will I escape from the evil villains clutches? What is this “unit testing” of which I speak? How I am going to implement all of this in real life? (If I survive)
Cue theme music…
She is disco
[Verse 1 – Jean Patrick Baptiste]
She is D, Dependency Inversion Principle
She is I, Interface Segregation Principle
She is S, Single Responsibility Principle
She is C, complicated Liskov Substitution Principle
She is O, Open Closed Principle, oh, oh, oh