Why in the world have I been authorised to spend a whole year re-writing the most vital programs in our business in a SOLID way? Surely that’s a waste of money given they work already? Read on….
Part 3 – Rewriting Programs – On my way from Misery to Happiness Today
First of all I cannot do any sort of programming if I am thrown into a live volcano. I imagine that is quite often not a problem for most developers, but it I something I keep running into again and again. It is even more problematic than the other common programmer problem of having to fight men with metal teeth on top of cable cars, before you can get your changes approved to go to QA.
At the end of the last episode:-
https://blogs.sap.com/2018/01/27/back-to-the-future-2018-reloaded/
https://blogs.sap.com/2018/02/03/back-to-the-future-reloaded-part-2/
Arch-Villain Casanova Frankenstein had captured me again, and this time made me drink poison, and then dangled me upside down above a live volcano, inside a straitjacket filled with dynamite, suspended by a burning rope, with flying monkeys firing machine guns at me. He is not a big fan of SOLID OO programming, like most diehard ABAP types.
Luckily I drink so much beer that it neutralises the poison (see the antifragile references later) and that has also given me such a big beer gut that it bursts the straitjacket, hurling the dynamite at the flying monkeys, blowing them up, and the resulting shockwave blew me out of the volcano, and at that point I could use the burning rope to blow up Casanova Frankenstein’s secret base. All very easy really, especially compared to the effort involved in rewriting a monolithic procedural programme.
Points of View – Why O Why O Why?
Now we get back to the question – why would the IT management of a global multinational company approve my crazy ideas? Those ideas being rewriting procedural programs into a SOLID state, adding unit testing and so on.
Alter all, as I pointed out last time, there is enormous pressure to think in the short term, and “do not fix what is not broken”. I would say it IS broken, but it does not appear to be broken.
One of my favourite books of all time is STARK by UK comedian Ben Elton. At one point one of the hero characters (female journalist) phones one the villains to ask him why he is doing something seemingly illogical. In actual fact he is selling off his business empire at a loss so he can fund a project whereby all the evil rich people can go off the moon and leave all us poor people to die in the aftermath of the environmental disaster they have inflicted upon us.
Regardless of what you think about climate change, in the above example what he was doing would seem illogical if you did not know the reasoning behind it. Naturally he cannot tell her the real reason for what he is doing as it is (a) a secret and (b) so ludicrous nobody would believe it.
Instead he explains it thus:-
“What’s the difference between you and me? I have ideas and you do not. That is why I am a multi-millionaire and you are clearing - what – seventy grand a year?”
Board Surfing
I am just a grunt at my company, I have no staff reporting to me, and good job too, I would hate that, especially having to sack people. Nonetheless every so often I have to attend a board meeting to answer technical questions about whatever fantastic new IT thing we have come up with. That is great as the only time I am there is when there is good news to report so naturally everyone is in a very happy mood.
What I do notice in the time before I have to support a presentation or the time afterwards, is that managers who get to board level do not think like other people. It is so easy to bag them out because they earn so much money but generally no-one even considers the idea they might be worth it. If the detractors had the opportunity to see what they actually do, maybe they would think differently, or maybe not as a lot of people are just filled with jealousy.
As Australian singer Guy Sebastian once sang:-
Is it really that bad?
Really that bad
I don't think it's really that bad
Come on, get with it
If you only think of things that you haven't got
You could have it all and still never have enough
So don't worry (don't worry), be happy (be happy)
There we go – I have solved all the problems of the world. Oh dear - no I have not.
Get to the point, you Foolish Fool You
Sorry Des – the point is the CIO has understood what I was saying, something that has been shouted from the rooftops by very many developers for so many years. That is why it is difficult to get programs written. All the good developers are up on the rooftops shouting.
That is traditionally computer programs have been FRAGILE – they start off OK and then with every change they somehow “rot” and get more and more brittle until they get to the stage where it is literally too dangerous to make any sort of change to a business critical program.
This in a world where the pace of change increases every year, if not every day? This is a seemingly intractable problem. How can you square the circle?
If I cannot speak in German, and I need to learn German, I learn German. If I cannot program in JavaScript, and I need to program in JavaScript (for UI5 for example) then I learn JavaScript. If it is really dark in my house I switch the light on. There is a beer glass in front of me right now and it is full. That is a serious problem and I am going to fix it by reversing the state of the beer glass i.e. making it empty.
In every case described above I had a problem, and I solved it by reversing something – dark to light, full to empty, lack of German speaking ability to ability to speak in garbled German that no-one understands.
Above are all actual problems I have solved in real life – the one with the light switch was a tough nut to crack but I got there in the end - so why not do the same thing again? Take the problem at hand, and see if you cannot reverse the situation?
In this case the problem is changes break the program – it is FRAGILE. So I cannot change it, as every change makes it weaker, the risk is too great. What if I could reverse the polarity of the neutron flow, and so instead of every change making it weaker, every change makes it stronger?
That means instead of the ever increasing number of changes making things worse every year, the more changes you have the better off you are. Impossible I hear you cry? You would be surprised because you see the same sort of solution all around you, perhaps without seeing it.
On other words there is an elephant in the room with you. It is a hundred ton Anti Fragile Elephant.
Aunty Fragile and Uncle Fall-To-Bits
https://en.wikipedia.org/wiki/Antifragile
Something the breaks when you make a change or even look at it the wrong way, can quite reasonably described as fragile. If the program (or whatever) does not break when changed then 99% of people would say that it is the opposite of fragile i.e. robust. The author Taleb says that no, robust is not the opposite. The opposite of getting ill when you eat beetroot is not beetroot having no effect when you eat it. The opposite is that when you eat beetroot you get some sort of nutritional benefit from it i.e. you are better off as a result. In his book he phrases it thus:-
Fragile – during stressful situations at best it will be unharmed and at worst it will be smashed to pieces.
Robust - during stressful situations at best it will be unharmed and at worst it will be unharmed.
Anti-Fragile - during stressful situations at best it will be better and at worst it will be unharmed
I keep on about this in my blogs and books and I am not the only one. Here is a link to a scientific paper suggesting this “antifragile concept gets baked into computer programming”.
https://www.sciencedirect.com/science/article/pii/S1877050916302290
That article is quite technical, so to simplify let us look at this concept through our programming glasses. I this case “stress” is defined as having to change your huge fragile program to add some new functionality and running the risk of everything going pear shaped as a result.
The idea is that excepted changes are handled by the framework you have created, using the open closed principle (OCP). The OCP tries to avoid the stress in the first place, by not changing existing working code, but as we know some changes are unavoidable.
So by definition, the stressful situation is an unexpected change – one you could not have foreseen when designing the program in the first place and making a guess as to what might possibly change in the future. The only way forward is to break the OCP and make changes to the fundamental design.
Unit Tests provide the assurance your program is “unharmed” by this change, no matter how dramatic.
To summarise, unexpected changes (the stressful situation) will therefore force a redesign – but due to unit testing you know you have not broken anything, so the resulting design can handle more i.e. it still does everything it did before, and now does more, and is hopefully more future proof as it can deal with the up till now unforeseen stressful situation if (when) it happens again, so therefore the resulting program is better as a result of stress.
You’ve Been Here Before
I have said something before, in dozens of blogs. I have said this in my book, I have said it to everyone I meet when I talk about programming. What I say is this:-
“Peter Piper Picked a Piece of Pickled Pepper”
They always look at me as if I was crazy! Hang On – that’s not what I say. What I actually say to programmers is Robert Martins “Boy Scout Rule”
The Boy Scouts have a rule: "Always leave the campground cleaner than you found it." If you find a mess on the ground, you clean it up regardless of who might have made the mess. You intentionally improve the environment for the next group of campers. Actually the original form of that rule, written by Robert Stephenson Smyth Baden-Powell, the father of scouting, was "Try and leave this world a little better than you found it."
What if we followed a similar rule in our code: "Always check a module in cleaner than when you checked it out." No matter who the original author was, what if we always made some effort, no matter how small, to improve the module. What would be the result?
They still look at me as if I was crazy. Maybe I would be better off sticking to the Pickled Pepper quote. In any event Uncle Bob is making the exact same point – instead of the program getting worse with every change, you can make a conscious effort to make it better/stronger each time, provided you don’t get sacked for even daring to think about this approach (which is actually a very real concern).
SLIN Lizzy and Pragma Baker
Put yourself in this not so hypothetical situation. You have a ten million lines of code program, and you have just completed your “ticket” which in this case was to correct the spelling of an error message. That took seconds and is the closest thing to a zero risk change I can think of. I have also found it to be a rule of nature that change requests to correct spelling mistakes program output WITHOUT FAIL have a spelling mistake somewhere in the body of the request, often in the title. One BSA eve managed to spell the word “spelling” incorrectly.
Anyway that is neither here or there. Here comes the billion dollar question – do you do an extended program check and code inspector run on the program? If so, and they come up with red lights are you allowed to fix those problems? Or is the risk of you getting sacked for making an unauthorised change too great? What about the yellow lights?
I am allowed to fix them all. Moreover, we have the transport system set up so you literally cannot get a transport to QA if it has any red lights. I fix all the other errors as well (or flag the false positives with pragmas and comments so someone coming along later can see why I did not consider it an actual error).
As a result guess how much Z code broke when we did an upgrade from EHP5 to EHP8 last week? That’s right – none of it, and we are talking 17 years of non-stop Z code development here.
I wonder how many companies take the opposite approach, don’t let the developers fix SLIN errors unrelated to the change at hand, pat themselves on the back for being so risk averse, and then wonder why things go so horribly wrong at upgrade time.
Give Peace Meal a Chance
Up till now I have been talking about changing programs one tiny bit at a time, making them stronger with every change. If I can convince even one person this is a good thing I will (a) fall off my chair with shock and (b) think to myself “what a wonderful world” as someone somewhere once said.
As I keep saying I am amazingly lucky to work in a company with a forward thinking attitude which lets me do the things I describe here.
This brings me on to the main thrust of this blog, which I have not even touched upon yet.
What? WHAT? Five pages of babbling and you have not even STARTED? Truly you are the Clown Prince of Foolish Fools!
Sorry Des – my brain just does not work in a very linear manner, which is probably why I am quite good at problem solving.
In any event this year is all about re-writing too huge programs in a SOLID manner. One of them is going to ten times worse than the other, which is not say the other one will be in any sense easy.
One of them is an enormously souped up ALV report. You could say there is already a “view” which is the ALV itself. At the top are about ten million buttons and menu options, but in case the user takes an option, something happens as a result, and no matter how complicated that something is you end up looking at the ALV report again. A circular process flow that always begins and ends at the same point.
The other one is a DYNPRO program. As you are all too horribly aware during the lifetime of a DYNPRO transaction the screen flow could be described as “choose your own adventure”. There are subscreens and tabs, and header functions, and functions that get called for a line in a table control, and pop up screens where you can press another button which brings up another pop up screen and so on. It’s horrible. Well it’s not a horrible program – it does what it is supposed to do, and does it incredibly well.
So why are we going to be tying up two developers for six months plus then, re-writing such a beast, if it works already? To recap the aim of the game, it is that the program is literally crawling with conditional logic where behaviour varies by business line and country. The more countries (and business lines) we introduce the more difficult it becomes to make a change for one country without stuffing up the behaviour of another country. So a change made for Polish aggregates can stuff up Belgian concrete for example.
Why? Because everything is tied together in one horrible lump, what is known in programming circles as a “big ball of mud”. In typical DYNPRO fashion you have business logic, UI logic, database access, authority checks and so forth, all jammed together in the PBO/PAI circle of life.
We want to get to the stage where every programming unit has a single responsibility and so when you change Polish aggregates you cannot possibly break the Belgian concrete logic because that lives in a totally separate place (class), ring fenced from the changes you are making. Moreover not only can you be sure you have not broken any countries or business lines other than the one you are fiddling with, but you can be sure you haven’t broken any part of the logic of the country you are fiddling with, as that class is crawling with unit tests.
This is why I always claim that dividing responsibilities between classes and unit tests are two sides of the same coin. You can do one without the other in the same way you can wear socks without shoes and vice versa but you really need both to be able to make changes to critical programs and still be able to sleep at night.
I had to start somewhere – so I Started There
This is a very daunting task as might be imagined. It is easy to freeze like a deer in headlights just thinking about where to start. You have to have some sort of plan though not least because management are going to keep asking you for effort estimates, and amazing as it may seem they are not satisfied with answers like “How long is a piece of string?”.
I used to really worry about coming up with effort estimations (which are cost estimates really) until I read a Dilbert cartoon and saw how he had solved the problem. I cannot say the exact method here – as it is rather rude – but suffice it to say he has a magic button which he presses and then it guides him how to respond to the management request. Where would we be if we could not use magic to solve all our problems?
I used to be an accountant so the default response to any problem is to open a spreadsheet and start filling in cells. Ten minutes later I had the whole plan for the year worked out in just seven rows.
Seven Step Plan
That was easy! The only trouble is that each step is not actually that easy, and moreover every step is ten times more difficult than the one before it.
Consult and Pepper
I don’t pretend to have all the answers here, so by the end of this blog I am not going to have the whole thing working perfectly. Instead I am just going to list my thoughts thus far, and throw it open to the community to knock down my ideas and hopefully suggest something better.
This is a huge change from the consulting approach round about 1997 to 2001. In those days if I asked a consultant (not all, but some) how they got something working in SAP they would say “you don’t need to know” which would drive me up the wall and onto the ceiling again with Lionel Richie.
In other words some people used to guard their knowledge jealously in the belief they would be more secure in their employment as a result. In the same way if they did not know something they would rather die than admit it (or lord forbid, ask someone), as that would be admission that they did not know whatever it was. These were the people who were earning ten times more than the employees at the customer site, and getting charged out at twice that rate.
The world has changed dramatically since then – and with this the reputation of SAP consultants has improved accordingly- and if you want to know how to do something you go looking on the internet. You can even email the author of a textbook you have read and they actually respond. This is the good old “the more you give to the community, the more you get back from it” approach. If you have a difficult programming task then why not ask every single ABAP programmer in the world at once, and see if at least one of them has not already solved that problem? You were not actually able to do that at one stage, but now you can.
The disadvantage of course is that some people ask some really dumb things (actual SCN example was “is there an IF statement in ABAP?” or “I have just been appointed head consultant on a customer’s SAP project. What is the IMG?”) And that tends to drown out the more interesting questions. If I can spark a conversation here on the SCN about converting monolithic ABAP programs that would be wonderful, and as I said everybody would benefit.
General Approach and Private Methods
Let us go back to my “back of the envelope” list. What classes do I need?
I can think of three general categories:-
Next cab off the rank is the general rules – this is an incredibly iterative process as it is going to take a long time to get them right, but when they are progress will speed up dramatically These start off as proposals as how to deal with the problem, and then get changed or blown out of water when they encounter reality.
For example, since we are building this new improved application all out of global classes tied together with string and bogies, and DYNPRO programs revolve around screens, and you cannot call screens from within global classes, that is a problem right from the start.
I look at other languages where you create a new button by creating a new button object and passing in input parameters such as what area of what screen it lives on, and it all seems so simple. Just look at any UI5 example using JavaScript for the view. Would it really have been impossible for SAP to come up with something similar for the SAP GUI when they tried to start pushing OO programming in 2000? They say so, but I wonder. I also wonder what have convinced people more to travel down the object path. giving functional parity, or changing the name of the language so it has “object” in the name, writing that new name in big letters in SE80 and then hoping for the best?
Christmas Wrapping by the Waitresses
The standard (and only) solution is to wrap your screen inside a function module, and then wrap that function module inside a view class. Then you could, in theory easily swap between UI5, an ALV, a DYNPRO screen, a selection-screen, or WRITE statements. I didn’t mention Web Dynpro because that would be silly.
The problem of course is there is no inbuilt separation of concerns in a DYNPRO screen or even a function module, usually everything is jammed together.
So I want my view classes to call screens that do nothing at all, to all intents and purposes. Into the view is passed a structure with every single screen field on it, be it persistent or transient, that structure is passed into the function module, the screen is displayed, the users enters some data or presses a user command button, and the changed structure and/or user command flows backwards out of the function module, out of the view, back to the controller where the model can do something with it.
NB I am not saying this is the correct way to do things. It is the best way I have thought of to date, which is an entirely different kettle of fish.
I have this process running in an endless loop until such time as the user cancels, or the transaction completes, or the time has come to jump to another screen, never to return.
What I also really wanted to avoid was the code you see when you debug ME21N which it is literally impossible for anyone to understand. Pop/Push and lots of code moving things around stacks, but no code that seems to actually do anything. If you write a program this way it is literally unmaintainable because you cannot change the behaviour if there is no possible way to understand that behaviour in the first place.
A PBO/PAI process flow might break every OO rule in the book, but it is pretty easy to understand.
So how can we have our mega-cake and eat it too? That is, have a flow that is easy to follow, but sticks to the good old SOLID rules?
Cedric Soft and BOPF Bacon – did you spill my PAI?
In the latest programming model by SAP (the called RAP model) there is a layer for database access (persistency layer), a layer for the business logic (model), and a UI layer (controller/view). In this case they use CDS views, the BOPF, and UI5/Gateway, each layer initially generated by the one below it and then I hope the developer can change the higher levels to their hearts content, though I am not sure about that yet.
The BOPF transactional flow for doing CRUD type things on an entity seems very intuitive to me, and, horror of horrors, rather like a PBO/PAI screen flow.
So in my prototype I have the model doing the BOPF type things, helped by the controller as nothing can get to and from the model to the view without passing through the controller first. Before the screen comes up we have “derivations” where you can fill in values based upon others e.g. an unchangeable text description for a sales organisation, or calculated fields, or maybe initial default values.
You then have “action validations” where you hide (user command) buttons on the screen based on the current state of the model i.e. gray out the SAVE button till all mandatory fields have been filled (let us leave drafts out of it for the moment).
After the user has dealt with the screen you then have “validations” to check the data integrity, more action validations in case they have done something on screen that would disable a user command, and then the actions (user commands) themselves.
I am sure there are other things in the PBO/PAI flow that do not fit tidily into that framework, but I shall deal with them as and when I cross that bridge.
When you start trying to categorise the routines in an existing program you often find the current program has funny names for the routines, which is may be an idea to standardise in the new version.
For example, in one monolithic monster I looked at there were dozens of routines which all filled in the value of a field, either with an initial value, or based on the value of another fields. The routines encompassed the following naming conventions:-
READ
SELECT
SET
GET
CALCULATE
INIT
FUNNY_GERMAN_WORD (routine copied from SAPMV45A)
Etc…
That means, with a list of hundreds of routines, in an alphabetical list in SE80 you have to check about fifteen areas to try and find what you are looking for. Oh dear! You really need to pick one verb (method names should start with verbs generally) as they all do the same thing. DERIVE would be good if I want to stick to the BOPF model.
An Error Message from the Swedish Prime Minister
Going back to the segregation of duties, as it were, the job of the model is to see if the data entered is inconsistent, and if so then the job of the view is to tell the user this fine fact.
Therefore in my endless WHILE loop in the PAI section the model checks the data and if it finds something wrong raises an exception object containing one or more error messages, all wrapped up in an application log object. That is the application log object is an attribute of the error exception object.
The controller catches that exception, and says that we must do the loop once again. When the view is called the error log object is passed into the view. This in turn gets passed into the function module (I pass the view object instance into the function module) and in the actual DYNPRO screen flow there is a PBO module which asks the view object, do you have any error messages, and if so, up pops the application log in a popup box hovering over the actual screen. No ABAP statements inside the function module other than calls to methods of the view, and DYNPRO specific statements like MODULE and ON-EXIT-COMMAND.
This is the approach taken by both Web Dynpro and UI5 as naturally the good old ABAP “message” statement does not work too well in either of those environments.
One Million Green Bottles on the Wall – One has fallen!
I was very happy with that solution, as would be a union official. No one is doing a job someone else should be doing – what unions call “demarcation”. That concept works really well in OO programming, but in the real world caused a lot of strange problems in the UK in the nineteen seventies, which both brought the country to its knees and also spawned some great comedy films like “I’m All Right Jack” with Peter Sellars and “Carry On at Your Convenience”. My favourite was the Not the Nine O’Clock News sketch where the union refused to let management sack an employee just because he was dead.
This is pretty much as far as I have got. The next stage involves trawling through half a billion lines of code i.e. a series of PBO and PAI modules and the FORM routines they call (which are deeply nested and often call other DYNPRO screens) and seeing where each chunk of code should live in my proposed OO structure.
As I mentioned this is going to be horrible at first – each new situation (like the error handling above) is going to take some serious thought and I will probably end up like that lawyer in “A Tale of Two Cities” whose brain got so hot he had to keep sticking his head in a bucket of water to stop it from exploding. It is a far, far better programming thing I do now than I have ever done – it is to a far, far better OO place I go to than I ever been.
Take the next challenge I have – the dreaded POPUP_TO_CONFIRM confirmation prompt. These sit traditionally in the PAI logic, the entered values are checked, something dubious is discovered based upon business logic, and then an “are you sure?” prompt is presented to the user. In the new world the checking of the data based upon business logic is the job of the model, the confirmation prompt is the job of the view, and responding to the user input is most likely the job of the controller. I have thought of about ten ways to do this thus far and none of them seem very elegant. If something seems clunky it suggests there must be a better way to do it. Still, there must be confirmation prompts in Web Dynpro and UI5 and they manage to muddle through somehow so I will go looking in that area first.
The point is after error messages and confirmation prompts there will be something else which seems just as impossible to handle via the separation of concerns, and then something even worse, and then something worse yet again, and so on. Eventually there will be one so bad that when I look at the legacy code and realise what it is doing the very idea of having to work out how to re-write it in a modern way will cause to me to start screaming and never stop for the rest of my life.
Downside Abbey
That is the down side. On the B side of the record after a while my colleagues and I will start encountering problems we have already solved – we will come across the fifteenth error message or the hundredth confirmation prompt and my that stage we will know how to handle them.
No doubt there will be a very large number of situations to be handled (situations easy in DYNPRO not so easy in separation of concerns world) but not an infinite number.
I will naturally let the SCN world know from time to time how I am going, and quite possibly get down on my knees and beg for help when I come across a particularly gruesome bit of 30 year old code.
Naturally if you want to pre-empt this situation and give me some advice up front, then I am all ears….
So at this point it’s goodbye from me and its goodbye from him. Goodnight!
Cheersy Cheers
Paul
Part 3 – Rewriting Programs – On my way from Misery to Happiness Today
First of all I cannot do any sort of programming if I am thrown into a live volcano. I imagine that is quite often not a problem for most developers, but it I something I keep running into again and again. It is even more problematic than the other common programmer problem of having to fight men with metal teeth on top of cable cars, before you can get your changes approved to go to QA.
At the end of the last episode:-
https://blogs.sap.com/2018/01/27/back-to-the-future-2018-reloaded/
https://blogs.sap.com/2018/02/03/back-to-the-future-reloaded-part-2/
Arch-Villain Casanova Frankenstein had captured me again, and this time made me drink poison, and then dangled me upside down above a live volcano, inside a straitjacket filled with dynamite, suspended by a burning rope, with flying monkeys firing machine guns at me. He is not a big fan of SOLID OO programming, like most diehard ABAP types.
Luckily I drink so much beer that it neutralises the poison (see the antifragile references later) and that has also given me such a big beer gut that it bursts the straitjacket, hurling the dynamite at the flying monkeys, blowing them up, and the resulting shockwave blew me out of the volcano, and at that point I could use the burning rope to blow up Casanova Frankenstein’s secret base. All very easy really, especially compared to the effort involved in rewriting a monolithic procedural programme.
Points of View – Why O Why O Why?
Now we get back to the question – why would the IT management of a global multinational company approve my crazy ideas? Those ideas being rewriting procedural programs into a SOLID state, adding unit testing and so on.
Alter all, as I pointed out last time, there is enormous pressure to think in the short term, and “do not fix what is not broken”. I would say it IS broken, but it does not appear to be broken.
One of my favourite books of all time is STARK by UK comedian Ben Elton. At one point one of the hero characters (female journalist) phones one the villains to ask him why he is doing something seemingly illogical. In actual fact he is selling off his business empire at a loss so he can fund a project whereby all the evil rich people can go off the moon and leave all us poor people to die in the aftermath of the environmental disaster they have inflicted upon us.
Regardless of what you think about climate change, in the above example what he was doing would seem illogical if you did not know the reasoning behind it. Naturally he cannot tell her the real reason for what he is doing as it is (a) a secret and (b) so ludicrous nobody would believe it.
Instead he explains it thus:-
“What’s the difference between you and me? I have ideas and you do not. That is why I am a multi-millionaire and you are clearing - what – seventy grand a year?”
Board Surfing
I am just a grunt at my company, I have no staff reporting to me, and good job too, I would hate that, especially having to sack people. Nonetheless every so often I have to attend a board meeting to answer technical questions about whatever fantastic new IT thing we have come up with. That is great as the only time I am there is when there is good news to report so naturally everyone is in a very happy mood.
What I do notice in the time before I have to support a presentation or the time afterwards, is that managers who get to board level do not think like other people. It is so easy to bag them out because they earn so much money but generally no-one even considers the idea they might be worth it. If the detractors had the opportunity to see what they actually do, maybe they would think differently, or maybe not as a lot of people are just filled with jealousy.
As Australian singer Guy Sebastian once sang:-
Is it really that bad?
Really that bad
I don't think it's really that bad
Come on, get with it
If you only think of things that you haven't got
You could have it all and still never have enough
So don't worry (don't worry), be happy (be happy)
There we go – I have solved all the problems of the world. Oh dear - no I have not.
Get to the point, you Foolish Fool You
Sorry Des – the point is the CIO has understood what I was saying, something that has been shouted from the rooftops by very many developers for so many years. That is why it is difficult to get programs written. All the good developers are up on the rooftops shouting.
That is traditionally computer programs have been FRAGILE – they start off OK and then with every change they somehow “rot” and get more and more brittle until they get to the stage where it is literally too dangerous to make any sort of change to a business critical program.
This in a world where the pace of change increases every year, if not every day? This is a seemingly intractable problem. How can you square the circle?
If I cannot speak in German, and I need to learn German, I learn German. If I cannot program in JavaScript, and I need to program in JavaScript (for UI5 for example) then I learn JavaScript. If it is really dark in my house I switch the light on. There is a beer glass in front of me right now and it is full. That is a serious problem and I am going to fix it by reversing the state of the beer glass i.e. making it empty.
In every case described above I had a problem, and I solved it by reversing something – dark to light, full to empty, lack of German speaking ability to ability to speak in garbled German that no-one understands.
Above are all actual problems I have solved in real life – the one with the light switch was a tough nut to crack but I got there in the end - so why not do the same thing again? Take the problem at hand, and see if you cannot reverse the situation?
In this case the problem is changes break the program – it is FRAGILE. So I cannot change it, as every change makes it weaker, the risk is too great. What if I could reverse the polarity of the neutron flow, and so instead of every change making it weaker, every change makes it stronger?
That means instead of the ever increasing number of changes making things worse every year, the more changes you have the better off you are. Impossible I hear you cry? You would be surprised because you see the same sort of solution all around you, perhaps without seeing it.
On other words there is an elephant in the room with you. It is a hundred ton Anti Fragile Elephant.
Aunty Fragile and Uncle Fall-To-Bits
https://en.wikipedia.org/wiki/Antifragile
Something the breaks when you make a change or even look at it the wrong way, can quite reasonably described as fragile. If the program (or whatever) does not break when changed then 99% of people would say that it is the opposite of fragile i.e. robust. The author Taleb says that no, robust is not the opposite. The opposite of getting ill when you eat beetroot is not beetroot having no effect when you eat it. The opposite is that when you eat beetroot you get some sort of nutritional benefit from it i.e. you are better off as a result. In his book he phrases it thus:-
Fragile – during stressful situations at best it will be unharmed and at worst it will be smashed to pieces.
Robust - during stressful situations at best it will be unharmed and at worst it will be unharmed.
Anti-Fragile - during stressful situations at best it will be better and at worst it will be unharmed
I keep on about this in my blogs and books and I am not the only one. Here is a link to a scientific paper suggesting this “antifragile concept gets baked into computer programming”.
https://www.sciencedirect.com/science/article/pii/S1877050916302290
That article is quite technical, so to simplify let us look at this concept through our programming glasses. I this case “stress” is defined as having to change your huge fragile program to add some new functionality and running the risk of everything going pear shaped as a result.
The idea is that excepted changes are handled by the framework you have created, using the open closed principle (OCP). The OCP tries to avoid the stress in the first place, by not changing existing working code, but as we know some changes are unavoidable.
So by definition, the stressful situation is an unexpected change – one you could not have foreseen when designing the program in the first place and making a guess as to what might possibly change in the future. The only way forward is to break the OCP and make changes to the fundamental design.
Unit Tests provide the assurance your program is “unharmed” by this change, no matter how dramatic.
To summarise, unexpected changes (the stressful situation) will therefore force a redesign – but due to unit testing you know you have not broken anything, so the resulting design can handle more i.e. it still does everything it did before, and now does more, and is hopefully more future proof as it can deal with the up till now unforeseen stressful situation if (when) it happens again, so therefore the resulting program is better as a result of stress.
You’ve Been Here Before
I have said something before, in dozens of blogs. I have said this in my book, I have said it to everyone I meet when I talk about programming. What I say is this:-
“Peter Piper Picked a Piece of Pickled Pepper”
They always look at me as if I was crazy! Hang On – that’s not what I say. What I actually say to programmers is Robert Martins “Boy Scout Rule”
The Boy Scouts have a rule: "Always leave the campground cleaner than you found it." If you find a mess on the ground, you clean it up regardless of who might have made the mess. You intentionally improve the environment for the next group of campers. Actually the original form of that rule, written by Robert Stephenson Smyth Baden-Powell, the father of scouting, was "Try and leave this world a little better than you found it."
What if we followed a similar rule in our code: "Always check a module in cleaner than when you checked it out." No matter who the original author was, what if we always made some effort, no matter how small, to improve the module. What would be the result?
They still look at me as if I was crazy. Maybe I would be better off sticking to the Pickled Pepper quote. In any event Uncle Bob is making the exact same point – instead of the program getting worse with every change, you can make a conscious effort to make it better/stronger each time, provided you don’t get sacked for even daring to think about this approach (which is actually a very real concern).
SLIN Lizzy and Pragma Baker
Put yourself in this not so hypothetical situation. You have a ten million lines of code program, and you have just completed your “ticket” which in this case was to correct the spelling of an error message. That took seconds and is the closest thing to a zero risk change I can think of. I have also found it to be a rule of nature that change requests to correct spelling mistakes program output WITHOUT FAIL have a spelling mistake somewhere in the body of the request, often in the title. One BSA eve managed to spell the word “spelling” incorrectly.
Anyway that is neither here or there. Here comes the billion dollar question – do you do an extended program check and code inspector run on the program? If so, and they come up with red lights are you allowed to fix those problems? Or is the risk of you getting sacked for making an unauthorised change too great? What about the yellow lights?
I am allowed to fix them all. Moreover, we have the transport system set up so you literally cannot get a transport to QA if it has any red lights. I fix all the other errors as well (or flag the false positives with pragmas and comments so someone coming along later can see why I did not consider it an actual error).
As a result guess how much Z code broke when we did an upgrade from EHP5 to EHP8 last week? That’s right – none of it, and we are talking 17 years of non-stop Z code development here.
I wonder how many companies take the opposite approach, don’t let the developers fix SLIN errors unrelated to the change at hand, pat themselves on the back for being so risk averse, and then wonder why things go so horribly wrong at upgrade time.
Give Peace Meal a Chance
Up till now I have been talking about changing programs one tiny bit at a time, making them stronger with every change. If I can convince even one person this is a good thing I will (a) fall off my chair with shock and (b) think to myself “what a wonderful world” as someone somewhere once said.
As I keep saying I am amazingly lucky to work in a company with a forward thinking attitude which lets me do the things I describe here.
This brings me on to the main thrust of this blog, which I have not even touched upon yet.
What? WHAT? Five pages of babbling and you have not even STARTED? Truly you are the Clown Prince of Foolish Fools!
Sorry Des – my brain just does not work in a very linear manner, which is probably why I am quite good at problem solving.
In any event this year is all about re-writing too huge programs in a SOLID manner. One of them is going to ten times worse than the other, which is not say the other one will be in any sense easy.
One of them is an enormously souped up ALV report. You could say there is already a “view” which is the ALV itself. At the top are about ten million buttons and menu options, but in case the user takes an option, something happens as a result, and no matter how complicated that something is you end up looking at the ALV report again. A circular process flow that always begins and ends at the same point.
The other one is a DYNPRO program. As you are all too horribly aware during the lifetime of a DYNPRO transaction the screen flow could be described as “choose your own adventure”. There are subscreens and tabs, and header functions, and functions that get called for a line in a table control, and pop up screens where you can press another button which brings up another pop up screen and so on. It’s horrible. Well it’s not a horrible program – it does what it is supposed to do, and does it incredibly well.
So why are we going to be tying up two developers for six months plus then, re-writing such a beast, if it works already? To recap the aim of the game, it is that the program is literally crawling with conditional logic where behaviour varies by business line and country. The more countries (and business lines) we introduce the more difficult it becomes to make a change for one country without stuffing up the behaviour of another country. So a change made for Polish aggregates can stuff up Belgian concrete for example.
Why? Because everything is tied together in one horrible lump, what is known in programming circles as a “big ball of mud”. In typical DYNPRO fashion you have business logic, UI logic, database access, authority checks and so forth, all jammed together in the PBO/PAI circle of life.
We want to get to the stage where every programming unit has a single responsibility and so when you change Polish aggregates you cannot possibly break the Belgian concrete logic because that lives in a totally separate place (class), ring fenced from the changes you are making. Moreover not only can you be sure you have not broken any countries or business lines other than the one you are fiddling with, but you can be sure you haven’t broken any part of the logic of the country you are fiddling with, as that class is crawling with unit tests.
This is why I always claim that dividing responsibilities between classes and unit tests are two sides of the same coin. You can do one without the other in the same way you can wear socks without shoes and vice versa but you really need both to be able to make changes to critical programs and still be able to sleep at night.
I had to start somewhere – so I Started There
This is a very daunting task as might be imagined. It is easy to freeze like a deer in headlights just thinking about where to start. You have to have some sort of plan though not least because management are going to keep asking you for effort estimates, and amazing as it may seem they are not satisfied with answers like “How long is a piece of string?”.
I used to really worry about coming up with effort estimations (which are cost estimates really) until I read a Dilbert cartoon and saw how he had solved the problem. I cannot say the exact method here – as it is rather rude – but suffice it to say he has a magic button which he presses and then it guides him how to respond to the management request. Where would we be if we could not use magic to solve all our problems?
I used to be an accountant so the default response to any problem is to open a spreadsheet and start filling in cells. Ten minutes later I had the whole plan for the year worked out in just seven rows.
Seven Step Plan
That was easy! The only trouble is that each step is not actually that easy, and moreover every step is ten times more difficult than the one before it.
Consult and Pepper
I don’t pretend to have all the answers here, so by the end of this blog I am not going to have the whole thing working perfectly. Instead I am just going to list my thoughts thus far, and throw it open to the community to knock down my ideas and hopefully suggest something better.
This is a huge change from the consulting approach round about 1997 to 2001. In those days if I asked a consultant (not all, but some) how they got something working in SAP they would say “you don’t need to know” which would drive me up the wall and onto the ceiling again with Lionel Richie.
In other words some people used to guard their knowledge jealously in the belief they would be more secure in their employment as a result. In the same way if they did not know something they would rather die than admit it (or lord forbid, ask someone), as that would be admission that they did not know whatever it was. These were the people who were earning ten times more than the employees at the customer site, and getting charged out at twice that rate.
The world has changed dramatically since then – and with this the reputation of SAP consultants has improved accordingly- and if you want to know how to do something you go looking on the internet. You can even email the author of a textbook you have read and they actually respond. This is the good old “the more you give to the community, the more you get back from it” approach. If you have a difficult programming task then why not ask every single ABAP programmer in the world at once, and see if at least one of them has not already solved that problem? You were not actually able to do that at one stage, but now you can.
The disadvantage of course is that some people ask some really dumb things (actual SCN example was “is there an IF statement in ABAP?” or “I have just been appointed head consultant on a customer’s SAP project. What is the IMG?”) And that tends to drown out the more interesting questions. If I can spark a conversation here on the SCN about converting monolithic ABAP programs that would be wonderful, and as I said everybody would benefit.
General Approach and Private Methods
Let us go back to my “back of the envelope” list. What classes do I need?
I can think of three general categories:-
- Classes to break the dependencies e.g. one for database access, one for authority checks, one for locking, one for an interface to an external system and so on. I need these to enable unit testing but even if there was no such things are unit tests that will still be a good design because it follows the Single Responsibility Principle.
- One for the business logic (model) and a controller to look after the incredibly complicated screen flow. Up until now my controllers have not been doing too much, as there has only been one view, but that s all going to change. You could say instead of Thin Controllers there are no going to be Fat Controllers on the Isle of Sodor.
- We have about 30 or 40 DYNPRO screens, so does this means 30 or 40 view classes? Since they (the screens) tend to look after very different things (and thus change independently of each other) the answer is probably yes. I will no doubt find out if this is a sensible or stupid idea as I progress down this path.
- Re-usable (between applications) “business object” classes which deal with entities such as purchase orders, production orders and deliveries, and of course monsters. SAP has had many attempts at solving this problem over the years, and we are still in the situation where hundreds of different “customer” companies all come up with a version of ZCL_DELIVERY as there is no standard SAP equivalent. The BAPI experiment did not really cut the mustard.
Next cab off the rank is the general rules – this is an incredibly iterative process as it is going to take a long time to get them right, but when they are progress will speed up dramatically These start off as proposals as how to deal with the problem, and then get changed or blown out of water when they encounter reality.
For example, since we are building this new improved application all out of global classes tied together with string and bogies, and DYNPRO programs revolve around screens, and you cannot call screens from within global classes, that is a problem right from the start.
I look at other languages where you create a new button by creating a new button object and passing in input parameters such as what area of what screen it lives on, and it all seems so simple. Just look at any UI5 example using JavaScript for the view. Would it really have been impossible for SAP to come up with something similar for the SAP GUI when they tried to start pushing OO programming in 2000? They say so, but I wonder. I also wonder what have convinced people more to travel down the object path. giving functional parity, or changing the name of the language so it has “object” in the name, writing that new name in big letters in SE80 and then hoping for the best?
Christmas Wrapping by the Waitresses
The standard (and only) solution is to wrap your screen inside a function module, and then wrap that function module inside a view class. Then you could, in theory easily swap between UI5, an ALV, a DYNPRO screen, a selection-screen, or WRITE statements. I didn’t mention Web Dynpro because that would be silly.
The problem of course is there is no inbuilt separation of concerns in a DYNPRO screen or even a function module, usually everything is jammed together.
So I want my view classes to call screens that do nothing at all, to all intents and purposes. Into the view is passed a structure with every single screen field on it, be it persistent or transient, that structure is passed into the function module, the screen is displayed, the users enters some data or presses a user command button, and the changed structure and/or user command flows backwards out of the function module, out of the view, back to the controller where the model can do something with it.
NB I am not saying this is the correct way to do things. It is the best way I have thought of to date, which is an entirely different kettle of fish.
I have this process running in an endless loop until such time as the user cancels, or the transaction completes, or the time has come to jump to another screen, never to return.
What I also really wanted to avoid was the code you see when you debug ME21N which it is literally impossible for anyone to understand. Pop/Push and lots of code moving things around stacks, but no code that seems to actually do anything. If you write a program this way it is literally unmaintainable because you cannot change the behaviour if there is no possible way to understand that behaviour in the first place.
A PBO/PAI process flow might break every OO rule in the book, but it is pretty easy to understand.
So how can we have our mega-cake and eat it too? That is, have a flow that is easy to follow, but sticks to the good old SOLID rules?
Cedric Soft and BOPF Bacon – did you spill my PAI?
In the latest programming model by SAP (the called RAP model) there is a layer for database access (persistency layer), a layer for the business logic (model), and a UI layer (controller/view). In this case they use CDS views, the BOPF, and UI5/Gateway, each layer initially generated by the one below it and then I hope the developer can change the higher levels to their hearts content, though I am not sure about that yet.
The BOPF transactional flow for doing CRUD type things on an entity seems very intuitive to me, and, horror of horrors, rather like a PBO/PAI screen flow.
So in my prototype I have the model doing the BOPF type things, helped by the controller as nothing can get to and from the model to the view without passing through the controller first. Before the screen comes up we have “derivations” where you can fill in values based upon others e.g. an unchangeable text description for a sales organisation, or calculated fields, or maybe initial default values.
You then have “action validations” where you hide (user command) buttons on the screen based on the current state of the model i.e. gray out the SAVE button till all mandatory fields have been filled (let us leave drafts out of it for the moment).
After the user has dealt with the screen you then have “validations” to check the data integrity, more action validations in case they have done something on screen that would disable a user command, and then the actions (user commands) themselves.
I am sure there are other things in the PBO/PAI flow that do not fit tidily into that framework, but I shall deal with them as and when I cross that bridge.
When you start trying to categorise the routines in an existing program you often find the current program has funny names for the routines, which is may be an idea to standardise in the new version.
For example, in one monolithic monster I looked at there were dozens of routines which all filled in the value of a field, either with an initial value, or based on the value of another fields. The routines encompassed the following naming conventions:-
READ
SELECT
SET
GET
CALCULATE
INIT
FUNNY_GERMAN_WORD (routine copied from SAPMV45A)
Etc…
That means, with a list of hundreds of routines, in an alphabetical list in SE80 you have to check about fifteen areas to try and find what you are looking for. Oh dear! You really need to pick one verb (method names should start with verbs generally) as they all do the same thing. DERIVE would be good if I want to stick to the BOPF model.
An Error Message from the Swedish Prime Minister
Going back to the segregation of duties, as it were, the job of the model is to see if the data entered is inconsistent, and if so then the job of the view is to tell the user this fine fact.
Therefore in my endless WHILE loop in the PAI section the model checks the data and if it finds something wrong raises an exception object containing one or more error messages, all wrapped up in an application log object. That is the application log object is an attribute of the error exception object.
The controller catches that exception, and says that we must do the loop once again. When the view is called the error log object is passed into the view. This in turn gets passed into the function module (I pass the view object instance into the function module) and in the actual DYNPRO screen flow there is a PBO module which asks the view object, do you have any error messages, and if so, up pops the application log in a popup box hovering over the actual screen. No ABAP statements inside the function module other than calls to methods of the view, and DYNPRO specific statements like MODULE and ON-EXIT-COMMAND.
This is the approach taken by both Web Dynpro and UI5 as naturally the good old ABAP “message” statement does not work too well in either of those environments.
One Million Green Bottles on the Wall – One has fallen!
I was very happy with that solution, as would be a union official. No one is doing a job someone else should be doing – what unions call “demarcation”. That concept works really well in OO programming, but in the real world caused a lot of strange problems in the UK in the nineteen seventies, which both brought the country to its knees and also spawned some great comedy films like “I’m All Right Jack” with Peter Sellars and “Carry On at Your Convenience”. My favourite was the Not the Nine O’Clock News sketch where the union refused to let management sack an employee just because he was dead.
This is pretty much as far as I have got. The next stage involves trawling through half a billion lines of code i.e. a series of PBO and PAI modules and the FORM routines they call (which are deeply nested and often call other DYNPRO screens) and seeing where each chunk of code should live in my proposed OO structure.
As I mentioned this is going to be horrible at first – each new situation (like the error handling above) is going to take some serious thought and I will probably end up like that lawyer in “A Tale of Two Cities” whose brain got so hot he had to keep sticking his head in a bucket of water to stop it from exploding. It is a far, far better programming thing I do now than I have ever done – it is to a far, far better OO place I go to than I ever been.
Take the next challenge I have – the dreaded POPUP_TO_CONFIRM confirmation prompt. These sit traditionally in the PAI logic, the entered values are checked, something dubious is discovered based upon business logic, and then an “are you sure?” prompt is presented to the user. In the new world the checking of the data based upon business logic is the job of the model, the confirmation prompt is the job of the view, and responding to the user input is most likely the job of the controller. I have thought of about ten ways to do this thus far and none of them seem very elegant. If something seems clunky it suggests there must be a better way to do it. Still, there must be confirmation prompts in Web Dynpro and UI5 and they manage to muddle through somehow so I will go looking in that area first.
The point is after error messages and confirmation prompts there will be something else which seems just as impossible to handle via the separation of concerns, and then something even worse, and then something worse yet again, and so on. Eventually there will be one so bad that when I look at the legacy code and realise what it is doing the very idea of having to work out how to re-write it in a modern way will cause to me to start screaming and never stop for the rest of my life.
Downside Abbey
That is the down side. On the B side of the record after a while my colleagues and I will start encountering problems we have already solved – we will come across the fifteenth error message or the hundredth confirmation prompt and my that stage we will know how to handle them.
No doubt there will be a very large number of situations to be handled (situations easy in DYNPRO not so easy in separation of concerns world) but not an infinite number.
I will naturally let the SCN world know from time to time how I am going, and quite possibly get down on my knees and beg for help when I come across a particularly gruesome bit of 30 year old code.
Naturally if you want to pre-empt this situation and give me some advice up front, then I am all ears….
So at this point it’s goodbye from me and its goodbye from him. Goodnight!
Cheersy Cheers
Paul
- SAP Managed Tags:
5 Comments
| User | Count |
|---|---|
| 3 | |
| 3 | |
| 3 | |
| 2 | |
| 2 | |
| 2 | |
| 2 | |
| 1 | |
| 1 | |
| 1 |
