Preamble

I was in the process of responding to Christian Drumm‘s excellent blog on Functional Programming in ABAP, when I realised that my mental ramblings should really be put into a blog (or perhaps more of a brain dump).

Please read Christian’s post for the context of the following comments…

I must say that over the last few years, I’ve somewhat lost touch with the latest language developments in ABAP, so its nice to see that in a 7.40 system, ABAP has these additional features and that they’re heading in the functional direction.

However, it does beg the obvious question though – why didn’t ABAP have these features 20 years ago?  But anyway, they’re here now, and that’s a good thing.

Real Functional Programming in ABAP?

Well, yes but not entirely.

Until ABAP is able to treat a function as a 1st Class Citizen, then only partial progress can be made towards making ABAP a true functional programming language.  For those who are not sure, a “1st Class Citizen” is any object that can be created and deleted dynamically, can be passed as a parameter to a function or subroutine, and can be passed as the return value from a function.  An ABAP function module does not meet all of these criteria.

The underlying principle of functional programming is not simply being able to use the map or reduce functions, but rather in the fact that the software is designed using the systematic application of referential transparency.

Adding various operators like COND, WHEN and LET certainly allows solutions to be programmed in a more concise, functional style; but this only permits an occasional, ad hoc use of referential transparency, rather than its systematic application.  Nonetheless, these additions to ABAP are a good step in the right direction and I’m sure people will find it very useful.

I’m sure the ABAP language developers are well aware of the need to create a unit of code that behaves as a 1st class citizen, and have already been thinking long and hard about whether and how such changes could be implemented.

Borrowing from Clojure?

Looking at the syntax and implementation concept behind the new ABAP operators like COND, WHEN and LET, they look very much like they’ve been lifted directly out of Clojure – which is certainly not a bad thing!

We just need to add operators like DEFN and FN and ABAP will have made the switch – but see the comments above about ABAP functions and 1st class citizens…

Functional vs Imperative Programming in General

I have been a long time advocate of the functional programming paradigm, and at the risk of sounding like a broken record, have been trying to convince people within not just SAP, but also the business programming world that this paradigm needs to be treated as a viable and beneficial development option.

However, the blunt fact of the matter is that there tends to be a large cultural divide between the functional and imperative programming camps.

I may use a little bit of exaggeration here, but this is only to highlight the polarities that exist between these two camps.  Each camp tends to view the other with suspicion on the basis that “we have the correct priorities” and that “they are focused on all the wrong things”.

The business programming world (the imperative camp) is focused on writing software as a means to make money.  Yes, yes, they want to solve customer problems, but in a world governed by a market driven economy, at the end of the day, the share price is king.

Over in the functional camp, this programming paradigm is presented as an implementation of Alonzo Church’s Lambda Calculus.  Here, the Platonic concept of “pure intellectual thought” reaches one of its pinnacles.  Pure functional programming languages such as Haskell have created rigorous implementations of this mathematical concept, and the result is a problem solving technique that, after you have first accepted the very restrictive modus operandi imposed by all functions needing to be pure (that is, they never have side effects), can produce solutions whose correctness can be determined with a very high degree of certainty.

Whilst there is far more to the situation than simply “making money” and “functional purity”, when you talk to people from these two camps about their differences, the conversation tends to come back to the importance of one or other of these priorities.

Why are the two camps so polarised?

Over in the Functional Camp

A large part of the polarisation here comes from the fact that many universities still teach functional programming as just an implementation of lambda calculus; therefore, if you’re not interested in mathematics, functional programming is of little use to you.

Having said that, universities are gradually seeing the need to change this approach.  I’ve spoken with Prof Kevin Hammond of St Andrew’s University in Scotland about this, and he is actively working to join up the worlds of business programming and functional programming.  However in general, functional programming is not really presented as a viable solution to the practical problems seen in the business world – such as creating warehouse stock management systems or running your accounting system.  Instead, students of functional programming are often told to write their own language interpreter that outputs an abstract syntax tree, or to design an algorithm that decides if a graph tour can be performed in which each edge is visited only once.

That’s all well and good, but it doesn’t prepare students for work in the business software world.  Consequently many students are implicitly taught to exclude this development paradigm as a viable option in the hard-nosed world of business (where success is measured solely in financial terms (Is this a good thing? Discuss))

For those software developers who do enjoy the supposed purity of mathematical thought, there is often the sense that being able to write in the functional programming style makes one intellectually superior to the lowly imperative programmers.

Well, there’s a small element of truth to this argument, but a larger element of blindness caused by intellectual snobbery.  This blindness then causes otherwise intelligent people to suggest that all business software is written by money-grubbing companies interested only in meeting their greedy revenue targets. (Yes, one functional programmer I spoke to a while back implied that SAP’s market dominance meant I worked for “The Great Satan”)

Meanwhile, back in Gotham City… uh sorry, The Imperative Camp

From the imperative side of things, this programming paradigm is easier to pick up than functional programming in the same way that the mechanical engineering solution to a problem is “visible” (you can see wheels and gears turning, drive shafts spinning and valves operating) and the electrical engineering solution to the same problem is “invisible” (you can see the resistors, capacitors and diodes on a circuit board, but you can’t see the current flowing through those components).

There also tends to be the overriding attitude that “time is money”; therefore, software must be pushed out the door as fast as possible to avoid loosing (the perception of) market share.  It is certainly true that from time to time, software companies push software out the door way too early (this is the “if it compiles, ship it” approach), but this is generally avoided.

Similarly, the business programming world must avoid the smugness of thinking that they deal with the “real” world, and dismiss the functional crowd as just a bunch of intellectual snobs who live an abstract world completely divorced from, and irrelevant to anything down here on terra firma.

Again, there’s a small element of truth here, but a larger element of blindness caused by thinking that success is measured solely in financial terms.

Can the two camps co-exist?

Yes, certainly.  And I believe the first step that must be taken is to remove the sectarian boundaries that been erected to protect the supposed rightness of each camp’s particular point of view.

Both sides can benefit from each other by exchanging techniques that they have found to be successful.

On the imperative side, the software needs to meet the customer’s needs at a price the market is willing to pay.  When a company does this well, that company makes money.  But making money is by no means the sole measure of success.

And on the functional side, the use of functional programming techniques often allows for a more concise definition of the solution. Also the correctness of that solution can typically be determined with a higher degree of certainty than is possible with the imperative approach; however, the rigour with which the policy of functional purity is applied should not override the other factors inherent to running a business.

The division between the camps is certainly less than it was 10 years ago, but it still tends to be true (at the macroscopic level at least) that when these two camps come into contact, they generally regard each other with suspicion, throw a few shallow, poorly considered insults over the fence, then as they leave, they congratulate themselves for having the “correct” approach to software development and get back to business as usual.

However, I’m happy to say that developers on the imperative side of things are starting to look more closely at the functional way of doing things, and that the universities are trying to tailor their functional programming courses to include business software as a target problem domain.

Unfortunately, there are some other factors that reinforce this cultural divide.

Functional Programming at SAP

SAP is already a highly successful software company, and none of that success is attributable to coding written using a functional programming language, or even the functional programming paradigm.  So this leads to two further problems; and I must stress that these problem are by no means unique to SAP, but apply generally to all fields where expertise has lead to success:

1. The “inertia of expertise” problem.  This is a well known phenomenon in which genuine expertise in a subject has on the one hand, produced genuine break throughs and resulted in the creation of valuable and highly successful solutions; yet on the other hand, that very success creates a large mental inertia that makes it very hard for experts to seriously consider the possibility that other, equally viable solutions may also exist.
2. The “innovator’s dilemma“.  This is also a well known problem that can be thought of as the flip side of the “inertia of expertise” problem.   This is where it is very difficult (or even impossible) to innovate in an environment within which complex problems have already been solved, and solved effectively.  Existing ABAP systems contain many highly successful solutions for solving business problems, so the overriding mindset is “if it ain’t broke, don’t fix it”.  The resistance to the use of innovative languages is compounded by the fact that the solution created by the innovative approach will often not be any better than the existing solution.  Thus the person trying to introduce a new way of solving an old problem is often viewed as someone who “doesn’t really get it” because “that problem has already been solved – move on!”.  This makes indifference to innovation appear entirely plausible.

Having said that, the functional programming language R is used within the HANA system, but at the moment there are no tools by which a customer or partner could extend or develop their own R procedures.

I strongly believe that if the business software world (SAP) is to survive the next major phase shift in computing technology (Cloud computing), then at least two things need to change.  By taking the perspective of an ABAP system, we could rather loosely describe these as “changes from within” and “changes from without”.

Changes from within (The Natural Growth Approach)

Adjust the ABAP programming language so that true functional programming can be performed.  This will require ABAP to be able to handle functions as 1st class citizens; however, since I am not an ABAP language developer, I don’t know whether this proposal is even viable.

Assuming that such a change is viable, then the syntax that has already been used by the new COND, WHEN and LET operators could be extended to create a language that looks and behaves very much like Clojure.  (Though there are other much deeper issues as stake here, particularly when handling concurrency).

However, this change (assuming its even possible) will be the slow option because it relies on the natural growth of ABAP software from within the developing system.  In realistic terms, natural growth like this takes 3 to 5 years to become an integral part of the system.

Changes from without (The Innovative Approach)

Due to the “Innovator’s Dilemma”, the adoption of functional programming as a way of life cannot take hold in any area where mature solutions already exist.  Therefore, new programming paradigms must occupy new problem domains.  This avoids the plausible accusation of trying to re-invent the wheel.

I believe the area where this approach has the most potential is in the Internet of Things.  This is a wide open field in which mature solutions do not yet exist.  Therefore, the possibilities exist to create cloud-based software based on the functional programming paradigm that can handle the huge volume of data and connections demanded by a connected world.

I further believe this paradigm shift is absolutely necessary for the following reason:

To survive the IoT, we must be highly scalable and be able to support massive levels of concurrency

Sorry to burst anyone’s bubble here, but if you think that concurrency and scalability are topics that can be handled after you’ve got your business app up and running, then you’ll be in for a very nasty surprise come go live…

Scalability can generally be solved by throwing more hardware at the problem (up to certain limits of course), but concurrency is a much thornier problem.  Concurrency poses a different type of challenge because it requires us to fundamentally change the way we write software.  But there is a natural human resistance to change – especially if we have existing solutions that have worked well in the past.  Sorry, but yesterday’s imperative solutions will not solve tomorrow’s concurrency problems.

Concurrency is of fundamental importance to any business scenario that expects to perform real-time communication with the number client devices expected with a realistic IoT solution.  These devices could either be mobile smart phones or static devices such as pieces of machinery on a production line or low energy Bluetooth beacons.

The exponential growth in computing device numbers means that any business involved in the IoT will, in the reasonably near future, have to handle in excess of a million concurrent connections as its normal runtime state, rather than as a peak-load exception.  From a database perspective, a HANA system can handle the number of queries generated by such levels of concurrency, but how about all the intermediate systems that connect the outside world with your backend HANA system?

A totally new approach is needed for the communication interface between the SAP world and the gazillions of devices all screaming for a timely response from the backend business system.  Concurrency is an area where functional programming can provide a solution with far less development effort than would be required if an imperative language were used.

Therefore, it is of the greatest importance to choose both a programming language and paradigm that will contribute towards the solution and not become part of the problem.  Such a language must not contain any unnecessary complexity, for as Tony Hoare has said:

“If our basic tool, the language in which we design
and code our programs, is also complicated,
the language itself becomes part of the problem
rather than part of its solution.”

Ignore this advice at your peril…

Therefore when writing a communication interface for the IoT, it makes perfect sense(to me at least) to choose a language in which concurrency is a natural feature, rather than an additional construct that must be implemented in that language.  Here, the obvious language choices would be either Erlang or Clojure – both of which are functional programming languages and are both very good at concurrency.

For the specific task of an IoT communication hub, my first choice would be Erlang due to the fact that this language is already battle-hardened due to having been used in the Telecoms industry for over two decades.  Among other things, Erlang was specifically designed to handle the massive levels of concurrency experienced by a telecoms switch, so this makes it an ideal candidate.

If you’re still sceptical about the commercial viability of a language like Er…whats-it-called (Erlang), then consider that WhatsApp is written entirely in Erlang and they regularly handle more than 200,000 messages per second.  See here for reviews and technical details.

Also Massively Multiplayer Online games such as the different Call of Duty variants all use Erlang servers to handle the client-to-client transfer of game state information.

Whilst the idea of using a niche language like Erlang for handling massive levels of communication concurrency might scare you, it really is the best choice for this particular problem domain.

However, if you want a more general purpose language that runs on the more familiar environment of a Java Virtual Machine, then Clojure would be a more comfortable choice.  This language is also very good at handling concurrency and since it compiles down to Java byte code, can easily be understood by experienced Java developers.

The choice to use either of the languages mentioned here (or any functional programming language for that matter – Haskell, Scala, OCaml etc) requires a willingness to move outside our comfort zone.  Generally speaking, this willingness is not always forthcoming until we have first experienced the failure that comes from thinking old solutions will solve new problems (I.E. our persistent attempts to rationalise not having to move outside our comfort zone).

The bottom line here is that the imperative programming paradigm cannot solve the concurrency issues raised by having to support massive levels of simultaneous connections.  If you don’t believe me, go away and write a solution in <pick_your_favourite_imperative_programming_language> and see how much scalability you can practically achieve, and how robust that solution actually is.  After this painful experience of blood, sweat and tears, I venture to say you will be more willing to consider a solution based on the functional programming paradigm.  😆

So what? What does this have to do with SAP software?

With the whole computing industry moving towards Cloud-based solutions, providing massively scalable, centralised communication access to your backend system/s is fast becoming a make-or-break topic.

The potential exists to use something like a Cloud Foundry Custom Buildpack to implement an IoT communication solution based on any programming language you like.  This in turn could be used to communicate with a HANA system…

The field is wide open here.  Let’s seize this opportunity to create new solutions for the new environment, and not make the mistake of thinking that our comfortable, old solutions will be suitable – they won’t.

Chris W