Why Integration is Easier and Harder than Ever Before
A few days ago, I had a conversation about the challenges ABAPers are facing and about current trends with respect to these challenges and the skills developers need to meet them. The person I was talking to mentioned that integration was specifically brought up as a key issue: Apparently, it is something that developers struggle with and that poses them great difficulties. My conversation partner asked why integration was more difficult today than it used to be.
My response was, it is so hard because it has never been easier. In a nutshell, when integrating systems, user expectations set the bar higher than ever before.
“My god, it’s full of silos!”
In the past, users understood that each IT system was a silo:
- Their login would be “I30540003” in on system, “MILLERTHEO” in another, “TMILLER0001” in the third system, etc.
- Each system had its own client software that you would have to launch from your desktop. SAPGUI here, a Borland Delphi windows client executable there, a Telnet TN3270 emulation connecting to an IBM host system there. Maybe not even functioning copy-paste between windows.
- Each system would have its own database, with long time spans between synchronization: A nightly file download and upload here, weekly synch there. A change in one system would show up in another systems days later.
Today, when using two different pieces of software that are at least remotely related, what we expect is roughly this:
- Real-time backend data integration: All data must be perfectly consistent with other data we consume from other related sources, in real-time.
- Comfortable single sign-on: It’s not user-friendly to require a user to logon as long as there is a valid logon with some other system which could vouch for them.
- Use identities from other system: Why should I create usernames and passwords with new systems if they could just accept my Facebook/Google/Windows domain identity and credentials?
- Cross-system personalization: Why should I have to teach two or three systems the same about what I like and what I don’t? Can’t they inform each other about these things?
- Seamless navigation: If a screen flow takes me from one system to the next, it should be a seamless and consistent navigation flow with, at the very least, properly functioning back and forth navigation, and ideally a consistent look and feel, screen layout patterns, icon language, etc.
- Consistent process flow and monitoring: Flows across multiple systems should be as robust and easy to monitor as flows within a single system. If something goes wrong, I want to know what went wrong and how, and how to get it going again.
From the point of view of an architect or software developer, there are even more requirements:
- Architectures should be designed so that there is the least possible amount of redundant development. Business logic, screen fragments, and data structures that are needed in two different applications (or IT systems) should somehow be built once and reused wherever else required. It doesn’t matter if one environment is a native iPhone application, a second environment is an offline HTML5 page running in a web browser, and a third environment is an ABAP background processing a nightly job – people already become nervous when even a trivial chunk of business logic such as “birthdate must not be initial” is implemented redundantly across these platforms.
- As the functionalities required by a single end-to-end process are spread across multiple systems, we have to find solutions in case one of these systems is temporarily down. So we have to use mechanisms like message queues, make sure that messages are sent exactly ones or that a web service that handles them is idempotent, we have to be able to monitor the message flow associated with a business process and track and correct errors, and maybe our process flows need to have a Plan B in case a required subsystem is down so we can tell the user: “Your entry has been submitted and is pending approval, you will get a notification as soon as it has been processed,” instead of giving them the usual instantaneous feedback.
- Applications should be maintenance-friendly: Even though the steps of an end-to-end process are spread across multiple systems and platforms, changes in one platform should not require adaptations in the other systems (or break them).
Fig.: The bar is high
Who raised the bar?
Putting it all together, we can see that the bar with respect to seamless frontend and backend integration is infinitely higher today than it used to be. But why? The answer lies in another question and answer: Why did the mountaineer climb the mountain? – Because it exists. Why did the architect create a tight coupling between the systems? – Because there’s a way to do it.
From silos to a service web
The first driver is SOA: In enterprise IT, the siloed landscapes were shaken up when SOA came along and it was possible to call functions provided by one system from another. This led to a reduction in redundancy: Functions that used to have many implementations on different platforms in a company’s application landscape were reduced to one implementation, which can be called as a web service by other systems. This possibility has changed people’s thinking and the trend has just been going on and on.
Internet experience as a driver of change
The second driver is the user experience people have with Google and Facebook. Think about how many Google sites you use: Out of Calendar, Gmail, Search, News, Drive, Music, Movies, Maps, there are certainly going to be a few you will use on a regular basis. They all log you on transparently and automatically, without asking for a password or username, and you never have to tell them anything twice. I don’t even remember my Google password, but I use my Google account on a half dozen sites almost daily.
It’s similar with Facebook. I don’t use any sub-sites of Facebook but whenever a new non-Facebook site I want to use gives me the choice between “Sign up” (create a new username and password combination that tortures me from now on) and “Log in with Facebook,” I certainly know what I do. Facebook offers great value to me, and many other users, as a leading identity provider for third-party web sites.
Users take these expectations to work and demand that the enterprise IT systems they work with live up to the same standards of simplicity and user-friendliness. They want to have as few identities as possible, and be able to use them with as many systems as possible.
Availability of standards
Getting back to the mountaineer who climbed the mountain because it existed – the availability of wide-spread IT standards for integration put us in a similar position. Single sign-on between your new custom system and an existing system running anywhere is possible thanks to open standards – and so it is seriously considered and added to your new project’s backlist.
Near real-time data replication between systems is easy to implement with SAP LT Replication, and so it is considered and possibly added to your project’s backlog.
The same goes for other features such as sophisticated message queuing while a system is down, provisioning data and functions to mobile clients, secure login for mobile devices through the internet, navigation between cloud-based and on-premise system, seamless user integration across companies (e.g. employees of other companies whose identities are maintained in that other company’s system signing on to your partner-facing portal, with single sign-on from their Windows domain).
New state of the art
The bottom line is that the state of the art has changed greatly, and the level of integration that was normal five years doesn’t cut it anymore today. An architect or developer who is tasked with creating that type of integration may find that they have to read up on internet standards, and get their feet wet learning how to actually implement use them in a real system.
This, I believe, explains why developers are struggling with integration now more than ever before. There may have been a time when IT departments were in the driver’s seat – but now we live in a consumerized enterprise IT world. New tools exist, and users demand that we use them. We have to learn how and do it.
...and now that the relatively 'simple' low-level technical integration points have been sorted out, these higher-level integration points bring about a whole different set of often more difficult integration challenges. Previously we struggled to integrate applications at the wire-level protocols - Java RMIs vs .Net remoting; EBCDIC vs ASCII - or at the level of the fundamentals such as authentication mechanisms. Now that these issues are largely "solved" with a handful of extremely popular protocols like OAuth, SOAP, et. al., the integration problems have moved further up the stack. Arguably these are the same fundamental problems as before - how to serialise and de-serialise data - but rather than talking about doing this to strings as in the example of on-the-wire protocols, we are now wrestling to doing this to entire data models. The example of Successfactors comes to mind; its data model of an org chart has evolved over years (decades?) independently of SAP's data model of the same concept. As a result, integrating these remains a non-trivial task!
Sometimes it feels as though we're making less progress on solving these issues, but instead simply moving the complexity elsewhere or discovering a new domain whose complexity must be addressed by our systems...
Exactly! And don't get me started on the complexity that occurs when technical users are replaced by user ids that represent actual people. A flat file download-upload integration typically uses a technical, impersonal users with god-like authorizations (SAP_ALL, SAP_NEW) that can do anything in the system. When the file upload and batch processing is replaced by a web service endpoint, you may realize that a) this represents a security loophole and b) it's insufficient for audit purposes to have user id like WEBSERVICE in all the audit logs, with no clue as to which person actually changed the record.
So you start looking into mechanisms for principal propagation, where the user in the target system is determined by the user in the source system. And for this you need a way to map the users - and it gets complicated quickly.
I don't think we're solving any of these issues, because it seems to me like we're diving deeper and deeper in a fractal landscape of technical complexity - but it has to be noted that as we move the complexity from one level to the next, the system to get better, more transpararent, more user-friendly, etc., as long as we stay anywhere near the state of the art. 😉
I wonder whether there is another, rather non-technical meta-motivation behind your observation: The widespread availability and ease-of-use of visual development environments that make it easy for anyone to create what looks like a brilliant application on the surface. "Anyone" in this case certainly includes the "real developers" who use these tools the way they are designed to be used and with the complexity of the entire system landscape in mind. Sadly, it also includes many "wannabe developers" (in my LOB, that's usually the brillant nephew of the head physician who is so far ahead of the school curriculum that his marks just HAVE to be that bad) who claim that they can solve any problem on a rainy saturday afternoon. They produce a shiny UI with little less than coded unmantainability underneath, and unfortunately, management tends to put rather too much trust in these people, especially since they are faster in producing visible results than the experienced in-house IT or partners. This creates another - and very frustrating - kind of pressure.
integration at another level in the stack, indeed. One phrase leapt out at me and made me smile:
"An architect or developer who is tasked with creating that type of integration my find that they have to read up on internet standards"
At least these days it's likely that the answer will be found in the non-proprietary section of the software and protocol library.
I'm not sure, if user expect the same experience like facebook, but in all other sights I'm with you.
I think today is a great time to be an ABAPer because of the ecosystem is now changing and we are the people writing this story! Title: How the (SAP-)business transformed (no, I'm not using the word).
Being someone who has never written a single line of ABAP code, and instead having a background on user-focused SAP Portals, I may have been lucky to almost never experienced IT departments sitting in the driver's seat.
Nevertheless, with all current -- and genuinely valid -- user expectations towards (enterprise) software with regards to tooling, standards, integration and UI capabilities, I am also jumping up and down from pure excitement of all things now possible that weren't 5-10 years ago.
At the same time, with so many possibilities, there are an exponential more ways to do things wrong -- ranging from 'slightly harder to maintain' to 'extremely hazardous' -- and I believe we should educate / be educated the do's and don'ts. This implicitly means looking beyond SAP, and learn from the greater development / integration / UX community outside SAP
Great blog, Thorsten, as usual.
Interestingly, sometimes we find challenges in our own IT "silos". E.g. we've been struggling to improve the performance of SAP interface with a homegrown application. But the team responsible for that application doesn't seem to be very enthusiastic about making any significant changes. So the most difficult part of the system integration might be integrating humans. 🙂