Skip to Content
Technical Articles
Author's profile photo Werner Dähn

The singularity event of ERPs

During my days as SAP employee I was responsible for various data integration solutions around the ERP system, today I am propagating its next evolution.

Recap

ERP systems aim to be a digital twin of the reality. When a product is shipped to a customer in the physical world, the ERP’s shipping document represents the data of this action. In the pursuit towards excellence, users of the ERP system have the natural tendency to add more detailed information, e.g. by customizations to store the additional data.

The next logical step is to integrate external information as well, for example the shipping status. No rocket science to call the shipping agent’s API and to retrieve the delivery status. But by adding more data, the complexity and hence the total costs of ownership of the ERP system increases exponentially.

Singularity

To be frank, this does not scale. The various systems and modules should rather look and feel like one system. A tightly coupled choreography working in harmony – the singularity.

File:Naked.Singularity,Overextremal.Kerr.Newman,Raytracing.png - Wikimedia Commons

Integration of processes is too rigid for that. If all works like clockwork the system is fine. But if reality throws a spanner into the works, the predefined processes fall apart leaving a cascade of failures in its wake.

We should rather change the way we perceive ERPs:

  • A chain of integrated processes → Network
  • Reality represented as data structures → software makes decisions which impact the physical world
  • Managing the business in tacts → Realtime business
  • Simple if-this-than-that business logic → considering a lot of related information

Looking at these changes, many SAP themes can be found, e.g. “Hana is the platform for realtime business”, “Automation”, “Predictive”, etc. In the area of integration customers can do better.

API first? Customer first!

The phrase API first means that there is a well defined interface others can use. If there is an API for everything, anything is possible. This assumption, catchy as it is, has multiple obvious issues.

  1. How many APIs does the SAP API Business Hub expose for S/4Hana? As of today it is 290, a fraction of the functions an ECC system has. The obvious interfaces exist, like creating a sales order, but it is far from complete.
  2. Every API is rather heavy weight, which has pros and cons. On the plus side it is powerful and many aspects of e.g. a sales order can be changed, but a single field update like setting a status flag, has lots of internal overhead.
  3. When is an API perfectly well defined? If it provides all the required fields, including customer specific and caller specific data. Or putting it differently: Never.

Things get even more nasty when APIs depend on each other. A Sales order application must know the list of current materials, the user picks some, for each the available quantity must be queried, production orders are created and much more. The advantage of the ECC System is that all is done in one database, and coupling these deeply nested processes is well supported by database transactions. With external systems things become much more complicated and, in consequence, unreliable, thus expensive.

see microservice dependency hell

File:Social Network Analysis Visualization.png - Wikimedia Commons

The ONE project to harmonize the data models makes the transformations between the various data structures easy – there are no transformations necessary if all systems use the same structures. It is an expensive option but as these costs are covered by SAP, no complaints here. It would not be required for the proposed solution but helps in those cases where the ONE data model works perfectly.

From a state machine…

The key reason of the integration drama is how the ERP systems was intended to be used: As a black box. Somebody performs an action: create sales order. The ERP system sets multiple database changes in motion and when completed, returns. Only a few methods exist (and are used even less frequently) that allow external actions on internal state changes.

In a world where the central ERP is just that, the core system to govern the business, there is no need for such things. But these times are long gone. The rules are no longer just predefined static workflows. The ERP is one of many systems, the most important one, but still just one of many that need to collaborate.

Collaboration means communication.

…to a team player

Let me make a suggestion:

  • Instead of providing many different APIs for everything and consumers must call each, let the ERP system communicate all state changes to the outside world proactively instead. People can listen to that stream or an interesting subsets, but it is the consumer’s choice.
  • Instead of providing attributes the developer of the API deemed to be useful, it should communicate everything.
  • Instead of informing that there was a change, provide the change.

This would look and feel for other consumers like a social media stream: “Hey, customerX just bought 100 pieces of productY!”. For humans this stream would be overwhelming but consuming it with software services, maybe even combining data from multiple streams, does enable powerful applications with little effort.

Feasibility and costs

The requirement to stay informed at all times is a direct effect of today’s society and technology determinism. Both pointing to the same outcome – communication networks. In other words, it is expected by customers and the technology to enable that has been developed – in the Big Data world.

For Big Data, the volumes of changes created per second in even the largest ERP systems is laughable compared to what companies like Netflix, LinkedIn an others produce in the same time interval. The requirements have a slightly different emphasis, e.g. consistency of data gets more important but can be fulfilled.

At the end, using an ERP system as described as a communication hub is not only more powerful but also cheaper to implement than even a single process integration step.

Technically speaking, the solution does capture all changes of all S/4Hana data structures, move them into Apache Kafka and is available for everybody interested and with the proper permissions. This provides the outgoing communication channel where our coworker, the ERP system, keeps everybody informed about every single state change it made.

In Kafka we have all options to implement more business logic without impacting the ERP system at a fraction of the costs.

The result is a win-win-win. The ERP system is used for bookings, one single API – the Kafka Data Producer – keeps all partner systems informed in realtime. I love it when a plan comes together.

Assigned tags

      7 Comments
      You must be Logged on to comment or reply to a post.
      Author's profile photo Ashish Tiwari
      Ashish Tiwari

      So what you are suggesting is that rather than taking the super expensive approach to sap’s way of doing it by having everything on hana and then integrating them. Instead customers can use kafka queues and this can enable the transactional system to stream real time information to all the downstream systems whether its a big data based datawarehouse solution like the Netflix or a reporting on planned and actual data?

      Author's profile photo Werner Dähn
      Werner Dähn
      Blog Post Author

      Correct.

      ... or a ML algorithm, or an alerting system, or a new intelligent automation process, or an external cloud application, or a short experiment with the realtime data, or.....

      Author's profile photo Thomas Ratliff
      Thomas Ratliff

      I like the concept of exposing events by default.  I have had countless discussions on process transparency as it relates to systems of record (single source of truth). Not all ERP owners like sharing instead they demand all events are shared with them.  Difficultly is getting agreement on ownership of the process.  As you state, the ERP system is involved in numerous processes but is not always leading. Where other systems are in the lead, seamlessly tapping into events would yield improved process transparency.

      Author's profile photo Cameron HUNT
      Cameron HUNT

      Hello Werner, if indeed "API first means that there is a well defined interface others can use" [provided by the software vendor], it would seem likely that APIs are provided by SAP to make changes TO SAP DATA = Inbound Interfaces. This is presumably why you state that today, SAP exposes 290 APIs on the Business Hub "FOR S/4Hana".

      Later however you suggest that "Instead of informing that there was a change, provide the change"? If there was a change in this S/4-context, it was because the SAP-provided API was called by an external system (which knows perfectly well that there was a change)?
      You also state that "Instead of providing attributes the developer of the API deemed to be useful, it should communicate everything". Yet Events that "communicate everything" in this context are Outbound from SAP, not Inbound as per the APIs you mention?

      I obviously agree that SAP should publish Events for all changes to its business Entities (e.g. via its provided APIs) -- using open standards like CloudEvents -- but I fail to see how that could possibly replace those Inbound APIs provided by SAP today for effecting changes to SAP-hosted data?

      Author's profile photo Werner Dähn
      Werner Dähn
      Blog Post Author

      Correct. I focus on outbound APIs of the SAP API business hub, e.g. "/A_OperationalAcctgDocItemCube" (type oData) or "/sap.s4.beh.billofmaterial.v1.BillOfMaterial.ItemCreated.v1" (type BusinessEvent).

      For getting data into the ERP system, e.g. creating a new sales order, we must use an official API, yes.

      Author's profile photo Cameron HUNT
      Cameron HUNT

      Outbound is easy, and takes about 1 day to put in place:

      1. use HANA Streaming Analytics and it's included ODBC Inbound (covers ECC) + Kafka Outbound Adapters
      2. ping Kafka using the SAP-provided ABAP REST Library, exploiting the 20+ y/o Workflow Event Engine which is triggered (as part of standard-SAP) when SAP's 'business objects'/Entities are created or modified = Event-driven

       

      Author's profile photo Werner Dähn
      Werner Dähn
      Blog Post Author

      ... and inbound is even easier: oData, ABAP Rest, BAPI, IDOCs. There are enough interfaces provided by SAP and most important: inbound will be just very specific ones. Outbound data we want all.