Final status of the queues after new subscription and events publishing
Some of the audience might also remember one of my childhood’s favorite animated TV series “Voltron”. It was a ”science fiction cartoon” featuring a team of pilots who control lion-shaped robots that combine to form a larger, “more powerful” robot called Voltron.
This is how I see BTP: “Voltron”. (BTP and the BTP services / Voltron and the lions). In fact BTP is and was already a very effective PaaS, but now more effective with the addition of Advanced Event Mesh, a very powerful Event Broker that joined the team (Advanced Event Mesh in fact promises more but this we'll discuss later..)
While formulating the architecture and design of the integrations, you have to know and make use of best combination of possible solutions you have in hand. Keeping in mind the pros and cons of each solution and having making use of powerful parts of each one is the ultimate target. Today in the cloud with SAP BTP, we have a very powerful combination / platform, which can add together the power of API based service oriented architecture with the relatively new kid on the block : Event Driven Architecture.
By combining services and solutions like Cloud Integration, Advanced Event Mesh and CAP, (and with other options I won’t be mentioning here); BTP forms and serves the Voltron, surpassing the sum of its underlying components.
I won’t be mentioning about some of these in detail as there are lots of good technical articles & blogs where the benefits and implementations of Cloud Integration, CAP Development are already presented. There are also various blogs covering the advantages of event driven architecture which you can check for introduction.
Instead, I will focus on a scenario where you can combine the power of these three(BTP CI, CAP and AEM), while mostly focusing on the “Advanced Event Mesh” technically, assuming that the other parts’ are already covered and readers have good knowledge.
And also the sample payload data has been modified for privacy reasons, without affecting the functionality.
Imagine a use case where you have a Sales System(C4C), integrated to multiple target/consumer applications that you need to notify about what is happening about the “Accounts”.(Obviously you can prefer other objects as well. In this use-case Account is picked as it's a master data that is requested often in many scenarios)
One very traditional method to handle this would be a file based async approach, or your target applications' querying your upstream application(C4C) in a pull based approach. This means multiple calls to your application(high cost) because you have multiple consumers. Anothe side effect is tight-coupling your consumers to the source application etc.
What if not all your consumers are interested in "all" the Account changes you will have. How this situation is handled?
This would mean different queries, different routing rules or intermediate data storage problems .. and much more.
Or..
What if our source system notifies the world about the changes in the “Accounts” via events once; and multiple possible consumers can pick whatever they are interested in. And not tightly couple producers with consumers (in fact they are not aware of eachother) so that adding and removing each one will be easier.
Very basicly the main steps of the use-case is as follows:
Below is the high level architecture where three third party application are the targets(interested in the same type of data but from different perspective) and C4C being the source system. The good thing with these scenario is all parties are loosely coupled and in fact source system doesn’t even know who the consumers are.
As the source of the events, in C4C, you first need to register a system to publish the events to and register the specific events to be published via “Subscriptions”.
Administrator > General Settings > System Administration > Event Notification
This is for C4C to publish events to other applications. ( in our case it’s the BTP, Cloud Integration iFlow Endpoint)
In this case I will only be publishing the changes in the Account object's Root node (all create, update and delete operations). Therefore whenever a change-delete-insert happens in this particular object, an event is triggered.
Below you can see some of many subscription options from C4C (you can pick among many and in which detail)
When you make a change to the object, as per design, below event is triggered from C4C (a sample) and published.
This event is published to the BTP, CI iFlow which acts as the orchestrator. For C4C, it’s an endpoint (an external consumer) as configured below
!! Replace the hostname and endpoint URL according to your requirements (whatever you configured in the iFlow)
With these settings, the publishing (source part) is finished.
In order to simplify the scenario, basic version of BTP, CI iFlow is demonstrated(many steps for logging, security, configuration-scripting for specific requirements are removed from the iflow), as the purpose of the blog is to present the overall architecture and the Event Broker : Advanced Event Mesh portion.
Our iFlow “IFLOW_EVENTRECEIVER_TO_CONSUMERS” will be doing the orchestration of the scenario and will be receiving current and as an extension possible future events from current / new source systems.
The main integration process is simply;
The http endpoint to receive the event data
The main and the sub-process
Advanced Event Mesh Connection Details (AMQP Adapter)
The sub process "Process Topics one by one" is doing the Advanced Event Mesh publishing part, where pCurrentTopicToBePublished is the property we set per each iteration as the topic name.
This topic is published to Advanced Event Mesh with the message payload. (We receive these two from the CAP service(API) described below.)
Bear in mind that there can be a single topic or even zero if we decide not to publish the information. It's all up to you, in this case I prefer 0..n topics and iterate it. In the last part we'll be discussing about the fun part (how topics can be used to route messages in Advanced Event Mesh in a very flexible manner)
In this scenario the responsibility for the CAP service is
We won’t be looking at the details for the CAP Service but the logic is simply:
Below is a sample request and response for the CAP Service:
Request
Response
Now it comes to the fun part, Advanced Event Mesh, event broker configuration and topic structure.
Advanced Event Mesh is an enterprise segment event broker in many cases compared to other event brokers such as Kafka, Rabbit MQ,..
The name with the SAP’s other solution Event Mesh is causing confusion as if Advanced Event Mesh is the same solution with Event Mesh but with more capabilities. The fact is, only the last part of this sentence is correct. (Obviously both EM and AEM are event brokers, providing messaging-eventing services, using queues and topics and subscriptions.. But, when it comes to different capabilities like “meshing” the broker services, easily combining async-sync scenarios, and filtering / routing capabilities using topics in a very effective way (which we will be using in this blog and scenario); Advanced Event Mesh behaves very different and shines with what it offers additionaly as an Enterprise Level Event Broker (and more).
Some of the capabilities are already demonstrated in events/blogs/webinars. Along with today's presented features, I will also try to demonstrate these in the following blog posts.
The fun part starts just after Cloud Integration (iFlow) publishes the message to the smart topic(s).(described below)
Assuming that you have already provisioned your Advanced Event Mesh Broker service within your BTP Tenant, you will be facing user interfaces such as below when you have done for the first time(subject to change with the new versions, but the idea is the same)
Let’s return back to our scenario where C4C is publishing “notifications” about the Accounts’ data to BTP.
In the scenario we have 3 different downstream applications expecting new/changed/deleted Account data.
APP_1
APP_2
APP_3
Topic / Topic Subscriptions
In event-driven architecture (EDA), topics serve as a way to categorize the data conveyed in event messages. Events are sent to one or multiple topics, and endpoints (destinations) can subscribe to one or more topics to receive events from publishers. Practically topics are nothing but additional information (free format string) attached to message (like an attribute / header of the message). However when efficiently designed, they are very powerful. (you can organize routing of messages (events) to multiple different consumers in a very flexible manner.)
We make use of topics via topic subscriptions. Topic subscriptions are to attract messages, for telling the world which messages are you interested in as a consumer end-point.
Advanced Event Mesh can make use of wildcards and more in topic subscriptions which gives you a lot flexibility with wildcards "*" and special ">" character. Our scenario is a small demonstration of this usage. (In fact AEM has more, might be a "topic" of another article about EDA / AEM)
Usage will be more clear with our example below..
Below is a sample topic structure that I used (see the placeholder attributes in {{XYZ}} to be replaced by the real values )
Topic Structure
c4c/account/{{version}}/{{salesorganization}/{{country}/{{division}}
Lets assume that 4 events are triggered from C4C. Based on the data within the event, our smart CAP service fills the topics as below:
So the topics formulated by the CAP services will be:
Now this is where the magic happens.
We have three consumers expecting only the messages that they are interested.
Based on the topics, Advanced Event Mesh routes this messages to zero or multiple subscribers.
Advanced Event Mesh uses wildcards and special symbols while making this filtering possible.
How does this happen?
c4c/account/v1/ECOM1/>
c4c/account/v1/ECOM2/>
c4c/account/v1/*/TR/>
or similarly
c4c/account/v1/*/TR/*
c4c/account/v1/*/*/*
or simply
c4c/account/v1/>
Below are our queues in the beginning:
And the topic subscriptions for these queues:
By this setup;
as they subscribed to.
Once you make your design, the extensions would be much easier in the future such as additional source and target systems' onboarding and/or additional new requirements for the current subscribers.
Lets say our new scenario requires APP2 to extend the coverage for the accounts of the France also. In this case The only thing you need to do in this scenario is adding a subscription to the queue for APP2 as below that’s it.
c4c/account/v1/*/FR/>
Once changes are done in accounts for FR, these are published to APP2 as well along with APP3 which is already receiving all countries' accounts.
Onboarding of new consumer:
Lets say a fourth application(APP4) would like to receive account-related information for a particular sales organization (stores with code(STR1)). Easily done in two steps: You assign this app a queue and adjust the subscription for this queue as
c4c/account/v1/STR1/>
and Voila! Start enjoying how easy the related data is routed to the application.
Although we simplify the scenario to demonstrate the intended points:
Now it's easier to consider other combinations of sync/async, combinations of API based and event driven scenarios depending on the target and source systems' capabilities.
Advanced Event Mesh and BTP has many more capabilities to add on top and in the next couple of articles and blogs, my intention is to demonstrate scenarios with some "advanced" features of Advanced Event Mesh and show how it's possible to modernize integrations with the use of Event Driven Architecture.
Please do not hesitate to contact me in case of questions / comments / recommended use-cases.
This is how I see BTP: “Voltron”. (BTP and the BTP services / Voltron and the lions). In fact BTP is and was already a very effective PaaS, but now more effective with the addition of Advanced Event Mesh, a very powerful Event Broker that joined the team (Advanced Event Mesh in fact promises more but this we'll discuss later..)
While formulating the architecture and design of the integrations, you have to know and make use of best combination of possible solutions you have in hand. Keeping in mind the pros and cons of each solution and having making use of powerful parts of each one is the ultimate target. Today in the cloud with SAP BTP, we have a very powerful combination / platform, which can add together the power of API based service oriented architecture with the relatively new kid on the block : Event Driven Architecture.
By combining services and solutions like Cloud Integration, Advanced Event Mesh and CAP, (and with other options I won’t be mentioning here); BTP forms and serves the Voltron, surpassing the sum of its underlying components.
I won’t be mentioning about some of these in detail as there are lots of good technical articles & blogs where the benefits and implementations of Cloud Integration, CAP Development are already presented. There are also various blogs covering the advantages of event driven architecture which you can check for introduction.
Instead, I will focus on a scenario where you can combine the power of these three(BTP CI, CAP and AEM), while mostly focusing on the “Advanced Event Mesh” technically, assuming that the other parts’ are already covered and readers have good knowledge.
And also the sample payload data has been modified for privacy reasons, without affecting the functionality.
USE CASE
Imagine a use case where you have a Sales System(C4C), integrated to multiple target/consumer applications that you need to notify about what is happening about the “Accounts”.(Obviously you can prefer other objects as well. In this use-case Account is picked as it's a master data that is requested often in many scenarios)
One very traditional method to handle this would be a file based async approach, or your target applications' querying your upstream application(C4C) in a pull based approach. This means multiple calls to your application(high cost) because you have multiple consumers. Anothe side effect is tight-coupling your consumers to the source application etc.
What if not all your consumers are interested in "all" the Account changes you will have. How this situation is handled?
This would mean different queries, different routing rules or intermediate data storage problems .. and much more.
Or..
What if our source system notifies the world about the changes in the “Accounts” via events once; and multiple possible consumers can pick whatever they are interested in. And not tightly couple producers with consumers (in fact they are not aware of eachother) so that adding and removing each one will be easier.
Very basicly the main steps of the use-case is as follows:
- Operations in C4C will trigger events and C4C publishes the new/changed information via these events.(mostly via what is called notification events)
- BTP, Cloud Integration (as the orchestrator of not only C4C Events today-but also orchestrator of possible future events from different source systems.) will manage the flow and make necessary calls.
- BTP, Advanced Event Mesh (as an event broker and router) will manage these event stream and communicates via multiple targets, routing the correct information to interested consumers
- BTP, CAP Services for implementing smart logic, rules for data, filtering etc. and exposing this logic to CI via microservices.
ARCHITECTURE
Below is the high level architecture where three third party application are the targets(interested in the same type of data but from different perspective) and C4C being the source system. The good thing with these scenario is all parties are loosely coupled and in fact source system doesn’t even know who the consumers are.
Architecture
COMPONENTS
Cloud For Customer (C4C)
As the source of the events, in C4C, you first need to register a system to publish the events to and register the specific events to be published via “Subscriptions”.
Administrator > General Settings > System Administration > Event Notification
This is for C4C to publish events to other applications. ( in our case it’s the BTP, Cloud Integration iFlow Endpoint)
Target for Event Notification and Subscriptions of Events
In this case I will only be publishing the changes in the Account object's Root node (all create, update and delete operations). Therefore whenever a change-delete-insert happens in this particular object, an event is triggered.
Below you can see some of many subscription options from C4C (you can pick among many and in which detail)
Registration of the target system and events contd.
When you make a change to the object, as per design, below event is triggered from C4C (a sample) and published.
{
"specversion": "1.0",
"type": "/sap.c4c/Account.Root.Updated",
"source": "/XAF/sap.c4c/000000000123456788",
"id": "bb19819d-66xx-1eed-bdc5-0c04a378e286",
"event-type": "Account.Root.Updated",
"event-type-version": "v1",
"event-id": " bb19819d -66xx -1eed-bdc5-0c04a378e286",
"event-time": "1981-13-04T03:30:30Z",
"data": {
"root-entity-id":"001981D0E1M2E3T4CA1EEC8AF58A7D84",
"entity-id":"001981D0E1M2E3T4CA1EEC8AF58A7D84" }
}
This event is published to the BTP, CI iFlow which acts as the orchestrator. For C4C, it’s an endpoint (an external consumer) as configured below
!! Replace the hostname and endpoint URL according to your requirements (whatever you configured in the iFlow)
With these settings, the publishing (source part) is finished.
BTP, Cloud Integration (formerly CPI) iFlow
In order to simplify the scenario, basic version of BTP, CI iFlow is demonstrated(many steps for logging, security, configuration-scripting for specific requirements are removed from the iflow), as the purpose of the blog is to present the overall architecture and the Event Broker : Advanced Event Mesh portion.
Our iFlow “IFLOW_EVENTRECEIVER_TO_CONSUMERS” will be doing the orchestration of the scenario and will be receiving current and as an extension possible future events from current / new source systems.
The main integration process is simply;
- Receiving the event information from via Http Adapter
- Setting some header/properties to be used later (depending on the complexity of the scenario)
- Calling the CAP Service in order to receive the data(the full enriched object data and topics to be published. (Notification Event Payload only have the changed data but we want to publish more detailed information than that)
- Iterating of multiple topics(possible with the scenario) and publishing to these topics.(0..n topics)
- A local integration process to publish to Advanced Event Mesh for each topic received via AMQP adapter.
- Handling exceptions and notifying related parties via another generic process.
The http endpoint to receive the event data
Http Endpoint (start of iFlow)
The main and the sub-process
Main Integration Process
Sub-process sending the message to AEM
Advanced Event Mesh Connection Details (AMQP Adapter)
The sub process "Process Topics one by one" is doing the Advanced Event Mesh publishing part, where pCurrentTopicToBePublished is the property we set per each iteration as the topic name.
This topic is published to Advanced Event Mesh with the message payload. (We receive these two from the CAP service(API) described below.)
Bear in mind that there can be a single topic or even zero if we decide not to publish the information. It's all up to you, in this case I prefer 0..n topics and iterate it. In the last part we'll be discussing about the fun part (how topics can be used to route messages in Advanced Event Mesh in a very flexible manner)
AMQP Adapter Connection and Processing Details
CAP SERVICE
In this scenario the responsibility for the CAP service is
- to formulate the data to be published to the target applications
- to determine which topics to publish this data
We won’t be looking at the details for the CAP Service but the logic is simply:
- CAP Service API receives the event data payload (JSON)
- CAP service queries the C4C System in order to enrich the data (via OData services)
- As per the logic, return one or multiple topics to publish to and enriched object to the caller (CI iFlow)
Below is a sample request and response for the CAP Service:
Request
{
"specversion": "1.0",
"type": "/sap.c4c/Account.Root.Updated",
"source": "/XAF/sap.c4c/000000000123456788",
"id": "bb19819d-66xx-1eed-bdc5-0c04a378e286",
"event-type": "Account.Root.Updated",
"event-type-version": "v1",
"event-id": " bb19819d-66xx-1eed-bdc5-0c04a378e286",
"event-time": "1981-13-04T03:30:30Z",
"data": {
"root-entity-id": "001981D0E1M2E3T4CA1EEC8AF58A7D84",
"entity-id": "001981D0E1M2E3T4CA1EEC8AF58A7D84"
}
}Response
{
"EventTriggeredOn": "1981-04-13T03:30:00Z",
"Topics": [
"c4c/account/v1/ECOM1/TR/DIV1"
],
"Entity": {
"AccountId": "20000001",
"Role": "CRM000",
"ERPAccountID": "358424",
"AccountName": "BB Test Account",
"AccountStatus": "2",
"SalesOrg": "ECOM1",
"DeliveryPostalCode": "34000",
"DeliveryCity": "Istanbul",
"Country": "TR",
"DistributionChannel": "01",
"Division": "DIV1",
"SalesRepCode": "1000132",
"TaxId": "TAX00001",
"CompanyID": "ABCD1234"
}
}Now it comes to the fun part, Advanced Event Mesh, event broker configuration and topic structure.
ADVANCED EVENT MESH
Advanced Event Mesh is an enterprise segment event broker in many cases compared to other event brokers such as Kafka, Rabbit MQ,..
The name with the SAP’s other solution Event Mesh is causing confusion as if Advanced Event Mesh is the same solution with Event Mesh but with more capabilities. The fact is, only the last part of this sentence is correct. (Obviously both EM and AEM are event brokers, providing messaging-eventing services, using queues and topics and subscriptions.. But, when it comes to different capabilities like “meshing” the broker services, easily combining async-sync scenarios, and filtering / routing capabilities using topics in a very effective way (which we will be using in this blog and scenario); Advanced Event Mesh behaves very different and shines with what it offers additionaly as an Enterprise Level Event Broker (and more).
Some of the capabilities are already demonstrated in events/blogs/webinars. Along with today's presented features, I will also try to demonstrate these in the following blog posts.
The fun part starts just after Cloud Integration (iFlow) publishes the message to the smart topic(s).(described below)
Assuming that you have already provisioned your Advanced Event Mesh Broker service within your BTP Tenant, you will be facing user interfaces such as below when you have done for the first time(subject to change with the new versions, but the idea is the same)
Let’s return back to our scenario where C4C is publishing “notifications” about the Accounts’ data to BTP.
In the scenario we have 3 different downstream applications expecting new/changed/deleted Account data.
THE CONSUMER APPLICATIONS
APP_1
- E-commerce application
- Interested in the accounts belong to E-Commerce Sales Organizations(ECOM1, ECOM2)
APP_2
- Regional store application / Turkiye region
- interested in the accounts for country : Turkiye (TR)
APP_3
- Global Application
- interested in every account data regardless of the country, sales organization
THE TOPIC STRUCTURE - DESIGN
Topic / Topic Subscriptions
In event-driven architecture (EDA), topics serve as a way to categorize the data conveyed in event messages. Events are sent to one or multiple topics, and endpoints (destinations) can subscribe to one or more topics to receive events from publishers. Practically topics are nothing but additional information (free format string) attached to message (like an attribute / header of the message). However when efficiently designed, they are very powerful. (you can organize routing of messages (events) to multiple different consumers in a very flexible manner.)
We make use of topics via topic subscriptions. Topic subscriptions are to attract messages, for telling the world which messages are you interested in as a consumer end-point.
Advanced Event Mesh can make use of wildcards and more in topic subscriptions which gives you a lot flexibility with wildcards "*" and special ">" character. Our scenario is a small demonstration of this usage. (In fact AEM has more, might be a "topic" of another article about EDA / AEM)
- * in each level means everthing matches with this level (level is everything between "/" )
- > after the last level means matches every level after your last .. It's like XYZ/*/*/* forever.
Usage will be more clear with our example below..
Below is a sample topic structure that I used (see the placeholder attributes in {{XYZ}} to be replaced by the real values )
Topic Structure
c4c/account/{{version}}/{{salesorganization}/{{country}/{{division}}
Lets assume that 4 events are triggered from C4C. Based on the data within the event, our smart CAP service fills the topics as below:
- First event is for a change in an account from sales organization ECOM1 || from Turkey(TR) || division DIV1
- Second is for an account from sales organization Stores1(STR1) || country: Turkey(TR) || division DIV1
- Third is for an account from sales organization ECommerce(ECOM2) || country: United Kingdom(UK) || division DIV2
- Fourth is for an account from sales organization Stores(STR2) || country: Germany(DE) || division DIV2
So the topics formulated by the CAP services will be:
- Message-1 Topic : c4c/account/v1/ECOM1/TR/DIV1
- Message-2 Topic : c4c/account/v1/STR1/TR/DIV1
- Message-3 Topic : c4c/account/v1/ECOM2/UK/DIV2
- Message-4 Topic :c4c/account/v1/STR2/DE/DIV2
Now this is where the magic happens.
We have three consumers expecting only the messages that they are interested.
Based on the topics, Advanced Event Mesh routes this messages to zero or multiple subscribers.
Advanced Event Mesh uses wildcards and special symbols while making this filtering possible.
- * in each level means everthing with this level
- > after the last level means everything after that
How does this happen?
- All these consumer applications are listening to their own queues(there are different options but widely used one is this)
- APP_1 is a consumer for QUEUE_1Subscription (accounts for E-Commerce Sales Organization(ECOM1, ECOM2))
c4c/account/v1/ECOM1/>
c4c/account/v1/ECOM2/>
- APP2 is a consumer for QUEUE_2Subscription (accounts for country : Turkiye (TR). regardless of other information
c4c/account/v1/*/TR/>
or similarly
c4c/account/v1/*/TR/*
- APP_3 is a consumer for QUEUE_3Subscription (All accounts regardsless of sales org, country, division)
c4c/account/v1/*/*/*
or simply
c4c/account/v1/>
Below are our queues in the beginning:
3 new Queues are created
And the topic subscriptions for these queues:
Queues are subscribed to the topics very easily
By this setup;
- APP_1 -> QUEUE_1 will receive Message 1, Message 3
- APP2 -> QUEUE_2 will receive Message 1, Message 2
- APP 3 (Generic) -> QUEUE_3 will receive all messages : Message 1, Message 2, Message 3 and Message 4
as they subscribed to.
Once you make your design, the extensions would be much easier in the future such as additional source and target systems' onboarding and/or additional new requirements for the current subscribers.
Lets say our new scenario requires APP2 to extend the coverage for the accounts of the France also. In this case The only thing you need to do in this scenario is adding a subscription to the queue for APP2 as below that’s it.
c4c/account/v1/*/FR/>
Once changes are done in accounts for FR, these are published to APP2 as well along with APP3 which is already receiving all countries' accounts.
Onboarding of new consumer:
Lets say a fourth application(APP4) would like to receive account-related information for a particular sales organization (stores with code(STR1)). Easily done in two steps: You assign this app a queue and adjust the subscription for this queue as
c4c/account/v1/STR1/>
and Voila! Start enjoying how easy the related data is routed to the application.
What we have done ..
Although we simplify the scenario to demonstrate the intended points:
- We have enabled C4C system as an event source publishing data to the loosely coupled consumers in a controlled manner.
- With Advanced Event Mesh and BTP, we have enabled multiple consumers to be notified about the changes that they are interested.
- Design a flexible architecture to make future subscribers to easily join the scenario. (and possibly new sources/producers as well)
Follow-up Plans..
Now it's easier to consider other combinations of sync/async, combinations of API based and event driven scenarios depending on the target and source systems' capabilities.
Advanced Event Mesh and BTP has many more capabilities to add on top and in the next couple of articles and blogs, my intention is to demonstrate scenarios with some "advanced" features of Advanced Event Mesh and show how it's possible to modernize integrations with the use of Event Driven Architecture.
Please do not hesitate to contact me in case of questions / comments / recommended use-cases.
- SAP Managed Tags:
8 Comments
