How to deal with heterogeneous eSOA Landscapes

peter_kulka · ‎06-04-2008

How to deal with heterogeneous eSOA Landscapes

Scope of the Project

In this blog we would like to share experiences and results of the Community Advisory Group (CAG) Heterogeneous enterprise SOA (eSOA) Landscapes that is conducted by SAP together with selected partners and customers. To have good a representation of partners and customers, the selection criteria to join the CAG were the size of the SOA deployment, an advanced usage of SOA technologies and products as well as a good coverage of regions and industries. In regular face-to-face meetings and online workshops, the most urgent challenges were identified and consolidated in a set of five main topics. Then subprojects were set-up to work on the various topics. Here we would like to give an overview of the identified challenges and provide the current state of the project.

Motivation

Developing enterprise applications following the service-oriented architecture (SOA) paradigm is not only popular, but almost common these days. Most sophisticated and complex enterprise application will leverage services that are either developed by customers, systems integrators or provided by multiple software and applications vendors. In larger companies with multiple lines of business or in case of mergers or acquisitions the resulting SOA system landscapes typically comprise multiple Enterprise Service Buses (ESB), service registries and repositories that need to interact with each other. With best-of-breed and multi-vendor strategies applied to the landscape the community members face a new set of integration challenges.

Integration Aspects that need to be considered

In several meetings and workshops the following main challenges in heterogeneous eSOA landscapes were identified in the CAG:

ESB coexistence
When integrating ESBs from different vendors, the integration aspects appear on several level. First of all, you need to define the transport protocol that should be used to exchange information. Good candidates are:

   - SOAP/http
   - SOAP/JMS
   - WS-RM (Web Service Reliable Messaging).

Furthermore, for productive scenarios, the connection between ESBs must be secured. To provide a good measure of security, the messages themselves must be secured (Message Level Security) rather than just applying security concepts on transport layer, i.e. the messages that are exchanged between the ESBs must be signed and encrypted. Additionally a unified user management with a common authentication concept is required in most cases.
Registry coexistence
With the coexistence of ESBs comes the coexistence of registries. Having more than one registry in your heterogeneous SOA landscapes raises questions like:

   - How to federate coexisting registries? Is there a hierarchy of registries or are they all equal?
   - How can registry content be synchronized?
   - Does the WS client need to know, which registry to contact or is this handled by the ESBs?
   - Can development environments handle multiple registries?
Repository coexistence
The challenges that arise from having multiple repositories in your heterogeneous SOA landscape are very similar to the coexistence of registries, i.e.:

   - Is there a leading repository or are all equal? How do I federate multiple repositories?
   - How to avoid inconsistencies? How to keep the repositories in sync?
   - Are different roles in the company in need of different repositories?
   - Is it possible or even necessary to migrate content from one repository to another? If yes, how?

However, integrating repositories is more challenging than integrating registries, due to the fact that there is no standard for repositories available. The UDDI-standard for registries defines a base-set of SOA entities that are part of the registry interoperability, whereas for registries there is currently less consensus to what should be contained in a repository and how to exchange objects between repositories.
End-to-end monitoring
To sustain the TCO of heterogeneous SOA landscapes comprising ESBs from different vendors within a feasible level, it is necessary to provide an end-to-end monitoring. The challenge here is to consolidate the monitoring information that is distributed across the different ESBs, so that it can be accessed centrally.
Non-WS protocols
A key idea of developing composite applications is to leverage functionalities of existing applications and build new applications on the top. Therefore it is necessary to connect existing systems to the ESBs. In case of legacy systems it is often not possible to use Web Service protocols to integrate such systems due to the missing Web Service support. Instead proprietary protocols such as ALE or RFC must be used. A challenge with non Web Service protocols or proprietary protocols is the integration into the ESB infrastructure with regards to the design time, models, runtime and monitoring infrastructure. Further questions are:

   - Are there any other protocols?
   - Does it make sense to use more than one protocol suite to exchange information between two ESBs?
   - Does the client need to know, which ESB to contact in order to call a particular Web Service or do the ESBs forward the corresponding calls?

In addition the question arises how to make these legacy services visible in a repository as part of the overall service map.

Demo System Landscape

For the heterogeneous eSOA landscapes project we set up the following system landscape hosted in the Co-Innovation Lab (CoiL) in Palo Alto:

The SAP NetWeaver Composition Environment 7.1 (SAP NetWeaver CE 7.1) system is used for developing all composite applications that are needed for the project. It also runs the monitoring tools for the end-to-end monitoring subproject. Of course, to gain experiences with different ESBs, we installed an SAP NetWeaver Process Integration 7.1 (SAP NetWeaver PI 7.1) and a 3^rd party ESB including the corresponding registries and repositories. On the application server of the 3^rd party ESB we also deployed EJBs to simulate multiple backend systems (compare the test scenario description below). On the same system we set up an LDAP server for centralized user management. An SAP Discovery System serves as an ERP backend that provides all required Enterprise Services. For the registry co-existence subproject we also installed a 3^rd party UDDI compliant registry as replacement for the UDDI part of the Services Registry. Additionally features of the 3^rd party registry like policy management can be used that way.

Test Scenario

To address the challenges of heterogeneous SOA landscapes described above, we were looking for a scenario that is sufficient generic to be a suitable for many industries, but where we can also show all relevant aspects of heterogeneous eSOA landscapes. As a result we implemented the following scenario:

In this scenario an SAP ERP 6.0 (SAP Discovery System) serves as a central customer master server, which holds all general customer data. This system is connected to the SAP NetWeaver PI 7.1. Other systems that hold additional customer data, such as the bank account data, are connected to an 3^rd party ESB, so that the general data of customer queries is handled by the SAP ERP 6.0 system, whereas all the additional details are obtained from the systems that are connected to the 3^rd party ESB. We simulate the additional systems through EJBs that are deployed on the same application server as the 3^rd party ESB and that are exposed as Web Services. The choice, which backend, i.e. which EJB, must be called, is determined from the number ranges (Nummernkreise) of the customer IDs. The service calls (Enterprise Services and Web Services) for customer queries are synchronous calls. Thus, the SAP NetWeaver PI 7.1 addresses all general customer queries and the 3^rd party ESB provides all additional customer data.

We are planning to provide a set of detailed tutorials about the various topics as soon as results are available, starting with the ESB coexistence tutorial. This blog will then be enhanced with links to the tutorials respectively. Stay tuned!