Consideration of the Duet Enterprise XML-Funneling pattern

[Former Member]

XML-Funneling pattern as answer for dealing with complex SAP data structures

In the blogpost ‘Architecture Pattern XML-Funneling of complex SAP structures for Duet Enterprise interoperability’, I introduced the XML-Funneling pattern as pragmatic approach to handle the Duet Enterprise limitation regarding complex SAP data structures. This Duet Enterprise limitation has 2 origins:

1. The SAP NetWeaver Gateway generator tools are not capable of preservering hierarchical structure; their simple execution model is to flatten the SAP structure in the Gateway DataModel.
2. The SharePoint BCS UI-controls are not capable of rendering hierarchical structure; their default rendering model is list/row based.

An often-heard misunderstanding is that Duet Enterprise itself, via the foundational layers SAP NetWeaver Gateway and SharePoint BCS, cannot be used to exchange complex data structures between SAP and SharePoint. This is not true. Gateway can expose complex data structures, but currently then requires you to model the interface and data signatures in ESR Builder, and next manually develop the Gateway Mappers. Although in FP1 less cumbersone as in Duet Enterprise 1.0, it is by far not the preferred approach to expose SAP building blocks as Gateway services. SharePoint BCS as integration layer can consume complex external data structures, but the rendering via standard controls is limited to the root level. In case the child levels also must be visualized, you need to develop a custom view that interoperates with the BCS API. This is doable, although a painpoint is that the SharePoint BCS API is weakly typed, and therefore programming against the API is laborsome and error prone.

I invented the XML-Funneling approach to simplify the development process at the SAP side. By mapping the SAP hierarchical structure to an XML string, the Gateway generator tools can be applied: a string is namely a ‘simple’ type.

On the SharePoint side the XML-Funneling approach does not really make developer life easier. A marked disadvantage is that it requires custom development to deserialize the received XML-structure into a strongtyped data object, before you can next use and visualize the received complex SAP data structure in a SharePoint view. 

Trade-off decision whether or not to apply XML-Funneling

A trade-off decision is then whether in the total SAP + SharePoint system architecture and development effort, this front-end side / .NET disadvantage outweights the advantage on SAP side? Is it sensible to abandon the XML-Funneling approach in case of complex SAP data structure? My advice is to still apply the XML-Funneling pattern. Justification herefore are the following considerations.

Starting-points

1. SharePoint is the target front-end platform
2. Favour out-of-the-box SharePoint above custom development.
3. Application of SharePoint BCS; i) for the connectivity handling to external data administrations, ii) for the out-of-the-box UI controls to render external data, iii) and for integration within SharePoint Enterprise Search and FAST.
4. Complex data structure is common within SAP data entities

Consequences for dealing with complex external data on SharePoint side

1. SharePoint BCS can import complex external data structure
2. The out-of-the-box UI controls (ExternalList, BusinessData webparts) do not support visualization of complex data structure
3. A dedicated custom webpart can visualize the via BCS consumed complex data structure; by programming against the weakly typed BCS API.

Are there sensible alternatives for XML-Funneling approach?

WCF + DataContracts as alternative for Duet Enterprise to consume SAP data

First, Duet Enterprise is itself a WCF variant: the SAP data is consumed in SharePoint BCS as WCF service endpoint.

Second, on SharePoint side BCS is the limitating factor wrt complex structures, not Duet Enterprise. What if you bypass BCS and directly consume the Gateway Services as WCF endpoints? The result is that you not only loose the added interoperability value of Duet Enterprise: Single Sign-On, Authorization, Logging and Monitoring. But in addition also all of the native values of SharePoint BCS: standard UI-controls to visualize data (not all exposed SAP data is of hierarchical / complex structure), exposal to offline usage, use in SharePoint Search or FAST context.

Conclusion: this alternative is not sensible; it does not comply with the starting-points.

Expose complex SAP data structures via Duet Enterprise into SharePoint BCS

SharePoint BCS is a generic integration layer, usable for arbitrary external administrations: SAP, Siebel, Oracle, SQL-databases, ... This transparant interoperability capability is supported through a script-like, weakly typed BCS programming model. Whenever you can operate with BCS imported external data through any of the available standard UI controls, this doesn’t impose a problem. The controls abstract the weakly typed BCS programming model. But the standard controls are not able to visualize complex external data structures. For them also in this alternative you are required to custom build a view. Due the weakly typed programming model of the BCS API this is laborsome and error-prone.

Conclusion: this alternative is on SharePoint side an hardly better, programmer-friendly approach for consuming complex BCS consumed data; merely different. At SAP side, the consequence is that you cannot use the Gateway Generator tools/pipeline.

Expose complex SAP data structures as multiple associated simple structures

SharePoint BCS supports the concept of associations between two external content types. As parent-child is a relation, you can use this concept to expose complex SAP data structures through Duet Enterprise into SharePoint BCS.

BCS associations certainly has value, e.g. for master-child data model and UI-formats. But it also suffers from limitations that make it less applicable in other scenarios. For one: the net effect is that to retrieve a single complex SAP structure, multiple invocations from SharePoint to SAP must be done to retrieve the entire structure. And the responsibility for retrieving the entire structure is placed on the front-end side; instead of the data provider side. Another, the standard SharePoint UI controls all operate on a single BCS entity. If in the User Interface Design the ‘child’ data is directly mingled with(in) the ‘parent’ data, the standard controls cannot be used.

A Duet Enterprise design time issue is that the SAP Duet Enterprise generators are focussed on a single Gateway datamodel, and have no support for defining associations between 2 datamodels. Usage of BCS associations requires you to afterwards modify the BDC Model that is generated by BDC Publisher to add the associations. Either via SharePoint Designer, but in my experience you will more often than not need to resort to manual modification of the BDC Model. Which is time consuming, and error prone.

Conclusion: associations suffers from limitations and software architecture + performance drawbacks.

Final thought

Given that you want to benefit from the interoperability values of Duet Enterprise, and that within the exposed SAP business package there are entities with complex structure; XML-Funneling is a feasible approach. Consistent application of the pattern results in recognizable code setups for multiple Duet Enterprise scenarios, on SAP and SharePoint side. And in our experience, it is also possible to develop reusable building blocks for SAP and SharePoint to ease in the application of the pattern.