The “advanced” DataStoreObject – renovating BWs persistency layer

With BW7.4 SP08 another “feature package” was delivered with a lot of new functionality and optimizations based on the HANA platform (see http://scn.sap.com/docs/DOC-35096). In accordance with the BW strategy to deliver the HANA-based innovations and renovations non-disruptively to our customers the SP08 delivered a first step in renovating our Persistence Layer. With the possibilities of HANA we are able to consolidate all BW InfoProviders with persistent data into a single object with a single and simple data base layout. This is similar to the consolidation and simplification in the Virtual Layer based on the CompositeProvider.

Since its properties, table layout and services are closest to today’s DataStoreObject we simply kept the name. Only if we need to explicitly distinguish between the current object and the new object we add “advanced” DataStoreObject or “classic” DataStoreObject. This also emphasizes the role of BWs DataStoreObject as THE central building block for a Data Warehouse architecture.

Major features

New intuitive Eclipse-based modeling UI

The UI renovation continues. The ultimate goal: bring the heavy-weight modeling UIs to the Eclipse-world and the administration and monitoring UIs to the Web. With the Eclipse-UI for the advanced DSO it is now possible to model all InfoProviders of a complete end2end scenario in the BW Modeling Tools (from an OpenODS View, to the advanced DSO, CompositeProvider and BW Query). All this is deeply integrated and aligned with the HANA Studio and Modeler for all “mixed-case scenarios” and a unique modelling experience.

Combine field- and InfoObject-based modeling

The “higher” up in your architecture an InfoProvider is located the more important are the rich features of BWs masterdata modeled in an InfoObject (like re-usability, data consistency, visibility, and quality). But for new scenarios the InfoObject approach is a high hurdle, since it requests a top-down approach right from the start. With the Open ODS View we introduced in BW7.4 SP05 already a virtual field-based approach to integrate data without the need of having InfoObjects. With the advanced DSO this is now also possible for the Persistence Layer. I.e. you can load data into BW without assigning InfoObjects to each and every field without losing functionality.

In combination with the Open ODS View this approach allows you to integrate new data easily and fast, switching seamlessly from an e.g. even virtual data access to a managed persistence with an increasing level of data integrity. All this especially tabbing into “non-classic BW data” with the support additional data types and adapters.

High frequent data loads – based on optimized request management

The request management is, if you like, the cornerstone of BWs managed approach to Data Warehousing since it organizes availability and status of the data from the time it is requested of the source system up to the caching by the Analytic Manager (aka OLAP Engine) or the delta buffers of the integrated planning. With the size of the BW systems growing and growing, the complexity increasing and the demand for more and more (close-to-) real-time scenarios we decided to re-write our request- and process-management for the advanced DSO.

The current request management still works for the “classic” objects (DSOs, InfoCubes, …) and a single dataflow can work with both types as well. The “new request ID” (internally called “TSN” – Transaction Sequence Number) is no longer an INT4 value, but a timestamp plus a generated, increasing postfix. This not only removes the 2 billion limitation (for the system-wide number of requests), but also allows for new logic and new semantic derived directly out of the request. The new request management comes with a new “manage UI” directly accessible from the Data Warehousing Workbench that allows you to quickly navigate through very large sets of requests and protocols and perform manual tasks or monitoring activities.

Reporting performance optimizations

With the goal to replace the InfoCube we also had to make sure that some of the more complex (or “exotic”) reporting scenarios perform well. Two features are important to achieve this goal. Only in case of an INNER JOIN between the table with the facts and the masterdata table, can HANA perform optimal and BW can push OLAP operations to HANAs Calculation Engine. The check whether or not an INNER JOIN can be used (instead of a LEFT OUTER JOIN) is the BW referential integrity check during loading or activating the data (aka SID-creation, but is much more than just determining an INTEGER value). This check exists in the classic DSO as well (the so-called “BEx-flag” – the “create SIDs” option). But for the advanced DSO it is possible to set this flag not only on InfoProvider level, but individually per characteristic of the DSO. The data integrity check then is only executed on the selected characteristics.

The second feature is also related to the JOIN with the masterdata tables. Also HANA benefits from the fact that for an InfoCube JOINs to the masterdata tables are via INT4 values, compared to STRING JOINs for a DSO. In rare cases the performance difference can be crucial, e.g. if you have a compound, high-cardinality InfoObject like 0MAT_PLANT and the reporting mostly includes navigational attributes of this InfoObject and therefore forces this JOIN to be executed very often. In such cases you can, for individual InfoObjects(!), turn on the persistency of the InfoObject-SID in the DSO tables. An additional column will then be created in the active data table to hold the SID and will be filled during data activation.

And More

Just to mention a few additional highlights:Up to 120 key fields are supported. Template-based design using references to classic BW InfoProviders types or best-practice LSA++ templates. Integrated Remodeling for structural- or type-changes. DB-partitioning on all key fields. … and so on …

Positioning

The advanced DSO will be the central persistency object in BW-on-HANA replacing especially InfoCube, classic DSO, HybridProvider and PSA. While there are still some gaps to cover the complete functionality, we recommend considering the advanced DSO for all new projects as the central (and only) persistency object. A widespread conversion of existing scenarios into advanced DSO is currently not recommended, but should be done only case-by-case and demand-driven. We plan to provide a conversion tool to support the conversion of objects in future SPs. The current persistency objects are of course still supported and do co-exist with the advanced DSO. There are no plans to change this.

Roadmap

New minor features will be added and some gaps will be closed in subsequent SPs (SP10, SP11, SP12) – a list can be found attached to note 2070577. Another “feature package” of the BW7.4 release is planned with SP13 (scheduled for Q4 2015). This “feature package” shall then close all major gaps between today’s functionality for InfoCubes, DSOs, PSAs and the advanced DSO. Additionally the advanced DSO is planned to support several new features like Dynamic Tiering (fka ExtendedStorage) support, direct- and Bulk Load enablement, … . And a conversion tool will then also allow you to convert existing InfoProviders into an advanced DSO.

Availability

Please check note 2070577 for the details according to your support package level. Additional and detailed information can be found in the SAP online help.

See here for a short video demonstrating the most important aspects.