Real-time Consolidation from the Designer’s Perspective
As the architect of Real-time Consolidation, I love this product. It is stopped for further developing since release 1809, and will be continue in maintenance until year 2022. Yet I still see a lot of customers are interested at it, which gives me the momentum to write this blog.
It was all based on a very simple idea: connect BPC with Accounting. The background was also with the arise of the Central Finance, which is supposed to have all the accounting data in a central S/4HANA system. Together with the embedded BPC, we thought we can start something to bridge the 2 worlds together now.
Every successful product behinds an simple philosophy. From every perspective, you can see the simplicity of Real-time Consolidation, even from commercial. All you need is a S/4HANA system, and a lot of BPC experts are already there. Here, I also gives the essential philosophy from technique perspective: Real-time Consolidation is a view generator for consolidation. This is the entrance for understanding the design of all the features of RTC.
Before we open the door to the internal of RTC, I would like to pay tribute to SAP’s existing consolidation solutions. RTC learns a lot from them. Besides the tight integration with BPC, RTC borrows ideas like document concept, posting level, and selection from EC-CS and SEM-BCS. Especially, as a comparison, I would say SEM-BCS is a BW cube generator for consolidation. We also learn from FC for its data collection. But we don’t replicate data all around, instead, we release data with timestamp controls. Local Accountant and Group Accountant roles are thus differentiated even in a centralized system.
Below diagram is extracted from SAP online help. Sorry for the 2 blank boxes which I should have removed them. But at that time, I was a little resistance to remove such 2 attractive features: the left one is “Document Validation”, and the right one is “Elimination”. They are unfinished features, and leave some regrets.
Detail interpretation can be read from the online help. What I want to complement is the “Foundation View”. This terminology is introduced by RTC, because it really acts as the foundation for all the stuff above. And this is the only place I would leave to HANA Studio. Although I know consolidation people don’t like working in HANA Studio, but I must leave the flexibility by leveraging the SQL modelling.
A lot of complains comes from the modelling phase, as it requires a broader skill set in your project team. You need HANA guys, you need BW guys, you even need good Basis guys. But trust me, you only need the basic understanding of the technique stuff, which a studious consultant can master quickly. We once had a plan to hide the technique complexity. The planned product won’t ask for HANA Studio for HANA and BW artifacts, they will all be automatically generated by a powerful modelling tool.
Real-time in my opinion is not only read accounting data in a real-time fashion, but also means real-time modelling. Which allows you adjust your consolidation model, and see the changes immediately after you activate it. Of course, data restatements are necessary for certain changes, but changes like adding a field, or add a consolidation version to your model should not be a big deal.
Talking about the “consolidation version”, it is the same concept as BPC Category. Actually they have a one-on-one mapping when defining a model. The reason we introduced consolidation version is to achieve data isolation and sharing in the new S/4HANA context. In BPC, as you may already know, different BPC categories can, for example, share ownership cube. But with consolidation version, and its 3 version-related fields, we can isolate and share data in more detail granularity. Details can be found from the online help. But I want to talk more on the second version-related field: “Consolidation Ledger”.
The second version-related field is not “Consolidation Ledger” in my design, but “Accounting Principle”. Logically, an accounting principle is a group of consolidation ledgers. Accounting principles can be “US GAAP” or “IFRS”. It contains data from one or multiple consolidation ledgers. Thus I introduced a way of sharing consolidation journal entries. If 2 accounting principles want to share some data, then just post the data into the ledger that is contained in both accounting principles. In ACDOCC, you can not find a field named accounting principle, but only the consolidation ledger field(RLDNR). That is because, like ACDOCA, ACDOCC is also a ledger-based booking table.
Since some surrounding features were not ready, the accounting principle concept was not introduced. You can just regard the consolidation ledger as an accounting principle but only contains one ledger with the same name.
As said, RTC is the view generator. It generates consolidation views on the foundation views. So here views are SQL views which allow to directly access to DB tables. There are 2 main view technologies in SAP: Calculation view and CDS view. RTC supports both. You can define your foundation views either using Calculation View or CDS view. Concerns are received on why RTC still supports Calculation view, isn’t CDS the future? I had the answers in SAP Note 2657410.
The generated consolidation views provide various consolidation functions. You may have already seen a view list in TCode: RTCMD. Each view has long text explanation on how it is/can be used. But maybe following diagram can give you a much clearer picture:
The above view stack is the soul. And most importantly, the Base Views form a solid layer that supports everything on the upper level. You will find the base views not only read data from the foundation view, but also from ACDOCC. So the consolidation journal table and foundation view are merged together on base view level, from which you can see a complete set of the data. Besides, Base Views help to finish the local to group mappings on Entity, Financial Statement Item, and Time. The magic of data release is also utilized in the Base Views which uses timestamps to do data snapshots. Base Views are both CDS views and Calculation Views, with CDS views are consumed by S/4HANA applications, like Validation and Data Release Cockpit; And Calculation views for the BPC consumption. The 2 branches are just sharing the same base and the base only have one data source, which is the foundation view.
Although it is not mentioned in the online help, but it is quite safe for customers to build their own views on the generated views. For example, you can define CDS query views on the generated view “_U00” to enjoy the S/4HANA Embedded Analytics. Examples are given through SAP Note 2621381. You can of course define Calculation views on the generated Calculation views, and through BW virtual provider, you can still use all the BW reporting technologies like Analytic Office or EPM Add-on.
The application layer, as you can see, is still view driven. Validation Engine is based on the generated validation views(with suffix “_V00” and “_V10”). Data Release Cockpit as said is based on the base view which already embeds the snapshot capabilities. There is only one thing left to be mentioned is “ACDOCC_GTT”. It is a global temporary table which shares the same data structure with ACDOCC. You can find it using TCode: SE11. “ACDOCC_GTT” is used to store the intermediate result for consolidation tasks like Currency Translation and Elimination. Although the later is not realized, but you can imagine a use case like this:
The group accountant wants to see the current consolidated statements for management purposes. He chooses the preliminary category and runs the report. Yes, the report reads data directly from ACDOCA, and does translation and elimination on-the-fly. The consolidated result is saved in “ACDOCC_GTT” temporary. However, the group accountant can choose whether to persist the result permanently to ACDOCC or just throw it away by closing the report.
Of course, there are also technique reasons to introduce ACDOCC_GTT, like performance, reducing complexity, and so on. I hope you have already enjoyed the high performance if you run currency translation of RTC, it is all pushed down to HANA. It provides sexy delusion yet solid possibilities on how future financial consolidation looks like. Talking about the currency translation, RTC provides 3 options which are always consulted by a lot of customers on which one to choose. I wrote my answers in SAP Note 2694157.
So when we are talking about “Real-time”, it is not an easy word. “Real-time” means changing all the time. The best situation one can imagine is when I want to stop, then I just stop to take a snapshot. Things become complicate when not everything is under your control. Like in consolidation, data is collected from local subsidiaries, you just cannot ask them to stop and provide the consistent data at the same time. And figures like YTD are based on the cumulative amounts, then what if the amounts were adjusted in prior periods. So it’s all about timelines, and when you are dealing with timelines, you are doing God’s job.
Before, we simplified the job by replicating data. That’s why the ETL concept takes a very important role in all BW/BI solutions. Timelines are bundled together in an simple and rude manner with ETL. Then, claims come on how to recover the data granularity, how to make one trust the data, and how to in-timely report the truth? Those are mainly because ETL takes time, it is a compromise solution that is given under both technique limitation and system landscape status quo.
Now that S/4HANA comes to the game, under the new concept “One Single Truth”, we can go for the direction to merge OLTP and OLAP, from real-time transaction towards real-time analytics. At least, in a controllable scope, like consolidation, we can achieve this.
Two consolidation types are introduced in RTC: Preliminary Consolidation and Final Consolidation. They are represented as consolidation versions under a consolidation model. If a consolidation version is defined as “without data release”, it is for running preliminary consolidation purposes. There is no data control and data is picked up from table ACDOCA without timestamp filtering(I assume the foundation view is on ACDOCA). That is to say, the current update data status of your accounting system is taken into the consolidation system. It is a very simple approach, yet very useful for management reporting, and a preview before the final financial close.
On the other hand, Final Consolidation is a more conservative approach. It requires consents from both local accountants and group accountants. The local accountants can decide to which timestamp the data needs to be released to the group level, while the group accountants can check to accept it or not. Thanks to the particularity of journal entry tables, data changes are only utilized by inserting new journal entries, thus we can simply use the physical timestamp to represent the different statuses of data. The consents are actually based on timestamps before which the data is filtering in and exposed to the consolidation system.
There is one regret thing here. I should have allowed a preliminary version can refer a final version so that the former can reuse the later’s consolidated data from year beginning to the previous period. The benefit is to make the preliminary result closer to the real.
Data release(or submit) request is such an object that is used to record the timestamp(we also call it cut-off time). It is grouped by consolidation entity and fiscal year period. That is to say, each entity in each period has its own data release requests. Yes, we allow multiple data release requests and only the last approved one takes the effect.
Along with the cut-off timestamp, the status of a data release request takes more control means on the data release process. BPC can only see the data with status code “50”. Before “Approved 50”, there are statuses as described in the bellow table, among which I would prefer only to talk about “Deferred 18”.
|Currency Conversion Executed||10||3~9 are reserved for activities before CT|
|Validation Executed||16||11~15 are reserved for activities before Release|
|Rejected||17||Once rejected, no change is allowed|
|Deferred||18||18 can be switched to 17|
|Released||30||30 can be reversed to 1, 19~29 are for activities before release|
|Approved||50||Once approved, no change is allowed|
If a data release request has the status “18”, then the journal entries between the last approved one and this one is copied to the next available period. This is to allow adjustments from prior periods to be counted in the current or next period. The main reason that the YTD model is superior to the period model is that YTD model automatically counts the adjustments of prior periods into current period during data collection. However, the deferral concept is more advanced in coping with those prior periods adjustments. With the data release requests organised in chronological order, deferral is more convenient than submitting YTD data. You can easily trace back to know when the changes take place. The latest online help gives more details on “Defer”.
So far I always assume that the local data is in ACDOCA. But the reality is not always beautiful. There would be still some entities which are not in the central S/4 system. So Flexible Upload seems always a hot topic among all the RTC users. It supports uploading period based data for multiple entities. You always upload the whole period data for the entities, then the app will help you to calculate the delta between the last upload, and it only posts the delta into ACDOCC. This behavior is to align with data release process, but it was argued by some customers who require the “append” mode which the user only needs to upload the new entries. Fortunately, we have released a SAP Note 2721801 to support the “append” mode.
But what I was thinking at that time is to deliver a dedicate consolidation journal posting APP, just like the Accounting’s. Then I didn’t think “append” mode is necessary for data upload cases. This is another regret that I have to mention. At least, with the Note 2721801, you can do similar things like journal posting using excel templates. And you can also self-define validations and checks before uploading. Please check SAP Note 2686132.
There was also requirements that asked “why not support YTD upload?”. As we have introduced the defer concept, I think for external entities, they can upload the data again for the changed periods. The “Consolidation Data Release Cockpit” then displays the “New Posting” notification for the changed period. You can defer them just like for internal entities.
Asynchronous Run Framework
When you run currency translation and validation in Data Release Cockpit, it actually starts a background job and polls the status of the job until it is either finished successfully or terminated with errors. This asynchronous run behavior is illustrated in following diagram:
Here, “Run CT and Validation” are actually 2 ABAP reports, which you can also run them in TCode: SE38. The report “RTC_RUN_VALIDATION” is not only for validation execution, but also for data releasing. If you execute the report, you will see options like: 1) Release if validation pass,
2) Approve if validation pass, 3) Defer if validation pass, and so on. That is also to say, you can schedule them as recurrence background jobs to achieve automatic data releasing. Please read SAP Note 2541326 for details.
There is one thing I need to address here is the concurrency control during run. Once you run CT or Validation for a scope of entities in a certain period, the scope are locked against other concurrency running processes. That is not to say that other runs are prohibited, but the intersection part is filtered out, only processing the entities that are not in the locked scope. In exceptional cases, for example, the server is stopped abnormally, then the locking entries may be not unlocked by the processes. You have to manually unlock them in TCode: RTCAM. Or you will see the entities are always in processing status from the Data Release Cockpit/Monitor.
It’s already a lot. I hope you enjoy Real-time Consolidation, and wish you having a successful implementation. I would also encourage you to enhance it by yourselves. You can build your own CDS or Calculation Views for reporting; You can write your own consolidation tasks by reference “RTC_RUN_VALIDATION”; And you can post consolidation journal to ACDOCC by calling the posting API “CL_RTC_POSTING_METHOD”; You can define your own currency translation keys for your specific translation logic. Just so much I can name.
I also list some hot SAP Notes for your reference: