The productive BW system of one of our clients had a serious problem. Some tables (SID tables of several central InfoObjects) were deleted directly from the database and could not be re-constructed. The only way to get back to a consistent state was a complete restore to a point a few hours back. The connected sources systems (ERP and CRM) were not restored.
What was the consequence?
Since many delta loads took place during the time when the process chains were stopped and the restore point, the delta information were not consistent anymore between the restored BW system and the connected ERP and CRM systems. The situation is described in note 731682 Backup in BW and OLTP: Experiences, risks & recommendations. There it is stated “Therefore, you must usually re-initialize all of the DataSources for which delta data was loaded since the backup time.” Re-Initialization would have had a huge impact as it is not possible to repair the data for all data sources with selective full data loads and the consequence would have been a complete re-load. Nevertheless, even a repair full for a few dozen DataSources would have taken considerable time.
However, we were aware of one fact: Most of the InfoPackages were only executed once in the respective period. This also means that the delta information, which we were missing after the restore, was still sitting in the source system because the majority of data sources support a delta repetition (especially the critical ones like 2LIS). The above mentioned note supported us in our thinking: “You have a generic option to reload the data again by repeating the delta data request and therefore achieving a consistent status again only in the cases in which delta data was loaded precisely once since the backup time”. Basically the last request in BW has to be changed to red and the relevant tables containing the delta information on source system side have to be “adjusted”.
Since the information in the note is not very detailed, we will go into more depth here. If you ever come into such a situation, the information might be helpful to do the right things:
- Identify the DataSources, which have been extracted only once. The best way is to check the loading processes before the restore. If the system is not available anymore, you could still check the scheduled loading processes. If you are not sure, you should always consider a re-initialization or repair load (depending on the capabilities of the DataSource and the data model within BW).
- Then you have to do the following steps for each affected DataSource after the restore:
- The relevant DataSources can be found in the table RSSDLINIT on BW side and in the table ROOSPRMSC in the source system. Typically these two table are synced but not after the restore of the BW system. In the field RSSDLINIT-LAST_DELTA_RNR you will find the last request loaded to BW before the restore point.
- You will need to find the corresponding monitor entry for this request and set the total status to red. This will trigger a delta repeat of the last request from the source system.
- Then you have to go to the source system and check the corresponding entry in the table ROOSPRMSC. The field DELTARNR contains the request id of the last successful load. This has to be changed to the request number which was set to red in BW. The table is typically not maintainable. Therefore, you have to find a way to change the table entries (SE16N, Debugging, Create a small ABAP … ).
- Then you start the Delta-InfoPackage in BW and you should get the warning, that a delta repeat is being executed. This needs to be confirmed.
- After that, we started the Delta InfoPackage another time to capture the normal delta records.
Once all deltas were captured, the normal process chains can be re-started and pick-up the delta information from the PSA.
In those cases where it did not work or where the Delta-InfoPackages were executed more than once, a repair load or a re-initialization was done. Within a few hours, the system was completely back to normal.
There are few general things, which need to be considered to get this working:
- As soon as you realize serious problems in your system, which might cause a restore, you should stop the loading processes as soon as possible. This increases the chances to save the delta information.
- It is important to ensure that the loading processes are not being started right away after the restore. This would typically happen, if the restored system starts as it was. As soon as the next InfoPackage for one DataSource is started, the repeat information in the source system is gone. The basis people supporting this restore ensured this by starting the system after the restore without any batch processes. Once the system was back, first, all relevant process chains were re-scheduled to a later point in time. Then the system was started again and we had enough time to secure the delta information for the different DataSources.
- The approach described here is not transparent. If you check later on the process chains or the monitor of the respective InfoPackages, you will see red traffic lights. However, the data is there.
- While the system is restored, you can also save the delta information for the relevant DataSources in the source system by using RSA7 or RSA3. The information taken from the there can be very useful to validate the result at a later stage.
- There are also other ways to restore the delta information. However, these are typically very DataSource specific and the analysis would have taken some time.
Whatever you do in such cases, you need to be careful.