End of Purpose Check and ILM-based Archiving

[johannes_gilbert]

Table of Contents

• 1. Introduction and Simplified Determination Approach for the End of Purpose
• 1.1. Right to erasure
• 1.2. Blocking
• 1.3. End of Purpose
• 1.4. (Oversimplified) End-of-Purpose determination
• 2. Extending the End-of-Purpose Determination Using Waiting Periods
• 2.1. Waiting Periods
• 2.2. Criteria for Defining Waiting Periods
• 2.3. End of Purpose for master and transactional data
• 3. Waiting Period in More Detail
• 3.1. Overview of Waiting Periods
• 3.2. Divergent Definition of Waiting Periods K₁ and K_2.2
• 3.3. Application-specific Waiting Period for Data Archiving - K_2.1
• 3.4. Accounting regulations for companies

Legal Disclaimer

The following article deals with a mixture of legal as well as business requirements. Thus, it is in general infeasible to provide legally compliant and universally correct statements for all possible use cases. This article does not provide any legal guidance, nor should it be interpreted as giving it as such.

Executive Summary

This article serves to increase the understanding and should be a work of reference for the topics of end-of-purpose determination and ILM-based archiving. The terms usage, blocking, deletion, waiting and retention period are explained and put into context. In the course of doing so, various constellations of these time periods are analyzed. The analysis leads to practical consequences for usage.

Acknowledgments

I would like to give special thanks to my colleagues Reinhard Scheid, Manuel Brand and Paul Townsend. Their comments, reviews and knowledge corrected, improved and smothend this article.

Preface

I should point out that the following introductory words are to be taken with a good dose of humor. The group of topics this article deals with is primarily complex. Newbies should not consume this article all at once. Allow yourself breaks. Read each single chapter with nice gaps in between. And don't despair if you quickly feel like you are 'lost in space' - stick with it, it'll make sense in the end.

1. Introduction and Simplified Determination Approach for the End of Purpose

1.1. Right to erasure

Chapter 1 of the General Data Privacy Regulation (GDPR) defines priciples for processing of personal data.

Personal data shall be [...] collected for specifed, explicit and legitimate purposes and not further processed in a manner that is incompatible with those purposes. (Art. 5 p. 1 GDPR)

These principles result in the necessity to erase personal data when the so-called purpose has ended. An example:

Customer Miller puts in an order. Three days later, this order has not yet been processed. Miller revokes his order. During these three days, the order plus all necessary data are allowed to be stored. The primary purpose 'Order-Sales-Delivery' is active during these three days. After those three days, this primary purpose has ended. Thus, the personal data (the order plus any related personal data gathered in addition) needs to be deleted.

Figure 1.1 Usage and Deletion

1.2. Blocking

Deleting (personal) data directly at the point when the primary purpose expires is usually prohibited because of other laws. The most prominent example for the jurisdiction of Germany is the Tax Code (German term: Abgabenordnung (AO)). The Tax Code defines retention obligations and thus prevents data deletion for a certain period of time. In this case Art. 17 p. 3b GDPR comes into play. If other laws prevent deletion, the (personal) data needs to be blocked until its final deletion. I.e. access to blocked data must be prevented for typical users. Only users with special authorizations (and due to legally compliant reasons) are allowed to display the data (e.g. a tax auditor). There are no concrete requirements mentioned in the GDPR regarding the ways or rather the technical approach needed for blocking. However, the measures that are used need to be effective and applied. Setting a blocking indicator (e.g. a boolean value) might be suitable. In combination with authorization checks, the blocking indicator forces an access and thus also a processing restriction.

As a result, blocking takes place after primary purpose expiration. The blocking lasts until the data is deleted. The retention period (according to the German Tax Code (AO)) lasts for the same duration.

Figure 1.2 Usage, Blocking and Deletion

1.3. End of Purpose

Metaphorically speaking, the end of purpose is the arrowhead of the yellow arrow.

Figure 1.3 End of a Business Transaction

Strictly speaking, it is a determination of a (calendar) date. In our example of the cancelled order it is the cancellation date. Depending on the business transaction, the so-called end of business can differ. E.g. for a non-cancelled order there will be no cancellation date, rather the delivery date might be used instead. Please be aware that the end-of-business determination within SAP software is not only product specific but in addition module specific, document-category specific etc. Thus, it is highly specialized.

The end-of-business determination for transactional data (e.g. orders, billing documents, …) is business-process specific. For master data, this determination is far more complex. Terminology already indicates that a direct transfer of the concept for transactional data is impossible. For master data it is more suitable to use the term end of purpose.

For master data, it is essential that there is no ‘best-before date’ defined by a process or by something else. Let’s take the example of a customer (KNA1-record). Such a customer exists. Until when? Forever, without data privacy or other legislation. There is no process or organizational measure which limits the usage period and/or forces deletion of customer master data. Due to the GDPR, such behavior is made impossible. A customer master record must only exist as long as there are no expired primary purposes. The end of purpose for a (person-related) master record (blue dotted line in the figure shown) in general results only from the end dates of all business transactions (yellow arrows) related to this master record.

Figure 1.4 End of Purpose

From a mathematics perspective, the end of purpose of (person related) master data records can be described as follows:

EoB_Partner = max( EoB_Bus.Process1, EoB_Bus.Process2, ..., EoB_Bus.ProcessN)

In human words: The end date of the primary purpose for a business partner results from the set of business transactions of this partner, namely the maximum end date of all these transactions. Partner in this case is synonymous for SAP Business Partner (BUT000), customers (KNA1), vendors (LFA1) and contact persons (KNVK).

1.4. (Oversimplified) End-of-Purpose determination

For the determination of a partner’s end of purpose, the following process applies:

• Determination of all business transactions of the partner.
• Determination of the end date of each business transaction.
• Calculation of the maximum value of all end dates (= end of purpose).

This determination process has a major disadvantage: There is a fixed and direct coupling between the business transaction’s end dates and the partner’s end of purpose. In particular, there is no possibility to incorporate some kind of waiting period for this (over-)simplified approach. Hence, it is not taken and implemented in SAP products.

However, this (over-)simplified approach does show the basic principle of end-of-purpose determination without digression and (too) complex details. It should be understood before moving on to the more complex details that now follow.

OK, time for a coffee or tea break

2. Extending the End-of-Purpose Determination Using Waiting Periods

2.1. Waiting Periods

Well-informed readers might have already related the term waiting period to the so-called residence period from ILM. For readers who are new to the topic, as well as those who have already fallen out with the ILM nomenclature - and all other readers for that matter - it is advisable to stick with the term waiting period for educational reasons. Strictly speaking, the ILM residence period is only one occurrence of a waiting period for the end-of-purpose determination.

The Pareto principle (80-20-rule) applies to the end-of-purpose determination too. The (over-) simplified approach explained so far only requires 20% of the cognitive and implementation effort to reach 80% of the overall solution. Let’s now tackle the remaining 20%, which unfortunately requires 80% (or more) of the cognitive performance and effort.

From the technical perspective, the non-existence of a waiting period is not an issue for the determination process. However, from operational perspective, such a waiting period might be required for the following reasons:

• Company-internal processes, e.g. annual audit
• Returns, complaints, cancellations
• Guarantee periods
• ...

These different motivations for needing a waiting period typically result in different waiting period durations, e.g. depending on the process in question. In addition, it needs to be possible to define a waiting period using different criteria. Several overlapping waiting periods might be necessary too. To visualize this, let’s go back to the example we used previously:

Customer Miller puts in an order. This time it is not cancelled after three days but processed and delivered as usual. With the delivery, the invoice is issued. Customer Miller pays a few days after receiving the goods.

Figure 2.5 Business Transactions Order, Delivery and Invoice

Assuming the invoice is the final business transaction for customer Miller in this process (i.e. for reasons of simplicity we won’t consider accounting or similar processes here), the (over-)simplified determination approach produces an end of purpose which is equal to the invoice’s end date. However, waiting periods should be applied too for the reasons already mentioned above. We therefore define the following waiting periods:

Table 2.1 Waiting Periods

Type of Business Process	Waiting Period
Order	1 week
Delivery	8 weeks
Invoice	4 weeks

The schema of business transactions changes as follows: There is still a direct succession of business transactions. However, the end-of-purpose determination now considers these waiting periods.

Figure 2.6 Business Transactions with and without Waiting Periods

It should be noted that depending on the waiting periods (which might be nonsensical), some other business transaction could also be controlling the end of purpose.

Figure 2.7 Waiting Period variation for Business Transactions

OK, time for a relaxation exercise

2.2. Criteria for Defining Waiting Periods

So far, we only distinguished the waiting period based on the type of business transaction involved. In general, there is nothing stopping you from further differentiating within a type (only the enormous increase in complexity). For example, the delivery’s waiting period could be derived from the delivery type. Depending on the type of business transaction involved, there are specific criteria that are only available and/or relevant for this type. There might be some criteria, however, that apply to several types of business transactions. Let’s have a look at some concrete examples:

Criteria for defining waiting periods for orders:

• Order type
• Sales document category
• Sales organization

Criteria for defining waiting periods for deliveries:

• Delivery type
• Sales district
• Sales document category
• Sales organization

Criteria for defining waiting periods for invoices:

• Billing type
• Sales organization

In this example, all three types of business transactions use the criterion sales organization. Similarities can be seen for the sales document category. One criterion, the sales district, only occurs for deliveries. We enhance the table for defining waiting periods as follows:

Table 2.2 Waiting Periods in dependency of criteria

Type of Business Process	Order type	Sales Document Category	Sales Organization	Delivery Type	Sales District	...	Waiting Period
Order	*	*	DE	-	-	...	1 week
Order	*	*	FR	-	-	...	2 weeks
Delivery	-	*	DE	*	*	...	8 weeks
Delivery	-	*	FR	*	*	...	9 weeks
Invoice	-	-	DE	-	-	...	4 weeks
Invoice	-	-	FR	-	-	...	5 weeks

Explanation: Criterion not available ⇔ - | Criterion irrelevant ⇔ *

Due to the additional criteria, it is possible to define the waiting period specific to a sales organization (this is shown in the above table). The three original waiting periods are distinguished per country (DE, FR). In addition, the waiting period for country FR is always one week longer compared to the one for country DE.

Table 2.2 also shows a clear disadvantage. Nearly all columns (criteria) are only used by a single type of business transaction. Only few are used multiple times. Thus, this is a sparse table (refer to sparse matrix). Adding more types of business transactions with additional criteria would vastly increase the size of the table. At the same time the table’s usability would tend towards zero. Apart from the poor usability, there are also technical disadvantages for such a table concept:

• Adding new types of business transactions results in the necessity for new columns.
• The same applies for adding new criteria for an existing type.

A solution for such sparsity is to split the single table into several ones. Due to the different types of business transactions available, it is advisable to use one table per type. This enables you to add further criteria per type in future or to implement extensibility features per type.

Table 2.3 Waiting Periods for Orders

Order Type	Sales Document Category	Sales Organization	Waiting Period
*	*	DE	1 week
*	*	FR	2 weeks

Explanation: Criterion irrelevant ⇔ *

Table 2.4 Waiting Periods for Deliveries

Delivery Type	Sales District	Sales Document Category	Sales Organization	Waiting Period
*	*	*	DE	8 weeks
*	*	*	FR	9 weeks

Explanation: Criterion irrelevant ⇔ *

Table 2.5 Waiting Periods for Invoices

Billing Type	Sales Organization	Waiting Period
*	DE	4 weeks
*	FR	5 weeks

Explanation: Criterion irrelevant ⇔ *

The used schema changes as follows:

Figure 2.8 Usage, Waiting Period, Blocking and Deletion

The waiting period is offset from the blocking period. This might be surprising, but the explanation is as follows: Firstly, the waiting period cannot be part of the usage period as the latter ends with the business transaction’s intrinsic date. Secondly, according to §147 AO DE/EN there is no relation between the retention period and a waiting period . The retention period (in Germany) is just determined using the creation date (cf. §147 p.4 AO DE, EN).

2.3. End of Purpose for master and transactional data

Some readers might have already noticed that the relationship between master and transactional data has not been differentiated clearly in the previous figures. The following figure illustrates the periods for usage, waiting, blocking and deletion separately for master and transactional data. Let’s start by looking at the order again. Its figure changes as follows:

Figure 2.9 Usage, Waiting Period, Blocking and Deletion for the Order

We enhance this figure with customer master data that we haven’t visualized up until this point. The usage period of the business partner master data record (green arrow) is by definition equal to the order’s usage period plus its waiting period. At the end of the order’s waiting period, the order itself as well as the business partner master data record are to be blocked.

Figure 2.10 Usage, Waiting Period, Blocking and Deletion for the Order and the Business Partner

For auditing and tax auditing, it is not sufficient to only retain the order. The business partner needs to be retained too. The retention period of the business partner master data record is determined by the order’s end of purpose as well as the order’s retention period. Both periods end at the same point in time (or rather on the same day), since the business partner master data record must not be deleted before the order is deleted. In the case here (we’re only considering the order), the blocking (period) of the business partner master data record comprises the same period as the blocking of the order. Moving on from that, now we want to add the delivery and invoice to the figure to get the big picture. First, we add the delivery to the figure.

Figure 2.11 Usage, Waiting Period, Blocking and Deletion for the Order, Delivery and Business Partner

As the difference to the previous figure shows, considering the delivery in addition changes the business partner master data record’s end of purpose. It has now moved into the future. In addition, the retention period for the business partner master data record has changed. It starts with the end of purpose that results from the order and the delivery, and ends with the maximum retention period of the order and the delivery. Let’s add the invoice.

Figure 2.12 Usage, Waiting Period, Blocking and Deletion for the Order, Delivery, Invoice and Business Partner

Again, the end of purpose for the business partner master data record has changed. Now it results from the dates of all business transactions (order, delivery, invoice). The business partner master data record’s retention period aligns with the maximum end date of all three retention periods for order, delivery and invoice.

3. Waiting Period in More Detail

3.1. Overview of Waiting Periods

The examinations of the waiting period made so far are too coarse-grained. Apart from the end-of-purpose determination we have considered up to now, a waiting period is also present for data archiving. Overall there are the following waiting periods:

• Waiting period considered at end-of-purpose determination (K₁)
• Waiting period considered at data archiving (K₂)
  • Application-specific waiting period at data archiving (K_2.1)
  • ILM-based waiting period at data archiving (K_2.2)

The following table includes information about each definition location, definition details, and the commonly used synonym for all three waiting periods.

Table 3.6 Details on the Definition of Waiting Periods and Common Synonyms

Waiting Period	Location of Definition	Definition Details	Common Synonym	Usage
K1	ILM	• Transaction IRMPOL • Policy category residence periods (RST) • Audia area (always) BUPA_DP • Arbitrary policy name • ILM objects for SAP Business Partner (CA_BUPA), Customer (FI_ACCRECV), Vendor (FI_ACCPAYB), Contact Person (FI_ACCKNVK)	residence period	Residence policies in audit area BUPA_DP: determine the waiting period for the EoP-check
K2.1	Application customizing table	• Via Customizing reference guide or SM30 • Application-specific table	residence period	determine the waiting period for archiving
K2.2	ILM	• Transaction IRMPOL • Policy category residence periods (RST) • Audit area (always) ARCHIVING • Arbitrary policy name • ILM objects for Sales Order (SD_VBAK), Delivery (RV_LIKP), Invoice (SD_VBRK)	residence period	Residence policies in other audit areas: determine the waiting period for archiving

Looking at the table above, it’s obvious that the usage of the term ‘residence period’ is often a cause for misunderstandings, since the same term is used for all three waiting periods. However, all three waiting periods control different aspects. Thus, they should be looked at independently from each other. In addition, either waiting period K_2.1 or K_2.2 should be used, but not both at the same time. For this reason, only ILM-based alternative waiting period K_2.2 is considered in the following section to begin with.

3.2. Divergent Definition of Waiting Periods K₁ and K_2.2

Technically, there is nothing stopping anyone from using all three waiting periods at the same time. It would be possible to define a waiting period of 1 week for the purchase order for the end-of-purpose determination of the business partner master data record (K₁), and at the same time a waiting period of 1.5 weeks for data archiving purchase orders (K_2.2). The figure for this would look like this:

Figure 2.13 Waiting Period K₁ less than K_2.2

The issue with this configuration is the existence of the pink-colored time period. Within this space of time, the following is in force:

• The business partner master data record is blocked and cannot be displayed (by regular users) or used to create new business transactions.
• The order is not yet blocked (not archived) as waiting period K_2.2 has not yet passed. Waiting period K_2.2 has been defined like this due to business reasons, e.g. to be able to access and change the order in case of returns.
• Changing the order (no matter what the actual change is) is prohibited or rather not possible as the related business partner master data record is already blocked and the system prevents any changes being made for this business partner.
• As a result this configuration is inconsistent.
• It can be generalized that the waiting period for data archiving (K_2.2) cannot be longer than the waiting period for the end of purpose (K₁). Mathematically speaking: K_2.2 ≤ K₁

Now let’s examine the opposite case. In this case, the order’s waiting period for the end of purpose of the business partner master data record (K₁) is defined as 1.5 weeks. At the same time, the waiting period for data archiving of orders (K_2.2) is only 1 week. The figure looks like this:

Figure 2.14 Waiting Period K₁ longer than K_2.2

The issue with this configuration is once again the existence of the pink-colored space of time. Within this period the following applies:

• The business partner master data record can be displayed and used as usual.
• The order is already blocked (archived) as waiting period K_2.2 has already passed.
• Changing the order (no matter what the actual change is) is not possible as it is already archived.
• From an operational point of view there is no reason to not (also) block the business partner master data record. All business transactions (here only the order) are already completed (incl. waiting period). Taking a look into the system would show that only the business partner exists, but no business transactions. The business partner’s end of purpose is applied too late.
• As a result, this configuration is inconsistent.
• It can be generalized that waiting period K_2.2 must not be less than K₁.
• In addition to the above-mentioned considerations, this results in the fact that waiting periods K₁ and K_2.2 should in general be the same. Hence: K_2.2 ≥ K₁ and as consequence K_2.2 = K₁

The figure for K₁ = K_2.2 is as follows:

Figure 2.15 Waiting Periods K₁ and K_2.2 equal

3.3. Application-specific Waiting Period for Data Archiving - K_2.1

SAP Sales & Distribution (SD) Sales Order

So far, we have not yet examined waiting period K_2.1. The reason for this is a property of this waiting period. By definition, it is application-specific and it is (potentially) interpreted non-uniformly. For example, the SD sales order and the SAP Customer Relationship Management (CRM) sales order have been using waiting period K_2.1 for decades. However, the related data archiving processes differ when it comes to the technical check for whether or not the business (respective) order is archivable.

For the SD sales order the technical and operational checks on the termination of waiting period K_2.1 are performed within a single process step (refers to the archive write program). This step also checks waiting period K_2.2 (if defined). If the evaluation is successful, data archiving is directly possible.

Figure 2.16 Waiting Periods K_2.1 and K_2.2 for the SD Sales Order

The following configurations are possible:

• K_2.1 = K_2.2
  
  No divergent behavior compared to K₁=K_2.2 (see chapter 3.2).

• K_2.1 < K_2.2
  
  As both K_2.1 and K_2.2 are considered, only K_2.2 is significant. Thus, there is no divergent behavior compared to K₁ = K_2.2.

• K_2.1 > K_2.2
  
  As both K_2.1 and K_2.2 are considered, only K_2.1 is significant. As a result there is no divergent behavior compared to K₁ = K_2.2

After examining all possible configurations, it can be summarized that for the SD case either K_2.1 or K_2.2 should be used, but not both waiting periods at the same time. Moreover, these two waiting periods have the same effect and are thus equivalent.

CRM Sales Order

For archiving CRM-business transactions there are different possibilities. For the CRM sales order alternative A performs technical and operational checks as well as checks for the expiration of waiting period K_2.1 within a single process step. However, this process step is a preliminary step (the so called archive preprocessing program). The successful check results in a status change of the sales order (can be archived (refer to transaction BS23 and status I1100)). The business transaction is not automatically archived. Another process step is necessary which completes archiving. This second process step checks waiting period K_2.2.

Figure 2.17 Waiting Periods K_2.1 and K_2.2 for the CRM Sales Order

In contrast to the SD sales order the waiting periods K_2.1 and K_2.2 are not equivalent but different and cumulative. In addition, the CRM sales order is not changeable during waiting period K_2.1 and K_2.1 is one day at minimum (cannot be zero). The SD sales order is still changeable till the begin of the retention period. Hence, the waiting periods K_2.1 and K_2.2 are mutually exclusive for many CRM-business transactions (refers to One-Order-based business transactions. Non-One-Order based business transactions often follow different rules during archiving.). From operational perspective both waiting periods address the same requirements. Thus, one could conclude to define K_2.1 and K_2.2 to have an equal length. Due to their implementation and their resulting flow of both waiting periods *after each other one should - if possible - relinquish using waiting period K_2.1 (K_2.1= 1 day).

Alternative B avoids the application specific waiting period K_2.1 completely. For this purpose a different, alternative process step is used. In stead of setting status archivable the status archiving direct deletion (refer to transaction BS23 and status I1108) is used.

The following configurations are possible:

• K_2.1 = K_2.2
  The end of business transaction for the sales order is already reached (corresponds to the existence of status I1005 completed at the business transaction). The sales order's retention period has not started yet because K_2.1 has not expired. This results in the problematic aspects described in chapter 3.2.

• K_2.1 < K_2.2
  This configuration is further distinguished into:

• K_2.1 = 1 day
  With this configuration alternative A as well as B are defined suitable. No divergent behavior in comparison to K₁=K_2.2 occurs (see chapter 3.2) except for the one day.

• K_2.1 > 1 day
  For alternative *B there is no divergent behavior in comparison to K₁=K_2.2 (see chapter 3.2). However, for alternative A there are the problematic aspects described in chapter 3.2.

• K_2.1 > K_2.2
  There are the problematic aspects described in chapter 3.2.

3.4. Accounting regulations for companies

Besides the logical implications for waiting periods that are described in the previous chapters there are legal boundary conditions for waiting periods. Depending on the relevant jurisdiction a company must adhere to one or more accounting regulations. For example the International Financial Reporting Standards (IFRS) as well as the regulations of Handelsgesetzbuch (HGB) might be authoritative. In such a case it might be likely that waiting periods K₁ and K_2.2 are four years. A waiting period of such a length is obviously longer than the timer period of the active usage of a business transaction. Anyways the discussed logical implcations still apply for such long waiting periods.