Archiving – How it is designed, how does it work
In this blog I want to explain how we have designed Archiving in C4C (available from 2011, see this Product Information) and how it works (not immediately after you active it in the Business Configuration 🙂 ).
Also with an update for release 2208:
As we experienced in some customer systems with a heavy load we optimized for performance.
Let’s start with the activation:
As it is already stated in the Product Information you first need to choose in the Business Configuration for which of the Business Objects you want to switch Archiving on together with some retention periods..
We will come later to these retention periods.
Now this has the following consequence:
- Early morning each day 2 jobs are triggered for Archiving.
- Each job is working on all activated Business Objects consecutively.
- The first job Move2Archive looks for potential candidates, let them verify by the resp. application, move these verified ones to the archive, and deletes them for the local database.
- The other job DeleteFromArchive deletes the data finally from the archive
- These topics are applied to each Business Object individually: 2 jobs per enabled Business Object
This means if you activate Archiving in the afternoon you have to wait until the next morning to see a potential result.
But what are the results?
Now we need to discuss the Retention Periods (they are pretty well explained in the blog mentioned above, but let me repeat):
- Retention Period Before Archiving: This defines the “age” of the record. How long it was not changed anymore (for the techies: the LastChangeDate 🙂 ), giving a good estimate that this record is not used any longer.
Nevertheless, this is no guarantee that this record can be archived. Think of a long running warranty. It will not be changed over years, but still be in effect.
- Retention Period Before Deletion: Normally defined by legal requirements to keep the data for verification reasons.
This period starts when the data was deleted locally.
I’ve been asked about the maximum and minimum values for the retention periods.
Minimum is easy: 0. This means every record is a candidate for Archiving.
Maximum is harder: Technically it is MAXINT (more than 2 billion days), but the calendar system will fault on such a big number.
Therefore we limit it to 50.000 days, which corresponds to more than 137 years.
So, the Move2Archive job collects all “old” records and archive them. These candidates are handed over to some business checks individually defined and implemented by each Business Object. In most cases they check at least for status values like Completed, Closed, …. Of course they can be even more sophisticated.
Update: for some Business Objects these business checks can be disabled.
They are explained in more detail in this separate blog.
The business check returns not only the verified instances, but add also potential archiving relevant dependent objects (RDO) like pricing documents for quotes. These RDO can not exist by their own. They need the verified instance as a kind of anchor. Therefore we will collect not only the data of the anchor object but also of the RDO and place them together in the archive file. They will also be deleted together, from the local tenant as well as from the archive.
This first part of the job “Collect and Verify” runs for at max one hour. After that time (or earlier if all candidates are verified) these verified instances are handed over to a second part “Archive and Delete“. The experience shows us that we can verify between 80.000 and 120.000 instances during this hour.
Now all data of a given verified instance together with its RDO is collected, transformed into JSON, placed in a ZIP file, and sent to the archive. This includes also attachments.
BTW: the “archive” is currently an S3 bucket in the resp. Data Center. This bucket is not direct accessible.
Next we will update the Archiving status of the objects to Archived, so you know that the Archiving process has been finished.
Finally the instance will be deleted from the local tenant as they exist now in the archive.
This will speed up the standard search as well clean up the result list.
This second part of the job runs until all handed over instances are archived and deleted. Normally this is achieved before the next day. But if this second part is still running when the first part of the job is scheduled the next day, this new job will stop immediately to let the job from the day before finish.
This means from now on you will find the archived data only in the Search Archive (Work Center Administrator –> View Archiving).
Some details :
- Keep in mind that you need any search criteria (even a asterisk) to trigger the search.
- You may also filter by object type to narrow down your result list.
- By clicking on the link the Quick View will open.
From here you can navigate to the Thing Inspector.
This will also be the place from where you can restore the data. The data would be transformed back to the local database, the Archiving status would be reset and the data in the archive would be deleted.
Furthermore from here you can also download the archived data in a JSON format to your local hard disk.
As the Restore action is executed on the values of the archived data, there are concerns that restored data would not fit to the current data due to some realignment runs executed in the time between the archiving collected the data and now.
Therefore this action is currently disabled on some UIs,
If you want this action to be enabled for you, please create an incident on this topic. You will need to sign that you are aware of the above mention risk.
The DeleteFromArchive will delete the last remains of the record. After this job no restore would be possible. The data is gone.
Based on the comments below and other questions, let’s have a look at the impact for you as customers:
- Visibility of the Archiving status on the UI
There are several possibilities which may differ for each Business Object:
- The field is added to the UI and visible
- The field is added to the UI marked as “pers. hidden”
Here you need to switch to the Adaptation Mode to make it visible (maybe limited to a dedicated Business Role )
- The field is not available
Please request this field from the resp. application
- The field is added to the UI and visible
- What is the (visible) effect of the retention periods / executed jobs?
- If the data fulfills the business checks it is deleted. So you can only find it via the Search Archive.
- The data cannot be found anymore
- Is there any information about the result of the jobs?
- You can (re)use the following UI to detect, which object types are deleted:
- Work Center “Administrator”
- Work Center View “General Settings”
- Area “Data Management, View deleted data”
- You can (re)use the following UI to detect, which object types are deleted:
In the resp. OWL you can restrict the search on the objects which you have been activated for Archiving as well as the date “Deleted On”.
Important: The column “Deleted By” must be empty as Archiving is using a technical user which will not be shown.
- Any explanation why an object has been archived or not would depend on the business checks from each application. There is nothing available for that.
- The “Changed On” date in the Search Archive result list is the only information which tells you how long an object is in the archive. From this date and the Retention Period Before Deletion you can determine when the object will be deleted from the archive.
If there are requirements from your side for Archiving (I am sure, there are some 😉 ) please report them in the Idea Management, so we can collect and prioritize them.
Some more information on the performance optimization (sorry, very technical):
- We noticed that more Business Objects need to be verified if they are archivable. Therefore we doubled the runtime of the job Move2Archive and we tripled the number of processes used to send the data to the archive.
- As all SAP owned validations are disabled for Archiving, we did the same for the PDI validations.
- You may want to check inside your PDI solutions for the value of the element “ArchivingStatusCode” at the resp, Root nodes and don’t execute the PDI code if the value equals to 2 (“ArchivingInProcess“) or 3 (“Archived“)
Excellent write up. Thanks, Horst!
Thanks Horst !!!!, for the detailed explanation of the Archiving feature.
It could not have been explained better! Thank you Horst as Always.
Is this function available in the professional edition of C4C?
It's available in C4C in general without any specific restrictions or customer settings.
How GDPR is managed during the archiving process?
The access to archived documents will be logged (Read Access Log (RAL)).
With the next release 2105 the archiving is also honoring the Business Partner Deletion Block. This means that a document which refers a Business Partner (e.g. as Sales-To-Party) will not be archived at all and any archived document will not be removed form the archive.
thanks for the information!
We still have some questions with regards to the archiving:
1. It talks about an archiving status --> can we see that status anywhere at the object itself or only by knowing it also shows up in the Archiving area?
2. Currently the blog says that objects that are in the archive can only be recovered by SAP TIcket, but in the future via an 'Action'. We haven't seen anything, yet. Is that already within this release? How is the process of recovering objects out of the archive thought of?
3. - Retention Period: if e.g. it is 2 years and my objects wasn't changed in the last 2 years, then this object should appear in the 'Archiving' section. Currently (and also in the future?) it is there in both worlds, in the normal search as well as in the Archiving section --> Is that the correct process also in the future?
- Residence Period: Does this period start with the date that the object was sent to the archive first? So e.g my object is older than 2 years (retention) and then it is being put in the archive and left there for e.g. 30 more days (Residence period?) before it is actually deleted from the 'normal' search and only exists in the Archiving area? In the blog you talk about that this residence period will be removed in the future releases. What does this mean then? That the retention run will directly put the objects to the archive without having them in both worlds for a period of time? Is that correcdt?
- Retention Period before deletion: Objects older than this timeframe (last changed) are going to be deleted completely from the normal search as well as the Archiving area with no possibility to retrieve them back, correct?
4. Logging We can set the retention/residence and deletion periods. But we do not have any overview at all what happened at which run. (like we had e.g. in the SARA in OnPrem)
- Will there be a logging functionality so that we at least can see e.g. the amount or list of IDs that were archived that day/run?
- Will there be a logging functionality for the cases that could NOT be archived with an explanation why? Currently we have many cases that we would expect to be archived, but they are not. Without a logging we have no possibility to check why these are not being archived. (cases, where the last changed date is old enough, status in Completed and no document flow….is there more criteria in the background?)
Many Thanks for any info on the details.
Based on the long text, I see that you took archiving seriously. Thanks for that. 🙂
This depends on each application if the element is visible in the UI or even as personal hidden. You need to check in the UI and - if required - ask the application to add that element.
The framework itself already provides a generic functionality to read the data from the archive, transform it into the database format and inject this data into the resp. tables.
This functionality must be triggered by the applications. The current design foresees an action which is personal hidden and can/shall be added by the customer to specific Business Role which can be assigned to dedicated users.
As the realization of this design is with the applications it might be a good idea to push this via the Idea Management by you 😉
After 2 years your document is a potential candidate for archiving. Now the business checks described here in this blog determine if the candidate can really be archived.
The archived document is visible on the regular search only
It starts with putting the document into the archive.
If we have the Restore functionality available for all archived objects we do not need this period anymore and any document moved to the archive would be deleted from the local database in the same step.
During this period the document is still visible on the regular search only.
If a document is in the archive for more time that this period defines it will be deleted from the archive.
During this period the document is visible only in the archive search.
After this period the document can not be found any more.
Currently nothing is planned here. 🙁
Please describe you requirements in the Idea Management as I can read you have a detailed point-of-view here.
Thanks for the great blog!
One question about depersonalization.
Once a Ticket has been archived it cannot be changed any more and also depersonalization is therefor not possible. By law we need to depersonalize however whenever a customer/consumer is asking for that.
Is there any option to still do this?
Ronald van Beurden
I don't know how depersonalization for Ticket works, so I am not sure if the following assumption is true.
We read the data from the archive and place it into the buffer from which the UI is filled.
If there is not only a depersonalization run but also an in-situ depersonalization for the data retrieved form the buffer, this may work.
But not for a run.
Hi Horst ,
Thanks for the detailed blog on Archiving .
I had activated Archiving for 3 Business objects (Tasks, Sales orders, Tickets ) 10 days back in our system , But till now Archiving Field is not showing in Administrator WorkCentre , I can see the deleted data in (Administrator -- General Settings -- Deleted data)from the day i had activated but I cant See Archiving field .
Can you please let me know whether we need to activate any settings to get it .
I don't get what you mean with "Archiving Field is not showing in Administrator WorkCentre".
If there are archived data it should be possible (and that's the only way 😉 ) to go via
That's all folks,
Hi Horst ,
Thanks for your reply , Issue is Archiving is not enabled in Administrator Work center
Is there any other setting we need to do to enable Archiving field as it is not displaying anywhere .
You need to assign this WorkCenter View to your user.
Thanks Horst , It helps , I had assigned to my user now as it is not done automatically .
Thanks for the detailed blog. I had a question related to Sales quote. We have to delete few opportunities and sales quotes from C4C. Some of the sales quotes are in Submitted status. Ideally by SAP standard, submitted quotes cannot be deleted from C4C. If we configure the quote for archiving based on the last changed on date, then will these quotes in submitted status get archived and eventually be deleted. Please share your thoughts.
Did you have already a look into the other blog: https://blogs.sap.com/2020/12/18/archiving-what-objects-what-are-the-criteria/?
There are the criteria listed, which allow archiving.
Furthermore, we have plans to simplify - at least for the sales documents - the criteria.
Thanks for the quick response. Yes I went through both of your blogs related to archiving. For sales documents, below is mentioned.
So as per my understanding, even if the Retention Period Before Archiving (eg.2 years) criteria is met by a sales quote, if the quote is in Submitted status it will not be archived. Is my understanding right.
I have some questions on this topic.
1. In the list of business objects for archiving I do not see Leads and business partners (Accounts, Contacts).
a. How do we enable these for archiving or are these not available for archiving at this time?
b. If these are not available for archiving, are there any other SAP tools to archive/ delete this data for instance using the dataworkbench etc?
3. Once archiving is enabled, since the jobs run daily early morning:
a. Does this mean the jobs will run every day starting from the day archiving was enabled?
b. Do we need to reach out to SAP Support for any step in the archiving process? Or once enabled and configured, it will run every day without any other action needed by C4C Support?
c. What time in the early morning do the jobs run?
d. Can we control when we want the jobs to start?
e. How can we monitor the status of those jobs?
4. Is it possible to restore the archived data by ourselves using the view archiving option or do we reach out to SAP support for this through an incident?
5. Once we enable archiving, it cannot be undone. Can we at least change the ‘retention periods before archiving’ and ‘retention periods before deletion’ or those cannot be changed either?
6. Since we have never archived data, is it possible to get an estimate of how long it will take to archive when it runs for the first time? OR do the jobs run for a specific time period every day for e.g. 1 hour and it will archive/ delete all that it can within that hour and then pickup from there on the next day
7. When I turn on archiving in business configuration in the test tenant, will the archiving jobs run in the test tenant or is this only for production data?
1) Leads and business partners (Accounts, Contacts) are not supporting archiving. Please add them in the idea management. 🙂
AFAIK Data Workbench may be an option, but I am not sure.
3.b) No interaction from your side required / needed
3.c) About 1 AM server time
3.e) Not possible
4.) As mentioned above:
if you want this action to be enabled for you, please create an incident on this topic. You will need to sign that you are aware of the above mention risk.
5) You can always adjust the retention periods
6) You're right, the jobs run for a dedicated time (currently 1 hour) every day and continue the next day with their work.
7) You need to activate archiving per tenant
Thanks a ton Horst. This helps a lot. Appreciate you prompt response. I just have 2 follow-up questions from the business.
1.) You're a good guesser 😉
2.) The data is taken as it is and it is not possible to alter it in the archive.
But if it displayed again, the normal UI is used. If this UI supports anonymization, this will work
LOL. Thanks much again.
How about the Access Restrictions on Archived Object (Opportunity). If Archiving WC is enabled to user all Archived Oppt will be visible or the standard Oppt Access Restrictions will be Applicable?
Thanks for the Archiving Updates!!
The access restriction is the assignment of the Archiving WC to the dedicated user. 😉
So, every archived entity is visible to this user.
In the next step the regular UI is used.
I expect that the checks of these UIs still work and are evaluated.
A few more questions have cropped up after we enabled this in the test tenant. I created an incident for this but it fell of deaf ears.
1. We had enabled archiving in the change project in the test tenant. However, the archiving did not get enabled in the backend and SAP had to do it manually by applying a hot fix. Question is, when I merge the change project with production, will the archiving functionality get enabled automatically or will SAP have to apply the same hotfix that they applied manually in the test tenant, to production as well?
2. When I look at the archived opportunities under workcenter Administrator --> Archiving, I see the archived data list. But when I click the link to open any one, I do not see any data. Question is why is the archived data not visible? Further if I open the detailed view of the opportunity, it says you don't have authority to display this opportunity. Question is what authorization am I missing? Or is this by design that data in the archive cannot be viewed?
3. If for some reason, we do not want to activate the archiving functionality in Production, but we have already enabled it in test in a change project. if I cancel the change project in the test tenant, will that deactivate archiving in the test tenant?
It is not expected that you will use a Change Project.
You can go direct to "Business Configuration" -> "Overview" and search for "Archiving Configuration". Here you activate the Business Objects you want to be archived and - if you want - maintain the retention periods.
Save and that's it.
I am not a BC expert, but I would expect, that a Change Project should contain this setting.
But your comments above shows it doesn't. 🙁
Reg. topic 2: We had an unexpected change in the JSON format which is used for archiving. Please provide me (via PM) the incident and I will take a look if this is also true for your case.
Thank you for the answers to question 2 that it was due to some change in the JSON format. I looked now and see that it shows the data.
For 1 &3, as a best practice and based on SAP recommendations, we never enable any configuration directly in production. We always do it in a change project and merge that to production. So even for archiving, we have created a change project and enabled the archiving there. However when we had enabled archiving in the change project in the test tenant, the archiving did not get enabled in the backend and SAP had to do it manually by applying a hot fix. So my question is, when I merge the change project with production, will the archiving functionality get enabled automatically or will SAP have to apply the same hotfix that they applied manually in the test tenant, to production as well? I only ask because I want to know for sure when the archiving will get enabled because we are doing archiving in C4C with reference to ECC archiving and the dates should match otherwise we run the risk of one system archiving data while the other, does not.
As always, I appreciate your prompt replies.
I sent your incident to the Business Configuration team because I can't answer it.
As mentioned before, the way other customers chose is to activate archiving direct in a test tenant.
So, let's wait for an answer from the BC team.
Thank you Horst. Appreciate it. The more I think of it now, my question was more related to the hot fix than BC itself. I'll wait to hear from BC team anyways as that will help as well.
Regarding the hotfix, when I activated the archiving functionality in the test tenant, it did not activate and I wasn't seeing any data in the archive. I created an incident with SAP support and you had replied with the below
"Due to a wrong setting archiving was not correctly executed. HotFix 1000021382 is already provided to solve this issue. I have pre-implemented this HF to have archiving active immediately. You can see archived opportunities already."
So my question is, will this hotfix have to be manually applied to production as well or will this happen automatically as soon as I activate archiving in the production tenant?
The HotFix 1000021382 will reach your production system the next weekend. 😉
Dear Horst Schaude,
Please suggest a possible way to control the high volume of data that is currently present in the ticket (C4C).
There are nearly 1 crore tickets are present in the system. As a consequence of this, it can impact system load and performance.
As a solution, we have worked out ticket archiving functionality, but in this case, after a certain period of time, it's getting deleted from the system. We don't want to remove the data from the system after 5 to 7 years at all. (We want to keep the entire set of older data somewhere)
Kindly suggest any other way to reduce the high volume of tickets without losing the data.
If you set the second retention time "Retention Period Before Deletion" to let 's say 365.000, (it is possible to use max int, which is in the millions) the data will stay for 1.000 years in the archive.
Does this help?
Thank you for your quick responses!
If the retention period before deletion is 1000 years, this functionality is really very helpful.
You can set this value from 0 days (=> delete it immediately) up to 2.147.483.647 days (which is - more or less- keep it forever 🙂 )
It's your choice to adapt the SAP-delivered value in the Business Configuration for Archiving.
Thank You Horst, Got it!
At the moment, this is not possible. 🙁
But I suggest to wait for the next release. 😉
Thank you so much. I will follow this topic:-)
Have a nice day.
One more question or even two:)
Thanks in advance.
There is the possibility of the Restore action which needs to be enabled via an incident (as the "old" data may not fit to the current org and therefore we need a consent that no incidents are raised because of such mismatch)
The business checks are defined (and implemented) by the resp. application. No need / no possibility here for the customer.
I just setup archiving in our test tenant and I'd like to know how I can see the job Move2Archive?
I'd like to know I see when is trigger, how many objects was checks etc.
This is not possible for the customer on that detail level.
Like described above you can see the result in
I have a one more question from the business:
"Is it possible to bring back single objects which were already archived and disable archiving rule for this objects?"
Thanks in advance.
"Bring back" we already discussed:
"disable archiving rule for this object" instance is possible. The ArchvingStatusCode has the value "NotArchivable" (numeric 4). If an instance has this value it will never be archived.
But I don't know if the resp. element is accessible (either via UI or PDI).
As bring back object I mean not all Visit for example but only one and then make this one “NotArchivable” - but as I understand your answer was for that kind of scenario.
Thank you once again:-)
Can we link Archicing to any external DMS or would it be only within Datacenter of the tenant?
The customer can only decided which object shall take part of Archiving.
Everything else is decided / done be SAP.
So in short: no.
Thank you very much for your informative blog. It was crystal clear and incredibly helpful!
I do have one query regarding the behavior of archived tickets. Specifically, what occurs if a ticket is archived, its status is "closed," and a customer responds via email?
For instance, on our system, such emails are directed to an unassociated emails.
Would this be the same for an archived ticket?
as the Ticket is physically deleted and can only be accessed via the Archiving OWL, the system would direct this e-mail to the unassimilated e-mails.
Thanks for your prompt answer.
I am sorry, an other question came to my mind.
I realised by doing some test on the archiving search that the visibility right were removed.
User A doesn't have visibility on Ticket A
Then Ticket A is archived
In the end, User A have visibility on Ticket A when searching in the archives
Is it a normal behavior ?
The archiving OWL has no (object-) specific access control. It should only be assigned to special / authorized users.
Therefore the user A (if assigned to the Archiving OWL) will see the ticket in this OWL.
Even when navigation to the ThingInspector UI this is still seen as an archiving activity.
But after the Restore the ticket is fully back in the process, including access control.
So if I understood well, when a ticket is archived, access control are removed.
But once it is restored, access control are back.
Is my understanding right ?
I would prefer "access control is not applicable" 😉
But in principle you're understanding is correct.
Alright thanks for the confirmation. 🙂
In our case, there are sensitive data that should not be shared in between Sales organisation.
We can't activate the archiving process in this case.
Do you know if there is something planned regarding the application of the access control in the archives ?
The data needed for the authorization check is - at that point in time - still in the blob coming from the archive.
So, executing the checks will come too late. 🙁
Isn't it possible to assign the Archiving OWL only to special user who can have access to several Sales Orgs?
Similar to the user assignment for the Transport Management.
Thanks for all these clarification.
In our case it is not possible to assign the Archiving OWL to users that have access to all the sales org. Only the Admins have this right and it would be too complicated for us if we are the only one that have access to the archives.
We will be careful of the archiving process enhancement for the next release.
I understand your arguments but do you think there will be that many request to access archived data?
For example: The restore functionality (which I considered very important) is seldom requested.
I even got the answer from a customer that they don't expect to use it all.
So, for me, I will nowadays always ask the customer and don't trust my expectations. 😉
It depends how long we set it up, but our user told us they can search a 2 year old ticket for exemple.
Is archiving has a real impact on the performance ?
We don't want to implement a process that involves more issue than before.
The retention period during which tickets will at least stay in the DB and are accessible for the regular user can be set from 0 days to 50.000 days. 🙂
So, if you user needs data up to 2 years old, set it to 730 days and they will still find them in their normal UI.
IMHO the question is:
Which in turn involves somebody who has access to the Archiving OWL
That's the question (and the surrounding conditions) you need to ask. 😉
Yes, that makes sense. It could be good for us to set up the archiving flow with a retention before archiving that would be 2years.
And for data that is longer, we could (Admin) be the one searching for archiving data for the user.
I have 2 other questions:
Sorry, I didn't have any measurements about this 🙁
Just imagine if you have 1 billion tickets in your system from the last 10 (or more) years and by the above mentioned settings you will reduce this number to appr. 200 million (or even less).
I have used the search also with the change date. 😉
Thanks for all your support.