HANA MDC: Tenant crash while recovering other tenant on the same appliance

The blog post is to bring attention to an issue we have been facing on our HANA Multitenant Database container(MDC) setup

Background:

We have a Scale up MDC setup with more than 15 Tenant Database’s in non prod on SPS10

Part of quarterly release activities we refresh non prod systems from production MDC tenant backups

Until last year we had less than 10 tenants and the regular refresh was working as expected

Issue:

We had introduced more non prod tenants end of last year and during the next refresh cycle we started noticing a tenant crash while we were working on refresh of another tenant

A complete check of trace logs of the crashed tenant confirmed we had signal 6 errors exactly around the same time the other tenant was being refreshed

After multiple attempts to being up the tenant did not work, we had to involve SAP support to check the cause of the issue

Meanwhile we restored the crashed tenant using backups

Cause:
SAP Support took more than a month to identify the cause of the issue and another occurrence of the same issue while restoring a different tenant confirmed there was a correlation

SAP confirmed the following, when we have more than 10 Tenants on a single MDC appliance we will come across this issue(on version SPS11 revision 112.02 and below)

For example if we have 15 tenants and lets say the tenant with Database ID 5 is being restored using a backup of production tenant it will impact the tenant with Database ID 15 and this tenant will crash and fail to start up. Same issue would occur on tenants with Database ID 13 and 14 if tenants with Database ID 3 and 4 are recovered using a backup

Resolution:

SAP has addressed the issue in SPS11 Database maintenance revision 112.02 that released today 12-Apr-2016

Please find the link below for the same and the screenshot that confirms the issue in the note

http://service.sap.com/sap/support/notes/2300417

Please let me know if anyone has any thoughts or inputs on this issue and hope the blog is useful in understanding the cause of the issue and available solution