SLD CR Content Repair
You know the drill: every so often you need to download and import the latest CIM model and CR content into your SLD to keep things up-to-date. You meticulously follow the directions from Note 669669 (Update of SAP System Component Repository in SLD) every time, and you never have problems…
… until the day you do.
“Dingoes ate my SLD!” or, “How to recover when the unthinkable* happens”
(* “Unthinkable,” in this context, means “I downloaded and tried to import something that was apparently corrupt!”)
Yeah, this just happened to me with the import of CR Delta 11.0, aka “SP13 for SAP CR Content up to 2014” (this is the one that changes you from 10.x to 11.0 and preps you for the “Content up to 2015” deltas). At the end of the import I saw “3 errors occurred during import.” Catastrophe! All three errors were identical, something to do with Business Planning and Consolidation (a component we don’t even use) and a violation of how many of one type of instance can be associated with another type of instance, etc etc etc. I was pretty sure it was something bad in the delivered content delta zip file, and who knows, it might just have been fixed by the next delta I was about to apply, but you know the protocol: don’t apply any more imports until you know your SLD CR content is consistent.
But is it really corrupt?
Step one: find out if we really have a problem. So, you refer to your handy-dandy Note 1939864 (Check if SAP CR Content is corrupt in LMDB or SLD). You check to see if you have a high-enough Support Package in your Solution Manager system (sp11 or higher for SolMan 7.1), and if not, then apply the Note.
Once certain the Note or Support Package is applied, go to transaction SA38 and run program RLMDB_CR_CONTENT_HASHER. It’s time to hash some stuff up. Anyway, the program is pretty straightforward: check the radio button for Use remote SLD, enter the RFC Destination for the SLD in Source, leave the Namespace field alone (unless your suspected corruption is in a non-default namespace), and copy/paste the Expected Hash from the table in Note 1939864 for the CR content version you are checking (as reported by your SLD). In the example below, the hash represents content version 11.0.
The program kicks off a background job called RLMDB_CR_CONTENT_HASH. It will take a while to run (in my case, checking a remote standalone SLD, it took a little more than an hour). When the job is finished, check the job log. If you see something like:
Hash 864497b8-171c-7981-4c7f-16ffc5fccb90 does not match expected hash 123cb9c8-5886-4cf3-f01f-7ab9239981ec
well, then you just received the bad news. Your SLD content is corrupt, and you need to repair it.
Yes, it’s really corrupt. Now what?
All is not lost! This is not a Robert Redford movie! But, hopefully the support package level on your SLD meets at least the minimums specified in Note 1093168 (Repair of SLD CR content). If not, it’s not the end of the world, but some parts of new content may not be taken into account. However, the basic repair should still work.
Your next step is to create a reference namespace in your SLD that contains only the SAP-delivered content (not your landscape data), and then compare the reference namespace with your active namespace. You can do this online, but you do need to take into account a few restrictions mentioned in the Note, namely don’t try to import any CR content updates while executing the repair (duh, the content’s corrupt, and we know better than to try to import a new delta over a corrupt one, right?).
This is a good time to ensure your SLD database has plenty of free space. About 50% free should do it.
Create a reference namespace
Logon to your SLD (http(s)://<host>:<port>/sld) and select Administration -> Namespaces. Click on Add and give your new namespace a name (e.g. “sld/reference”).
Select the reference namespace as shown above, and select Administration -> Import. On the next page, make sure that the Target Namespace is the reference namespace and not your active one, then select the radio button for From Server and click Import:
You’ll notice that you’ll be able to import the “default” model for the release of SLD you have (in my example, this is on a NetWeaver 7.4 SR2 SLD). On the next screen, click Continue Import. The model import should go fairly quickly. When it’s done, if you return to Administration -> Details -> Data you should see something like this:
Now go back to Administration -> Import like before, again ensuring your target namespace is still reference, and again import from server:
In my case, this imported content version 9.0 and took just under half an hour.
Update reference to same model and content as active namespace
Now that your reference namespace is at model version 1.6.38 and content version 9.0, it’s time to get it up to model 1.6.46 and content 11.0 (or whatever is currently in your corrupt active namespace). This procedure is just the same as any SLD content update. You’ve already downloaded the new model update and the appropriate latest content update (because, well, you used them on your active namespace, didn’t you?), so all you need is any required intermediate content updates.
Check Note 669669 to find out what’s required to get from the current content version to the updated content version. In my case, to get from 9.0 to 11.0, I’ll need to download the intermediate update that gets us from 9.x to 10.0, which we find under Support Packages and Patches -> SAP Technology Components -> SAP CR CONTENT -> SAP CR CONTENT UP TO 2013 -> CRDelta91615_0-20010858.zip.
Back to Administration -> Import and this time check the radio button for From Browser and select the model update (the file that starts with cimsap…).
Repeat the import with the intermediate CR Delta to get from 9.x to 10.0 (or as appropriate for your situation), and the “final” delta to go from 10.x to 11.0 (or, again, as appropriate for you). This, by the way, is where you had errors before, right? So if this update fails again, then it’s a pretty safe bet there’s something wrong with the file you downloaded, and it could be time to submit a Customer Incident to SAP to let them know. If it doesn’t fail (and we hope it doesn’t), then the error was something else (cosmic rays, perhaps?), and we’ll continue with the repair process.
When you’re finished, you’ll have details something like this:
Don’t worry that the number of Instances and Associations aren’t the same between the active and reference namespaces. Remember, your active namespace has all your landscape data in it, whereas the reference namespace does not.
Start the repair!
Ok, that’s a lot of time spent creating a reference namespace that doesn’t do much. Now it’s time to make it useful. Be sure to get a backup of your active namespace before continuing (select the active namespace, then Administration -> Export -> Back Up -> All Instances).
Select Administration -> Maintenance -> Repair Software Catalog (or, if you’re on an older SLD, such as that included with SolMan 7.1, you might need to enter it directly as a URL like http(s)://<host>:<port>/sld/admin/CRRepair.jsp). Make sure your active namespace is in the Productive namespace field and your reference namespace in, well, obviously, the Reference namespace field, then click Start Repair.
The repair process will touch almost 1.6 million objects, so your first thought will be “wow, that’s going to take a long time.” However, it goes faster than you’d expect (faster, apparently, than an import of a fraction of the number of objects); mine completed in a little over five minutes. Afterwards, you’ll get some interesting statistics:
Check the result
After the repair is finished, go back to your Solution Manager and re-run RLMDB_CR_CONTENT_HASHER as you did at the start of this process, again using the expected hash for your content version from Note 1939864. This time you should get a match.
Once finished, you no longer need the reference namespace, so you can delete it if you wish. However, if space in your database is not an issue, you may choose to keep it… just in case.
Software Catalog Data aka CR CONTENT in SLD and LMDB
System Landscape Directory (SLD) | SCN
Performing Initial Data Import – Configuring, Working with and Administering System Landscape Directory – SAP Library
Note 669669: Update of SAP System Component Repository in SLD
Note 1939864: Check if SAP CR Content is corrupt in LMDB or SLD
Note 1093168: Repair of SLD CR content
Note 1891566: Repair of LMDB CR Content
Note 1751175: SAP CR content anomalies in SLD or LMDB
It is likely, after you are done, that you will need to repair the content in your LMDB as well. Wait until all synchronizations between SLD and LMDB are complete (you can check in LMDB Administration, or in SM37 in the logs for the most recent job SAP_LMDB_LDB_*. Then check the LMDB by once again running RLMDB_CR_CONTENT_HASHER, but this time choose the option Use local LMDB. The runtime will likely be about half what it was to check the SLD.
If you see the message in the job log afterwards about mismatched hash values, then your next step is to run RLMDB_SAP_CR_REPAIR_LMDB in background, as described in Note 1891566. As before, ensure there are no CR content updates made to the SLD during the repair.
Great post Matt !
One question Matt, (From someone who does not know this tool / module):
It is considered good practice to export this file mentioned (CIM) daily, weekly and / or monthly?
I don't think it's necessary to do regular exports of the namespaces in your SLD -- it's more of a precaution before undertaking something like a repair with the potential to change (read, ruin) a lot of data at once. It's likely to be easier to re-import your export than to do a full database restore. However, for normal ongoing operations your normal database and server backup mechanisms should be sufficient.
If you meant the imports of CR content updates, that depends. I typically import the latest CIM model and delta update just before starting any support package update or release upgrade project. So, it's not really a set schedule like weekly, monthly, etc, but as needed (however, at least once per year).
If are doing Java developments in your shop, with NWDI and NWDS for instance, then you'll probably want to update your SLD content ahead of starting a new development, though it's not absolutely required.
I tilt my hat on a nice write up. Prologue, Epilogue and references. A often neglected art-form.
I once got bitten on the part of the anatomy that doesn't see much sunlight when doing a SLD update so I always take a backup before starting CIM or CR updates now.
Thanks for a nice contribution that has been duly bookmarked for a rainy day.
Dingoes stateside ? 😏
Amerjit, thanks for the very nice comments!
No, no dingoes here, but I was in Australia somewhere around about the time the film came out about the infamous Lindy Chamberlain case, and that phrase has stuck in my head ever since: https://en.wikipedia.org/wiki/Dingoes_ate_my_baby.
We have a scenario where solution manager Java SLD content is outdated. We have to upgrade JAVA SLD to stablize system monitoring.
But our other PI systems SLD is also on lower level and getting synced to SOlution manager.
What will happen if we upgrade only Solution manager SLD. Is it going to current when PI SLD send Data to solution manager.
If your PI SLD is being sync'd from your SolMan SLD, then after you update the content in the SolMan SLD, after a while this should sync over to your PI SLD. You shouldn't need to manually or directly update the PI SLD content in this scenario, if I understood your question correctly.
Or is it the other way around? Between the two, one is the primary and the other gets its updates from the primary. In many landscapes, it's common to use PI as a primary SLD.
In our case, we avoided these questions and we have a single standalone central SLD, which is then used by both SolMan and PI (and also manages the rest of our landscape). So, we don't worry about SLD->SLD syncs, only SLD->LMDB.
You might do well to raise this question in the NetWeaver or Basis Q&A, rather than as a comment on a blog, however. I think you'll likely get more eyes on it, and more responses.