Invalid DDIC Objects During System Copy
First, a few caveats as I tell you my story of woe and redemption. Some of the techniques I will describe should be used only with extreme care and definite understanding of what you are doing and potential consequences. The story you are about to read involves a sandbox system (just in case), with plenty of backups. I always recommend trying out anything a little dodgy in a sandbox first, of course. Additionally, I should point out that the system involved was on a release that was out of maintenance (even extended maintenance), so obtaining official SAP support for this issue would likely be problematic (not to mention take a lot more time); otherwise I would of course recommend seeking official support. I needed to get this system upgraded, but before I could do that, I needed to get it copied, migrated onto new hardware. Starting with the sandbox.
The system involved is an old SRM 5.0 system (out of maintenance) that I am about to upgrade to SRM 7.0 (in maintenance). At the same time, I am migrating the landscape onto new hardware (the old hardware is ten years old), and taking advantage of new database compression capabilities available in SQL Server 2012 (the old system is SQL 2005), as well as redistributing the database to a larger number of disks.
In other words, I need to perform a database-independent (R3load-based) homogeneous system copy. This is the ideal way to evenly redistribute the database files, and it achieves the compression in a single step as well.
So, I’ve setup my sandbox environment, but I can’t justify production downtime to make a copy for the sandbox. No problem. I start with an ordinary database-dependent (backup/restore) system copy, which involves no downtime on the source system. I just make a copy of the most recent backup, restore that to my sandbox, and run SWPM for the target system installation. Easy-peasy.
After ensuring that the sandbox system is running fine post-copy, now I want to export it and re-import it to achieve the disk layout and compression.
And that’s where my troubles began.
The First Export
I prepared my system as usual for the export, de-scheduled all the jobs, shut it down, and fired up SWPM (again, since I had just used it to install this system). The export ran fine, quicker than I expected, but of the total 47 Export Monitor jobs, 1 of them failed. Oh oh. That’s not supposed to happen.
I drilled into the logs and saw that in one of the export packages there was an “invalid object name” for table /CCM/CUST. This is a table related to the now-defunct module Catalog Content Management, which briefly enjoyed status as the go-to catalog solution after support for Requisite BugsEye dried up and before SRM-MDM launched onto the scene. The only trouble is, we don’t use CCM in our system. We briefly experimented with it a number of years ago, but never went anywhere with it. I know for certain that we don’t need this table. However, I cannot successfully complete the export with this error.
Active DDIC Source Missing or Incorrect
Curious, I go take a look at the original source system, i.e. my production system. In SE16 or SE11 the table doesn’t exist. Hmm.
SE14 now starts to show the real puzzle. Here the table does seem to exist, or at least SE14 is able to display some data about it. It has a status of Active but also Does not exist in the database.
Ok, let’s check consistency of database and runtime objects. Sure enough, Table is not created in the database, but I expected that. The runtime object check, however, returns Active DDIC source for /CCM/CUST missing or incorrect. I can display the runtime object, but it’s not consistent.
I check the object log, and, oops, the log does not lie. The table was deleted, six years ago, via this very utility, SE14, by me.
Oh yeah, I kind of remember that now.
How exactly we got to this point is not really important, but after our abortive attempt nine years ago to install CCM, we simply abandoned it in place. A few years later, however, we upgraded our SRM system from 4.0 to 5.0, and — digging into the deep memory well here — we did a “passive deletion” of the CCM component during the upgrade. I don’t now recall all the details, but the passive deletion was not entirely successful, and ultimately I finished off deleting the tables involved via SE14. This worked, and the upgrade was a smashing success.
Fast forward six years to today, and yes, those tables are gone, but it appears that the SE14 utility did not completely remove the nametab entries for the tables. So, there is no object on the database for them, and there is no active DDIC runtime object, but there is still a data definition, or nametab, for each one.
How to Proceed From Here
SE14 is not very useful from this point. As the table doesn’t exist, there’s nothing for the utility to delete, nor will direct database tools be of any use. The only function active, other than consistency checks, is Create database table, but that’s not exactly what I want to do. Still, maybe if I create the table, I can then re-delete it, and this time it will work? Worth a try.
But I’m not doing this in production! I’m not even going to do it in DEV. This is what I have a sandbox for, after all.
So, I kill the SWPM process and restart the sandbox system. Sure enough, I see the same behavior in SE14 as I saw in DEV and PRD. I create the missing table. It works!
Or, does it? Not so fast. It actually doesn’t work, because the runtime object is still missing. So now I have a mismatch between database and runtime objects, and I’m no closer to a solution than before. In fact, I’m farther away from one.
I delete the table again, and we’re back to square one. Despondency sets in. How can I get at the inactive or inconsistent nametab source for a table that doesn’t exist in the dictionary?
Well, it turns out there’s a handy little function module, DD_SHOW_NAMETAB, which indeed shows my rogue ghost table. Great!
And, there’s another, even handier, function module. Dare I mention it? Yes, it is DD_NAMETAB_DELETE, and it does exactly what it sounds like. I give it a try. I put in my table, /CCM/CUST, and the result seems rather underwhelming. However, now no nametab shows for it, and SE14 now says this table doesn’t exist at all. This is promising! Get another backup, shut down the application, and restart SWPM for a new export.
More Than One
Ok, half-failure. The export did not fail on /CCM/CUST this time. Instead, it failed on /CCM/C_ASPT. It looks like there are a bunch of these ghost nametab entries out there. I need to see if I can find them all, because this one-at-a-time business, trying a full export each iteration, will keep me busy until I’m grey(er).
SE14 does not find any /CCM tables in a search. The nametabs exist, but apparently not in any search help. No utility seems to be able to successfully search for /CCM/*, nor for that matter for /CCM/C_ASPT, even though I can see /CCM/C_ASPT in SE14 if I enter the name directly. Nor does DD_SHOW_NAMETAB successfully work with a wildcard. It’s a one-at-a-time tool.
Going back to the logs from my SWPM export, however, I find a file, SAPAPPL1_1.TSK (SAPAPPL1_1.log is where the failure was recorded), which is, as its name implies, something of a task list for that migration monitor package. And, it has a handy list of every table that package was supposed to export, including almost three dozen /CCM tables.
Using my list of /CCM tables, I checked each one with SE14 to be certain that there were noexisting database objects and that all of the runtime objects had missing DDIC sources, yet all of the nametabs existed. Once certain that all of them were inconsistent, I ran them all through DD_NAMETAB_DELETE, and, again, restarted my export (third time!).
I am happy to report that this resolved the problem, and now you know how I spent my weekend.
Hey, great blog Matt.
I'll be sure to bookmark this for future reference. But I'm sorry this is how you spent the weekend 🙁
All part of the glamorous life of a Basis admin. 😐
I KNEW you guys had all the fun!!!! 😛
Sounds more like a "new implementation" than an "upgrade" at this point. haha
Hmm, not sure we're ready to take all that on. Although SRM 7 is significantly different from SRM 5, so this upgrade is a lot more work than previous ones.
No...I really meant....how hard would it be just to treat it as a straight fresh install versus an upgrade...and then just port/load data as needed. Seems like it would be less hoops to jump through....but then, that's why you BASIS and DB admins get paid all the big $$$$$$ (while folks like me just get paid scraps to make your life hard haha)
I suspect it would actually make it a bigger project, in this case. Ultimately, that would be a decision for the lead functional analyst responsible for the system, and he hasn't come close to suggesting anything like that. Since he's the one with the biggest workload for the project, I think it would be his call.
Thanks for sharing. I did encounter above issue when performing CRM upgrade and migration to HDB year ago via DMO and as advised by SAP support, using the same method to remove inconsistent nametab and table.
Besides DD_NAMETAB_DELETE, there's DD_TABL_DEL but just to use it with extra care and own risk.
Thanks, Nicholas, always good to know I'm not completely out on a limb!
"Matt Fraser A Series of Unfortunate Events". 🙂
Reminds me of an old experiment with a custom infostructure that we deleted but then had to re-create just because some code didn't get updated with deletion and SAP was screaming syntax errors for non-existent table. No help from SAP. I bet years later someone will be scratching their head why on earth this empty structure is there. Just hope they don't delete it. 🙂
Wonderful Matt !!! You made my day !
I was struggling to completely delete tables TASUVCAT, TASUVEXT, TASUVSVD, and TASUVSVO... due to SE14 drop 9 year ago ... by me... following note 623723, I assume.
Now I can continue with my unicode conversion.
Thanks a million !
Glad my old blog post is still helpful today!
You're my savior Matt! I've been trying to figure out the error in my system export even SAP Support cannot provide me straight answers.
Moving on with my System Import, hope everything goes smoothly this time!
Thanks, Jared. I'm glad to know this helped!