They say good news travels in pairs. Or is triplets? I'm not sure, but here's the concluding chapter of the Saga Of The Degraded Upgrade. In more ways than one, this is old news. That doesn't stop me from pulling out the Smith and Corona and blasting a few keys into caps. In parts 1 and 2, the path to the Big House (production) was laid out, sketchily, like a, well, sketch artist would. This go round, there was no going back.
My job wasn't on the firing line, pushing the uptimes and downtimes, it was running a longer term surveillance for any criminal data types that escaped the dragnet of a quality system upgrade and the post upgrade go live frisking. Not a stop-and-frisk, mind you. Let me back up. The upgrade caper was planned for a long time, and happened on a quiet February weekend. It's almost April now, so why the delay? No particular reason, just waited for all of the shoes to drop.
During the upgrade, technicians did their technical thing. Then, business people got busy. The "OK, move out" signal may have been a bit delayed due to the usual surprises, but nothing untoward happened. As in the previous cases, that was on a Sunday. Monday is the day when you can expect the "oh dear" moment, when either systems are unstable, or data has been misplaced, or the promised synergies uncork. Keeping my cool, I watched as various scenarios played out. No fatal instincts, no "let's roll back", no black helicoptered or jet-packed samurai from Walldorf. There were a few shoot outs, nothing worth putting on a wanted poster.
A few days later, I took a snap of the system fingerprints.
The blue area is the database server - other colors are unrelated virtual systems in the same frame. Friday, Saturday and Sunday show the load per the prior discussion, then Monday kicks it off with a bang. But a manageable bang.
ONE
The first tip for me was a missing data complaint. A bag job that worked silently for months fell over on the first try after ECC 6. Turned out it was an RSSTAT10. Meant nothing to me, although I had seen similar disappearances when the warehouse was burned down, I mean upgraded, a while back.
http://scn.sap.com/thread/488739
| "Report RSSTAT10 is available in ECC5, But In ECC6, Report RSSTAT10 is not available." |
This one was a goner. We could have fixed it beforehand if anyone thought about preserving the data. With no spare copy sitting around, there wasn't much point crying over that spilled milk. Next time, we'll know.
- 1005238 - Migration of workload statistics data to NW2004s
- 1006116 - Migration of workload statistics data to NW2004s (2)
TWO
The second report was not quite a missing data squeal, but practically the same. Dead interface. The wire racket boys had split, leaving me with an RFC call that was not too hot to handle. Fortunately, this wasn't causing major consternation in the dispatch office, meaning I got to handle the bugs in my own time, with my own bug-picking tools.
It looked like this on first view:
| /usr/sap/SID/SYS/exe/run/startrfc -3 -dSID -uUSER -pPASS -c999 -lE -hsap... -s## -gsap... -x...gw.. -t -FINCOMING -EPORT=XYZ -EPATHNAME=/.../data/statusidoc.dat |
Well, more or less. My Rollieflex ain't what it used to be. Or my Graflex neither. It was a command line, wrapped in a shell script, called from workload automation. Picking it apart, I delicately sensed the aroma of stale code. Also, dead code, since startrfc wasn't even in the kernel directory. A quick ABAP, er, APB, located the newer version in a suburb of exe/run, exe/run/nwrfcsdk/bin.
We could move it, or we could run it from there. It needed a short library booster shot (LIBPATH=$LIBPATH:.../nwrfcsdk/lib as I recall). Then, to the heavy detecting, er, debugging.
Forcing an error is sometimes a way to have a suspect cough up a confession, or for this case, a usage pattern.
Usage: startrfc [options] Options: -h <ashost> SAP application server to connect to -s <sysnr> system number of the target SAP system -u <user> user [...] |
What jumped out at me was the spaces between the options and the parameters. The old code ran together like the typeface on the city morgue. I played around with a few options, forcing the errors to be fewer and fewer until the worst symptoms were past, and the RFC connected to the development system. But I rewrote the code in a cleaner format, just because the options were swarming.
/.../nwrfcsdk/bin/startrfc \ -3 \ -d SID \ -u USER \ -p **** \ -c 999 \ -l E \ |
The tumblers clicked, and out came:
Error: EDI: IDoc '00099...' does not exist (please check data)
So, my work was done. Almost. It turns out ECC had another wrinkle. The command line worked, but a batch job died with a mysterious message. Well, maybe it's not dead. Maybe it's getting better.
Error: Password must be changed.
A few rounds with the Basis and Security teams, and Monday turned into Thursday. Finally, new passwords, case sensitivity checked, service access account type double and triple checked, and away we go:
| Changing the user type from a Communications ID to a system ID and making the password all CAPS fixed the problem in PROD. The job now completes successfully in schedule. Thank you all for your help! |
THREE
I'm just going to leave this picture up like a Most Wanted poster. I hit this caper in a previous chapter, and there's not much else to say other than combining a database upgrade with an SAP ERP upgrade can confuse the issue as to application code or database code (i.e. SQL optimizer).
FOUR
Though I intended to write a 3-chapter story of the degraded upgrade, there's always one that "got away". The tip didn't come in until recently, at least to the ABAP Detective desk. It's been on SCN a couple times, moved once so far by a Moderator, and has no stellar clues. I think we'll end up fixing it ourselves, in which case maybe there's a fourth part to this trilogy. It's been done.
- RSAOS_INT_METADATA_UPLOAD long running after ECC 6.05
- RSAOS_INT_METADATA_UPLOAD long running after ECC 6.05
Ta for now.
- SAP Managed Tags:
2 Comments
