Counting your Mistakes
This week I gave the wrong advice to a client and told them they had some additional steps to do as part of support packs for security activities. Fortunately, the only harm was a few unnecessary emails and egg on my face when I realised and had to send a correction through.
On the same day, I was also configuring something and rushing out the door to get home, so I was a bit quick to bundle my change and not check my transport before releasing. Luckily it was new functionality and I had time to fix the issue before testing commenced. But it was a few hours sifting through my build to find what I missed and put the correction in.
Both mistakes were obvious ones. I was annoyed and embarrassed with myself for being wrong. I know I’m the type of person who will shy away in a corner if I don’t know the answer out of fear of being wrong.
The reality is that we all make mistakes. After almost 15 years working in SAP I’m still making them. But each time I’m learning. Learning what not to do and why can be a better education than how to do something right.
It’s for this reason I thought it’d be great to pay homage to our mistakes and by doing so, reflect on those stressful moments in life, have a laugh, and think about what we learned from them in the form of a game.
Rules:
- This applies to the last 5 years of your careers/studies
- You accrue the points of each of the scenarios that you have caused or been
- You can double-count scenarios where the mistakes are worth of individual mention
- Write a comment with your score and optional to share a lesson or several
- You get to subtract from your score for each story you share. For example, if you caused a permanent loss of data and explain why then that’s 0 points
- If sharing the story, you can preface it with ‘I recall ages ago this happening to a colleague of mine blah blah blah’ instead of admitting it was you
- If you choose to share the story, it’s be great to learn your lesson and what you did to avoid it happening again
System Admins: you don’t get points for transport issues, etc if the transport owner told you the sequence, etc.
My score for past 5 years is 125. What’s yours?
My score 110 - I think 🙂 Nice job!
Fortunately, the 5-year span gets my score way down. Unfortunately, we’ve been through a major upgrade so I am sure I have several new nicknames, but nobody will tell me.
But as near as I can tell, my score is 125.
But this reminds me of a TechEd event (organized by the fabulous Marilyn Pratt ) called ‘Failfare’ – where we had a panel of people talking about the value of failing and how we have to try to embrace our failures.
I blogged about it here (https://blogs.sap.com/2013/10/29/saptechedlv-mid-event-mode/) but I can’t do the topic - or the Emcee Former Member or one of my friends who was also a speaker, Yariv Zur - I just can't do them justice.
Yariv posted a nice blog about #failfare here: https://blogs.sap.com/2013/12/26/failing-in-grandeur/
I’m sure everyone can benefit from learning how to fail, fail often, and learn.
I counted 195. Good thing there is a 5 year cutoff because otherwise I’d score close to 1000 probably. ?
In particular, I would’ve hit 65 points on sending emails from QA system to both internal users and customers. Lesson learned: make sure to delete the old emails in QA periodically. Should be some kind of a background job for this, ideally.
On a related note, once I forgot to put vanilla in the vanilla cupcakes. So there is that. 🙂
My score is 100 until I tell my stories, then it's 10.
Cause for system restore for BusinessObjects 1:
The company that I was working for periodically refreshed the User Acceptance Test environment by copying over the databases from Prod - including BOBJ's CMS database. So, the next time I logged in to UAT, it actually logged me in to Prod, but it still looked like UAT. So, I recreated what I thought was the UAT CMS database, but it ended up being the Prod CMS database. We were able to restore Prod and then put in place a policy that the CMS database in UAT would never be copied from Prod .
Cause for system restore for BusinessObjects 2:
I was in Prod to delete some reports that were no longer in use and accidentally deleted the whole folder and all of its reports instead because the interface in BI Launchpad was not clear as to what was selected. We restored to another environment and were able to move the folder and reports back and I had to write code to re-run specific instances as they were needed for research.
-Dell
I guess I’m at 55?
.
My last self-created issue was some weeks ago, when I build a little workaround in IDM for a bug in the provisioning (= writing data to the backend) to our portal.
See, you have portal roles and portal groups, who can work as permission sources for different content in the portal. We use both.
Now if you add one or several new roles to a user, IDM will always provision ALL roles to the backend, deleting every role that is already assigned to the user there. The same goes for the groups (the whole set gets pushed to the backend, not just the delta). No worries, great to keep order.
Bug:
If you assign roles and groups to a user at the same time, only the roles will be written to the backend and the groups would be set to “ok” in IDM, but not written to the portal. Or the other way around (only the groups would be updated in the portal, but not the roles). Bad for access, because something is kind of always missing.
Cue my “solution”:
So I created another step for the role assignment task, that after provisioning roles it should add a trigger group (valid for a day). This way, all the groups of that user would be pushed to the portal, too.
And then I added a step to the group assignment, that after provisioning groups it should add a trigger role (valid for a day). This way, all the roles of the user would be pushed to the portal, too.
Some might already see where this is going: If you add a role, the system will add a group. And that will add a role… Welcome to Loop Town!
Not a big issue, if it’s only one user currently under way, because it balances out, so I didn’t catch it during testing.
What I forgot was the job I created earlier this year, that adds the trigger role (for one day) to every portal user to synchronize the permissions between IDM and the portal. This job is scheduled for Saturdays.
Come Monday, I was greeted by a lot of red in my IDM log from all those nice “deadlock” errors I got. Because not only did IDM add this role to every user on Saturday and with that the trigger group too, it deleted the role a day later again from everyone, which triggered again the adding of the trigger group (for one day), which triggered the adding of the trigger role (for one day)… so… chaos, because IDM tried to remove the roles and groups after the day was up.
I cleaned up the mess and nobody else knew of it, because it had no impact to the system itself. But my head hit the desk, when I realized that I really didn’t think this through enough. ^^
.
Can’t find this on the points table, but I would call it “didn’t think a fix through” and assign myself another 25 points for this snafu.
.
I got 25 for “Deleted or changed the wrong data”. I did this during testing an IDM job that should remove a specific role assignment. Sadly it removed all roles, which in IDM also means deleting the corresponding accounts in the SAP backend.
Even sadder was the fact, that I tested this with my own identity. Took some time to manually re-create all those accounts and assign my roles again. And then the next day I did it again! -.-
After that I started using test accounts for even the smallest, easiest, “can’t go wrong” tests.
.
So after getting this off my chest, I guess I’m still at 55 points.
.
Regards,
Steffi.
You're right that doesn't quite fit. Perhaps running the incorrect script could also include 'bad script', 'incomplete' or something like that
But you are are right that 'non thinking it through' is missing. The solution with unintended consequences
I'm glad I'm not hands on with IdM... even though it's a system I really should get exposure to for provisioning.
Great Blog! My score stands at 105 good that it is for last five year else the pandora box would have been opened.
One of the biggest mistake i did was i changed an existing workflow task from Synchronous to Asynchronous and that too in same version. We tested completely everything but for got to check what impact it will have on existing one. The moment it went live you can imagine the kind of noise it would have created. That is one of the worst thing i did without actually checking the impact and all.
Thanks
Nabheet
I'm around 60 points but my memory is not so good to go more than a year.
I remember on project in Data Migration phase and Legacy System. I was responsible for uploading data and the end user for checking it, but all responsability is always on the consultant :-(. There were 3 conversion programs from the legacy to SAP and then I could upload the file to SAP with LSMW. When we were checking the documents I noticed something that didn't seem right, I didn't know what it was but it was there. It was Saturday, we were all tired so we let it go and moved on to PRD.
A week later they stop making payments until we correct all documents 🙁 , it was a hard night :'(
Another case of "Always trust your gut feeling!".
top 3 go live issues are usually data, security and interfaces
Your memory doesn't have to be great. sometimes we want to forget but sharing even one lesson is a benefit to the rest of us!
The 5 year cut-off saves me......
Nope - not telling 😉
Graham Robinson - okay, we'll all guess then 🙂 Our number might be worse!
Surely you have a good story or 10
NaN error...
Although I counter that if I have to write the new processes that that shouldn't count... (There are a hell of a lot of new cloud support processes I've caused to be implemented in last few years!
5 years is an awful large number of mistakes to count up!
If building agile do we count cosmetic "mistakes" delivered or only those fixed post implementation in prod... (First one has me in the thousands, second may reduce score down to hundreds ?)
Liking this thread ?
I find it amusing that most people are grateful for the 5 year limit
You can argue bleeding edge should get dispensation
In relation to Agile - does it count as a defect if it's not part of the definition of done?
Maybe I should have added a new rule that each incident can only count as one category. For example, if you caused a major loss of data that resulted in a root cause analysis, system outage over a day and a restore and ended up with a new procedure delivered whilst also job hunting then (1) I'm really looking forward to the backstory and (2) it should only count as 50 points
Glad you are enjoying this thread 🙂
I don't know which category it falls into, but I sometimes forget the exact release various tools were introduced - for example, I often suggest doing something via the Netweaver Administrator URL when the customer only has the Visual Administrator or vice-versa.
As an aside, do I get double points for being told to look for a new job ? (demonstrating a security flaw which lead to my employer at the time being questioned / embarrassed by the customer).
Martin E
another category missing
Maybe I should have added one of "not appreciating the politics at play" or a catch up relating to no-one likes a no-it-all who shows people up.
DING DING DING!!! Winner! Or we need to add that category!
Another one (happened 9 years back)which i remember now after reading the comments is also worth mentioning here. We have made a custom program to send payments to the bank. It was suppose to check credit/debit notes and send the final calculated payments. The moment it went live one of the vendor received 1.6 Million $ instead of 16000$? no one knew about it. It is the vendor only who approached us to cross check why this much money.
The RCA was we never considered debit/credit indicators and Function expert also tested it for one type of document
So many crazy stories
OMG..... this story is definitely not crazy but S C A R Y.....
K.Kiran.
Yes it actually was. Just to add to scariness i was at onsite for golive hypercare☹️ Imagine the beating we got from the client?
110 counting i was so frustrated 4 years ago that i started to look for a new job 🙂
I think i'm around the 70's mark.
My most embarrassing moment was when i mistake a dev and prod system and accidently send 3 pallets of pringles original to a customer. That is 3 pallets of 18 boxes with i believe 12 tubes in each. THAT'S A LOT OF PRINGLES!
I discovered my mistake fairly quicky and was able to do the correction in SAP, however i didn't think of their legacy warehouse system, so the customer was pretty baffled. I heard that story quite a lot after that.
that is awesome! Did the customer get to keep them?
Another lesson in integration and downstream systems or the remaining legacy systems that we rarely understand in depth.
WANT the Pringles now...
No the customer called the call centre. So they got them all returned. Luckily everyone just laughed at the story.
Nice idea.
My score is around 60 however I think lower as I earned points when I helped fix a problem with number ranges in PRODUCTION as a result of someone transporting number ranges from DEV through to Production. (Yes, I know a definite NO NO!).
I spent 2 days reinstating old numbers for every object in SAP! This was around 1997! 🙂
One of my most memorable was running an LSMW where I set address data for customers to the same value for all 300 records! Doh! Only did it once!
Lucky it was *only* 300 records. Would have been worse if you had a job run to send correspondence out to the incorrect addresses before you fixed it all!
There's still a few people out there who don't understand that you can't transport everything
What a fun game, mine is 85 and I'm grateful for the five year grace period.
I've made a similar mistake in the early years of my career where a conversion program updated the social security number to a different infotype, problem was after executing it everybody, around 35.000 was like the Borg since they all shared the same number 🙂
Good old lsmw. For the same previously mentioned customer, we discovered that we were double invoicing the customers. I was the only one who could put a query together in sq01 to figure out who was affected. Based on that extract a did corrections for about 3 mill aud out of an lsmw .
This was all with around 6 months experience with sap. Pretty intimidating
My score is around 40.
Raising the score will perhaps do my next go live...
The last upscore is running a generate-report for DDL-Sources and kicking out nearly everyone from the system. luckily it was just a QA and not the productive one 🙂
If you work a lot, you make many mistakes.
If you work very little, you make just a few mistakes.
Those who do not work do not make mistakes
and those who do not make mistakes … get promoted.
Now guess what happened to me since 2015
A number between 30 and 100, it did not lead to a total system outage, just components, and it even did not make it to P1 incident as it happened on a weekend. But 172000 emails and 3.4 million workitems was impressive and I blogged about: https://blogs.sap.com/2015/11/26/chronology-of-a-major-incident-and-a-sweet-revenge/
"If you work a lot, you make many mistakes.
If you work very little, you make just a few mistakes.
Those who do not work do not make mistakes
and those who do not make mistakes … get promoted."
E X A C T L Y....that is how the Corp World is 🙂
K.Kiran.
Seems,the more complex the landscape is, the more the count will be.
Nice blog and indeed good learning from the Comments as well.
10/10 to this blog and Collen.
K.Kiran.
that is quite true that a complex environment can open yourself up to a higher score
Glad you are enjoying and seeing the learnings. It was my hope the value would be in the comments.
240! I've been around long enough that the laws of probability picked my #.
If we consider just the list of “mistakes” in this post, I will score 70 points.
But there are other kinds of mistakes, that could be more on the “soft side”, like not doing a good enough negotiation (and learning with that). There are also failures that happened because they are even desired when you work with a “startup-like” mindset and seek to rapidly iterate and get better with your users or customers.
I learned that there are bad mistakes (that usually make you lose something) and good mistakes (that make you learn something new and valuable).
Cheers,
Douglas
I figure the definition of "mistake" is a personal definition but I assume there may have been a slight moment of forgetting to breathe, heart possible racing, saying oh XXXX and having to pause for a moment to figure out what you did and what you need to to do to fix it. Or you know someone calls you up and asks a question along lines of 'hey did you by any chance push a change to production last night' and it's followed by silence or 'well ummm <insert some major business impact>