One of the worst production issues I've caused resulted in one of the best interactions I've had with a manager.
It was tax season in 2005 and I was working for the state website and was at the most dangerous time in software person's career, about 5 years experience. Confident, but still untested. Today was the day that would bite me.
I was a subject matter expert on the billing system and had worked supporting and extending it for several years. Due to regulatory requirements, we had to add some columns for PCI compliance. We had already deployed this change in other offices and hadn't had any issues, but that wasn't during tax time. Our traffic was substantially higher and, for me at the time, that was an unknown unknown.
So, without checking with anyone, I fire up my script to apply the change to the staging database. No issues. But when I run it on production, the script doesn't immediately return. I give it twenty seconds and then I hear the on duty pager going off.
Long story short, my script has caused a table lock which may have escalated to a database lock while the script is being applied. We let it complete (which it does after about 15 minutes, having been blocked from completing by open transactions IIRC).
Remember it's tax time. My little escapade has caused citizens from being able to complete their income tax for about 20 minutes.
As we were assessing the damage, I'm nearly in tears at my desk. My manager calls me into his office. I'm assuming I'm getting fired on the spot. He gives me some of the best advice I've turned over in my head for years. "You made a mistake and you have caused a huge problem, but you learned something today. From now on, I want you to stop equating your work with your self. I want you to stop using the word I when you're talking about your code. You made a mistake and it happens to all of us. It's huge problem if it happens again, but this can happen and you need to learn to let it go."
It's not like I didn't get punished. I had to watch all my co-workers work late calling everyone who'd had issues during our payment system outage.
posted 2022-09-28