DBA heroes don't always wear capes. Sometimes they just have a bunch of forgotten permissions
Just because you could, doesn't mean you should
Who, Me? We start the working week with a Who, Me? reminder that in the world of databases, there is a difference between knowing you're right and knowing your rights.
"Joff" submitted today's story, which takes us back to the mid-2000s and the divesture of his company from a US multinational. He was running a team tasked with producing interface processing systems that needed to collate data from a multitude of information sources. That data, he said, "then passed to various other internal and external systems for processing once the finance departments in multiple countries had done their bits."
Data wrangling like that can be such a joy and, sensibly, "We determined that the optimal design was a database to verify all information was correctly pre-processed before any onward transmission."
A logical decision when faced with massaging multiple data sources into something that made sense. As is so often the case, timelines were tight but Joff's team managed to get through the design, build and test phases on schedule. Doubtless with the assistance of caffeine and the odd pizza or two.
"As an ex-techie now in management," Joff added, "I could appreciate the work they did and tried to be as involved in the design as much as I could without disrupting them too much."
A rare breed in the boss stakes, for sure. Then again, this is his story after all.
Development and validation complete, the team were ready to put the database on the production servers. While they might have been DBAs on the build and test environments, the reins for production were firmly in the hands of the corporate IT department.
Still, there were weeks to go until go-live. A request to create the database, setup the backups and so on was submitted.
"And we waited ... and waited ... and chased ... and waited..."
The clocked ticked down to the 48-hour mark, and the database had still not been created. Joff and all bar one of the team were at their wit's end. All bar one? There was an individual, however, who could pull on the cape of DBA power and save the day.
"One of my team," explained Joff, "came and admitted that during a major debugging exercise some months before he had been given extra permissions."
The team member claimed (to Joff's satisfaction) that he had both not realised he still had the permissions and also that he had not abused them.
However, he could use these permissions to create the needed database, although was a bit worried about how their then corporate overlords might react.
Joff pondered. Should they keep waiting and risk delaying the go-live, or proceed? He gave it another 24 hours before hitting the go button and agreeing to take any flak that might come the team's way.
All went well. The database was created ready for go-live the following day and Joff's team got on with the required pre-population work ahead of the data floodgates being opened.
Two hours later the phone rang. The corporate DBA team were incandescent with rage. They had tried to create the database, as requested two weeks ago, and it was already there.
All permissions were to be immediately revoked. This would also slam on the brakes for the go-live.
Joff took the phone and, we like to think, held it arm's length while the ranting went on. When he was finally able to sneak in some words, he told the caller: "look we are in the middle of a major go live at the moment. I understand your concerns, but can we talk about this after the weekend when we are done with this important work..."
The anger-froth continued to spout for a few more minutes until eventually agreement was reached and Joff's team could proceed.
The inevitable follow-up did indeed come in the next week. The corporate DBA team were back ranting on the phone and told Joff that all access was to be removed from his team.
Except... after that weekend the ownership of the server had shifted. (Remember that divesture?) So.... any changes would need to be decided by both parties and Joff certainly wasn't about to relinquish any permissions.
"In fact," he told the corporate team, "from now we would be needing to pre-vet any actions that they were recommending."
"I never heard anything back on the matter."
Ever found yourself taking matters into your own hands in order to save the day? Or been on the receiving end of fury-spewing after you got found out? Share your confession with an email to Who, Me? ®