Waking at 1am with the sudden realisation that you've messed up at work....

It's never as bad as it seems.

As an aside doesn't anyone review your work / cast a second pair of eyes over it? Seems like a poor process if not

You've spotted it now so can be corrected before delivery and implementation. Not great news but better than delivering as complete and it failing.

Ah damn. Sorry to hear that. Presumably the first bloke didn't spot it either so not really your fault. Try to announce it in a way that makes you look clever for identifying the problem maybe. Would have been worse to discover during final testing.

Honesty, humility and humbleness (and other words with "h") will probably save the day.

It's awkward, fess up, sort it and it'll be different next time around. We've probably all been there at sometime but its not a good feeling to have....

What is done is done, take a deep breath and m<span style="font-size: 0.8rem;">ove on to the next stage. </span>

Hope you will be fine and have the discussion with your boss asap, don't let it fester.

Then go for a bike ride.

Everybody makes mistakes. Its what you do once the mistake is found that counts.

Your work world is miles from mine but what i would be looking to do is go to the boss with something like " discovered a balls up. The solution seems to be A ; B and C. Timescale for the fix should be X ; how do you want me to proceed"

Back in about 1995, a Mobile Phone Co had launched its first prepay offer. Top-ups were done via the Post Office, and there was a promotion where each top-up trigged a free additional one.

A file came in every day, and there was a semi-manual process to go through this file and add the free ones.

Christmas Eve, someone realised that there wouldn't be any staff to process the top-up file over the whole Christmas and New Year period. A mate and I got the job of quickly scripting something to make the process fully automated. Essentially we wrote some code that said "if you see a top-up, add a free one". We were the last people out of the office Christmas Eve but got the job done.

Christmas morning I realised that our code didn't look at the top-up type, so a free top-up would be seen the next day and another free one applied. And the next day, and the next day...

Called out the on-call Christmas cover, talked them through the changes needed to the script over the phone (no laptops or remote access apart from the one on-call person).

Saved the company several million, or a PR disaster.

I may tell you about the time some of my code stopped all new mobile phone connections to Mobile Phone Co once you got to December.

Yes I sometimes wake with a start panicking about something I've done wrong at work. I've been retired 4 years!

Presumably the first bloke didn’t spot it either so not really your fault. Try to announce it in a way that makes you look clever for identifying the problem maybe.

This.

It is ALWAYS better to own up to the issue immediately it is spotted then delay or even try and hide it. The customer will ultimately be glad of the honesty rather than you winging it and hoping the issue never appears. The sooner people are aware of the issue the sooner it can be worked upon and fixed, regardless of the size of the problem. How many big projects have stalled or failed due to people hiding issues? Answer: thousands of them.

Based on a past career where there was some potential for it, my bar of CU is stuck at ‘no one died’. Anything else can be overcome.

But, yes, ruminating revelations of ‘nooooooo’ at awkward hours are a pain.

It’s why we put quality assurance processes in place?
And good design, review, and planning?
And we speak up when speed or cost may compromise quality beyond an acceptable limit?

**** ups happen

It's how you own them that's important

I mobilised a project at risk without all the terms having been agreed (Government contract) and once the legals has been reviewed, we’d never have done the work but had to proceed as we’d made employment offers (reputational risk was huge) eventually absorbing the millions it cost us. My director went off sick so I felt like I had to carry the can - it was rubbish.

Speak to your boss first and give him some possible solutions rather than letting the customer know - let those on the bigger pay grades deal with it.

Like others have said, be honest and try to fix it asap. That’s all you can be expected to do. Easy for me to say but try not to let it ruin your health.

Personally, I don’t judge companies by their mistakes, but how they deal with them.

Reiterate the above really, mistakes happen. It is not that you have made a mistake the most important thing is how you handle and deal with it. In my experience the fall out from a mistake has never been as bad as what I think it's going to be..... and the longer I have to dwell on it the bigger and more catastrophic the mistake appears

Usually when I wake up in the night worrying about something, turns out next day that I'd been worrying about nothing.

Not a big help, I know 🙁

Two spring to mind for me.

One was a rounding error on an order forecasting spreadsheet, if we bought 1000 tonnes of a material we received a rebate on every kg purchased. 1 million kg x €0.2 = €200K  My rounding meant we were literally about 3kg short. Supplier stuck to their contract and refused the rebate. My boss was understandably pissed off but basically said as long as it's not deliberate, we all make mistakes.

Second was an allocation decision I had to make which shorted a major toothpaste supplier of a key ingredient for a couple of weeks. Late Friday I got their purchasing director on the phone yelling and threatening all sorts, I had a nightmare weekend waiting to tell my boss on Monday. He was brilliant, said no-one talks to his staff like that and then called the PD there and then. Basically said if there were a few tubes fewer in Total on the shelves briefly (most likely in warehouses really) then he was sorry but just suck it up, no-one is going to die as a result.

It helps his previous job included making fuel allocations to the USAF in the first gulf war, where decisions were between a jet for fire support for troops in ambush vs a duster to casevac someone else somewhere, literally life and death, so against that he wasn't inclined to give many ****s about toothpaste or a few hundred thou in a $5Bn company.

when i got to the stage of waking up at night because i had done something wrong at work and because i did not want to go into work, i retired.
less money more time is a good ethos i think.

All I can say is your employer should be glad you have a conscience

As in understand it error free work is staggeringly expensive

I've woken up with the dreaded "I've screwed up" thoughts a fair few times over the years, my experience is that things are never as bad as they seem when your mind is running riot at 3am, in my case my mind starts going around in circles with a corresponding increase in stress reaction.
I reckon that going for a walk/ ride early on helps to calm and rationalise things and then go discuss some possible with your manager/ colleagues: it might be that they pick up on a detail you have missed.
Good luck with it

Generally speaking it is how you deal with issues that marks out a top performer. Issues happen. Those who can effectively manage issues are usually those who are marked out for higher things. It’s especially hard when it’s your own issue but that will usually be recognised by those above you so I’d suggest take the approach of “owning” the issue, go in on Monday with a clear explanation of what’s happened and what the options are to move on. Don’t email people about it over the weekend as (a) that gives you time to come up with a slick plan for Monday and (b) you want to be seen to be calm, collected, open and honest.

Don’t focus on the “why” it happened as that can come across as making excuses - just be factual.

You could easily come out of this better than you went in.

For a minute I was wondering if Kwasi Kwarteng was on STW 😉

The strongest relationships come when there have been headwinds. Don’t sweat it, do the right thing and it’ll be fine.

First. Take a breath and relax.

For a minute I was wondering if Kwasi Kwarteng was on STW 😉

No, no. The OP is asking how to acknowledge the mistake instead of walking into the office on Monday and going "we're challenging the orthodoxy, please don't tell me the numbers".

Two weeks into my first proper IT job I thoroughly screwed up. Got a “hope you’re enjoying your new job” email from a mate. Unfortunately it was riddled with a malevolent payload which dispatched itself to everyone in our global address list (university Admin department). This then went onward to everyone in the entire university. I went and saw my boss immediately. There was no fuss, no drama we just worked out what had to be done and did it. Turns out we had no mail plug in on the workstation AV solution, no AV solution on exchange and none of the fundamentals closed down like, you know: backups that worked, open relay shut off. It was a disaster waiting to happen, I just triggered it. Spent 2 solid days fixing things. All was good in the end. Tonight me a fundamental lesson: so long as everyone knows the situation it can be dealt with. There’s always a solution.

I’ve often done the during sleep processing thing. This week I woke up having worked out that a script I wrote would permanently disable users who went on paid leave. Doubt I would have found that out until go live given the customer’s testing regime.

No need to lose sleep here. You inherited a project with a flaw, and have spotted the flaw. Could have been spotted earlier, couid have been spotted later. As everyone else said, mistakes happen , how you respond is what matters. If it's going to help you mentally ,give your self an hour, spend that time to describe the problem, come up with a solution, tell your boss and park it till Monday. If your boss is pissed off, it should be at his project QC process.

when i got to the stage of waking up at night because i had done something wrong at work and because i did not want to go into work, i retired.
less money more time is a good ethos i think.

This - The stress was just too much.

And for the OP - Whatgoesup has the best plan - take some time, think through how to present the issue, what the solutions might be (including the difficult options such as cancelling the whole project), pros and cons, and present these to your boss on Monday. Might ant to make sure he/she is available Monday if that is possible though

**** ups happen

It’s how you own them that’s important

Absolutely this. Show me someone who says never had that horrible moment of realisation and I’ll show you a liar

We’ve all done it

Nobody died though, did they?

Mine was not updating the amended indesign files to the server at the end of the day. The next morning someone assumed they were right (which they should have been) and an unamended massive publication went to a big print run.

I was out of the office the next day and had that cold sweat ‘oh shit!’ moment when I realised

The lot then had to be pulped and reprinted at great expense and delivered late to an unhappy client

I fessed up straight away, said it was 100% my fault and took it on the chin. I was absolutely furious with myself for being so unprofessional

At the end of the day my boss came over put his arm around me and said ‘come on… I’m taking you for a pint’. He sat me down and gave me the ‘we’ve all been there…’ speech and I got some kudos for holding my hands up and then sorting it out

That was 20-odd years ago and I’ve never done anything like that again. Everything since is belt and braces!

The fact you’re waking up and you’re this concerned shows that you care. That won’t go I noticed if you take it on the chin, sort it out and make sure it never happens again

This stuff is complicated, mistakes will happen. My screwups have occasionally ended up on reddit, though so far, thankfully, nothing I've done has been bad enough to end up on STW.

Anyway, there's probably a python module to solve your problem.

Anyone that hasn't is either a liar or oblivious to their own mistakes.

Better you find and plan to make amends than someone else to find them and you get blind sided.

Based on a past career where there was some potential for it, my bar of CU is stuck at ‘no one died’. Anything else can be overcome

This.

The feeling as a suppression system discharges accidentally or a server room goes silent is a something else.
Especially compared to the fear followed by relief of opening an email about a fire on a ferry which was extinguished by a system I was the last person to touch.

I was Senior IT Manager/Head of/Director for many years. If one of my team made a cock up I just appreciated them speaking to me, accepting responsibility and to come armed with a resolution/mitigation.
Work on that then call him/her first thing Monday to tell them what the impact is and what you/them/team are going to do about it.
It may be just me but I would probably call over the weekend, it will ruin theirs but that goes with the territory and shows that you are committed by worrying about it yourself, that will help a little. Also look to introduce processes that mitigate against things like this in future (although dont necessarily mention that when you speak to them!!)

Call your boss up now, tell him you were working through the night looking everything over because something didnt sit right in what you'd been sent.

You've now found a big snag.

Don't tell him you messed up and woke up stressing. Make out like you are an amazing employee helping the company.

Reframing it will make a lot of difference.

No need to lose sleep here. You inherited a project with a flaw, and have spotted the flaw.

This.

Also something is wrong with how the project is structured if it is relying on individuals to spot things such as this. Too much resting on one person's shoulders. Needs to have multiple eyes on it to ensure problem are caught early, and to have some shared responsibility in situations such as this.

Personally I'd leave it unyil Monday.

What are you expecting will happen over the weekend that will be of any benefit?

I've had some brutally bad times at work, things where you just can't imagine how you can dig yourself out of the hole you're in. What you need to do is remind yourself that the sun will still come up tomorrow and the issue will get resolved one way or another. Constantly worrying about it will make it worse, so just go for a bike ride, drink a couple of beers or whatever, watch a movie, just do normal things when you're not working, don't let the worry take over your life.

What are you expecting will happen over the weekend that will be of any benefit?

Depends, presumably the OP will be looking at options regardless so their boss may as well not be blindsided and have the time set aside to deal with it.

@codybrennan this is a failure but its not yours. As pointed out the QC/QA seems to be severely lacking if its not been peer checked/independently verified up to this stage. You've caught it before it was deployed, think yourself lucky as I've seen an entire water tank for a safety critical function built before a designer realised it was actually above ground and needed a whole load of extra bracing that didn't exist. Thankfully, before it was filled.

Cody, I have no advice to add but I do sympathise and hope it works out OK for you.

Oh, and my ulcerative colitis has now suddenly flared up overnight for the first time in a decade as (probably) a result of the sudden stress.

You need to reassess your priorities. As above; if no human is negatively impacted by your 'mistake', if the ultimate outcome is only that it might affect someone else's profit margins, then it's really not something to lose sleep over and make yourself unhealthy. Perhaps it's time you walked away and looked at alternative work. Always put yourself first, in any job you're doing.

Had the wake up in the middle of the night going “that thing - if x, then y. **** how did I miss that??” many times to various degrees of seriousness.

The difficult bit is remembering about it the next morning!

There’s also it’s more friendly cousin - waking up going “oh that’s how you solve that problem!” (Also happens quite a bit in the bath, for me)

Waking up in the bath sounds terrifying 😉

I set back a €2m project (and by extension a much larger €8bn project) by almost two months by using a newer, unvalidated version of a piece of software before I'd had chance to test it's effectiveness. I was rushed, happened to use it by mistake and what I iterated in that version made the previous version incompatible, so I continued to use it. MASSIVE mistake. It was all kinds of wrong and the results it gave weren't outside the realm of possibility, so I didn't consciously notice (If I'm honest with myself, I knew something was off, but didn't follow that sense of wrongness to conclusion) until it all came tumbling down in the days before a major aircraft design review. I had to own it in front of a room of peers and superiors, apologise and promise to correct it with all possible speed and diligence. I've never felt so small and stupid in my life, but they were all remarkably accommodating. I had it fixed in 6 weeks, but that was a LONG six weeks and I still kick myself for my stupidity.

Everybody makes mistakes. Its what you do once the mistake is found that counts.

^^^ This. You know the problem now and has taken the responsibility to remedy it, that's a good start albeit at the last minute but you will solve it which is commendable.

I had a project that I spent several years (nearly a decade) working on (career dependent) but knew there was some issues but could not pinpoint the exact location of the problem. Handed in the project only to be spotted by the "clients" resulted in my ruined career. Practically destroyed everything I was supposed to be (career wise and life - lost everything) and ended up as zero hour worker. Spent another 20 years slowly picking up pieces. Now I am just at the start again. Can't turn back the clock but just to move forward.

+1 for the reframing.
...I've found an issue, we can't ship with it
...I think I know how to fix it, I will need x (time, resources)
...I expect it will lead to a delay of x

The culture of the company is also important here, how are staff treated when raising issues

The culture of the company is also important here, how are staff treated when raising issues

I’d also say that there’s only a certain amount of responsibility that a single person should own and if the company is that reliant on that person then they are actually not helping that person or being much good as a company,good companies make sure that processes and reviews are followed to capture oopsies early.

I’ve seen the odd manager go loopy(actually sectioned) in a bad company and off sick in others never to return quite the same.

Or maybe I’m the worst ever employee to manage 🙂

The cracker I owned up too was getting a million squid stuck in our system,was an automatic authorisation of life assurance payments to customers and I’d not set a flag correctly and the money was in no man’s land which was better than the bloke who debited the wrong payments from customer accounts that was a cracker but he survived another day although It took months to sort out.

One time (at band camp) I also a inadvertently triggered a purchase of 2O grands worth of shares but we made a profit on that so no pain.

OP,hope you are feeling better than you were at 1am. Some good replies in this thread,we are all human.👍

**** ups happen

It’s how you own them that’s important

^^ that needs to be on a T-shirt. 😀

Thanks all 🙂

Pretty low key compared to some people but the sinking feeling when I realised I was reformatting the hard drive containing loads of client tv shows instead of the internal hard drive containing something I didn’t need was something else. Instantly put my hand up though, and spent the next little while re-capturing a bunch of US reality shows from tape.

Lesson learned there, if reformatting a hard drive unplug all the other ones!

It wasn’t your design or quote, but you *did* find the problem in it. @wbo called it right.

That feeling when you wake up at 3am under a bush in someones garden in San Diego while on a work trip after getting roofied in a bar!

That worst feeling when you get back to hotel (thank you uber) and your boss starts messaging at 8am needing to desperately talk to you.

Then feeling of relief when you talk to the boss and find out your colleague got incredibly drunk, started a fight then spewed on the CFO of the company. Your slightly dishevelled and confused state forgotten about by all!

http://imgur.com/gallery/Bzy6fgN

@codybrennan have a read of this:

https://en.m.wikipedia.org/wiki/Just_culture

Anyone who works in aviation will immediately recognise it and why I am sharing.

Everyone makes mistakes - this will never, ever change, its human nature and is actually what makes us 'better' than AI or robots or software.

What we need to do, is learn from then, and we do that by being honest, opening up, and fixing the issue, and any decent human being will also understand that.

Oddly enough I'm on the other end as I'm the customer and one of our IT support suppliers has changed and configured a connection incorrectly. Not my job as we outsource but I found the issue. This affects all our reporting.
What is making mad is they are not telling me how it happened and being evasive. I just want it fixed but now I'm worried about their integrity, and may ditch them, not for the mistake but for not being honest about it.

Causally hold a claw hammer when informing the boss. It will soften his reply on the subject 😉

I once made a mistake that almost delayed the production start of a program by 12 weeks, a test wasn't completed correctly and had to be repeated for certification. (The vehicle weight changed late in the program and i missed it, in my defense there had been a dozen weight changes in the preceding 6 months).

I admitted the mistake and managed the whole thing, pulled it back to about 6 weeks in the end. Cost about 20 million quid in lost production and a few percent off the stock market value of the company. Ended up with a promotion.

On the other hand a colleague missed something major, tried to cover it up, failed miserably and caused a fairly major issue. Had to examine and rework about 6000 cars. About 1/2 had to be scrapped. (None had actually been delivered at that point).

He's no longer able to work in the industry as he's been blacklisted.

One time when I worked for a bank, we lost a load of money. Not the usual way a bank loses money, ie fraud or betting on the wrong imaginary dog. No, this was a bag of physical cash, there was no way we should have lost it. It basically looked like a member of staff had nicked it, and it all was pretty fraught and depressing.

Til my colleague Helen was visited in a dream by her long-dead father, who said basically, "Lo, Helen, I have returned from heaven with this important message for you- you accidentally gave that bag of cash to a particular customer, in place of a different bag of cash". Next day, she phones that guy up and he checks and yup, he had a bag of about five grand which should have been a bag of £500 in pound coins.

I nominated helen's dead dad for employee of the month but apparently that was not ok

You'll be respected for explaining the situation to your boss, giving you both the opportunity to work through it, figure out how to fix... We've all been there in our working life, monkey mind waking us up in the early hours to tell us about the error in calculations, why expensive bits of kit don't fit into a process operating in 2022 because the drawings haven't been as-builted since 1990..... Fortunately, the managers I've worked for have been suitably stressed by errors being identified, but equally appreciative that there's the chance to resolve....that's what the senior managers are paid for! I once worked on a project a few years after someone had been killed - the court case was in its final stages at that time, the ongoing stress for the project manager during that period and trauma the team suffered was horrendous - always think of the lads family, that he never got home from work that day. Not a great comparison, appreciate that, but when I do make a mistake at work, I balance it by saying everyone, including myself, got home safe to see their family at the end of the working day. Go easy on yourself, you inherited a mistake a spotted it. Good on you!

If it's any help OP i woke up in the night once in a panic just like you...

I'd left a tray of (mildly) radioactive biological samples with a research value of about £150k on the counter of a bank on campus when I was a PhD student. Luckily they were clearly labeled as such.

I did that on the morning of September the 12th 2001.

Neither the police or the bank found it very funny, although the police did kindly return them.

I occasionally still wake up thinking about it but look back at the responsities that were given to a clueless 21 year old as a good lesson for what happens when an organisation wings it.

As per many others I have buggered up from time to time costing companies a reasonable amount of cash and I have also spent time tidying up after others buggering up. In all the latter cases I have been asked to provide review and feedback about what should be done in response. Only time I suggested firing someone was a good idea was due to them trying to cover it up which turned it from a few hours to fix to taking about 5 days including the weekend. Aside from that its generally we need to fix processes and get proper cross reviews in place.
From what you have wrote my immediate response would be its a failing in management not to have kept a few hours a week/month (depending on complexity/length of project) from the promoted expert to review your work whilst you settled into the role.

Any good manager takes Watson Snr's advice.
“Recently, I was asked if I was going to fire an employee who made a mistake that cost the company $600,000. No, I replied, I just spent $600,000 training him. Why would I want somebody to hire his experience?”

Things to remember:
1. Everyone who is a builder, who does something, as opposed to sitting around watching others, makes mistakes. Everyone.
2. At the end of a project when it's bedded in and running, no-one remembers the mistakes, they just think about the value it's giving, not any bumps in the road to get there.

Don't beat yourself up mate! And don't think about it anymore until Monday.
You are exactly the kind of guy that employers love - you will own this and fix it :)}
Have a great Sunday!

I still wake up with a start/feeling of dread thinking I haven't done my homework. I left school 31 years ago 😂

That project I inherited? I've checked it and found this problem. Don't worry though, I've thought of these three possible fixes, these are the costs and benefits of each option, which should we do?
.
That kind of thing tends to be fairly well received🤷‍♂️

One time when I worked for a bank, we lost a load of money. Not the usual way a bank loses money, ie fraud or betting on the wrong imaginary dog. No, this was a bag of physical cash, there was no way we should have lost it. It basically looked like a member of staff had nicked it, and it all was pretty fraught and depressing.

Til my colleague Helen was visited in a dream by her long-dead father, who said basically, “Lo, Helen, I have returned from heaven with this important message for you- you accidentally gave that bag of cash to a particular customer, in place of a different bag of cash”. Next day, she phones that guy up and he checks and yup, he had a bag of about five grand which should have been a bag of £500 in pound coins.

I nominated helen’s dead dad for employee of the month but apparently that was not ok

This is a great story!

Try waking up and wondering if you put the correct solvent in the specimen processing machine and the 150 patients who the specimens belong to might have to be called back and re-biopsied 😳🤦

I love you guys/gals. So many points above that made me chuckle/wince/nod along to sagely. In terms of the impact of this thread- I slept well and deeply last night.

x

Been there before. As long as you own it and fix it, itll be forgotten about within a month at most even if it's really bad.

My ones often aren't revealed to me until I see them going wrong on TV to a global audience, never much fun when that happens. Layers and layers of checks but there's always a chance something will slip all the way through. Fessing up with a clear "this happened, this is why it happened, this is what we're doing to fix it and this is how we'll ensure it doesn't happen again" is the best I can do. I've still had clients who responded to my first proactive mention of it by CCing my boss in an arsey complaint back but I just think less of them for it and move on now. And hold a grudge forever, of course.

How did the morning meeting go Cody?

Yeah man update needed

Waking up in the bath sounds terrifying

It's ok, he was with his friendly cousin...

Big national bank. Got mortgage rate wrong on an ad. ££.

Oh and not my story to tell, but I know someone who loaded a cash machine wrong, to the benefit of the public of Bristol.

As an aside doesn’t anyone review your work / cast a second pair of eyes over it? Seems like a poor process if not

This x100. Any high level designs for IT projects we do are reviewed by at least two other people (senior architects) and often sections are reviewed by technical architects for each technology involved (e.g. a network architect and a DBA). Before costs are provided to the client they go through a whole other review process. It's mental that one person can submit a design with costs to a client without peer reviews for what sounds like a fairly big project!

I once worked for a company that provided software that made satellite television work, in this case for a popular provider named after the stuff that is above us, let's call them "Cloud" or "Air" or something like that for now.

Anyway. I worked on hard disk backup code in this software. At one point, I very nearly delivered an update to the entire customer base that would have caused the hard disk emergency recovery process to be triggered (whether it was needed or not).

Emergency recovery would (give or take) reformat the disk in the customer's unit.

"Oops."

I set back a €2m project (and by extension a much larger €8bn project) by almost two months

Sounds like it was still quicker than most industry projects of that size.

Update: all is well. I followed the advice given, saw the boss first. He couldn't have been any more helpful, but was keen to point out that I'd overstepped the role I have. He was right. He was keen to get the customer the solution, as was I, but pointed out that others had been failing to pick things up properly. We followed up in an email a bit later.

By the time I'd sent this, he'd managed to get a lot of assistance involved, which was great. And although I'll still be advising, I've to step back from the coalface on this one and let others own their issues.

I couldn't be more grateful for everyone's advice. I've known for a while that I'm not great at pushing back where others are slacking off, and I need to work on that I think- for the good of physical and mental health, if nothing else.

Best regards to you all!

Result, good response from the boss.