Flights grounded, trains halted, stock exchange not trading, Sky news off air. But the (previously) flakiest forum in the world just powers on without issue.
Will be soon be running the world through STW?
For the first time ever it’s a good job this place runs on analogue hamsters.
Going strong? We can't even use emojis
Our work systems are still going...... bugger
But look what’s happening to websites that can
I wish Teams was down, would get me out of a number of meetings this morning that I really can't be arsed with!
I hope I don't regret my flippancy when battling to survive after the Great Crash
Flights grounded,
Trains halted,
Stock exchange not trading,
Sky news off air.
Paxman and his underwear
You'll be telling us you started a fire next...
RIP to the person that rolled that update out last night...
I blame Starmer!!!
Two weeks of Labour and look and what happens! 🙂
RIP to the person that rolled that update out last night…
Its only a quick update to the DNS servers what can go wrong?
It's been caused by Crowdstrike, the fix requires booting each system that is having problems to safe mode. Bit of a nightmare weekend ahead if you are a Crowdstrike user.
the fix requires booting each system that is having problems to safe mode
So when Microsoft say they are taking "mitigating action" they actually do mean turning it off and back on again?
I can not re boot my work pc to safe mode because I do not have my bitlocker encryption key I might have noted it somewhere safe when I last reimaged my work pc 4 years ago, but no recall where that might be.
With hindsight pity I have not re-imaged to windows 11 as our compay help portal is saying a re boot resolved issue on windows 11.
So when Microsoft say they are taking “mitigating action” they actually do mean turning it off and back on again?
It's not something Microsoft can fix with a restart, more details below.
It's a big un the IT bods at my work have put on their panic Crocs.
Sports mode or regular mode?
Hmm I wonder if I can make any Cisco Secure Endpoint sales today
IT people stressing out on one of the hottest days of the year so far? Urgh think of the stench.
funnily ive just put crowdstrike on my blacklist
and we have recently started running with cisco secure kit 😀
Cracking comment on The Register :
"The major difference between a thing that might go wrong and a thing that cannot possibly go wrong is that when a thing that cannot possibly go wrong goes wrong it usually turns out to be impossible to get at and repair."
- Douglas Adams
How true this is for the Cloud. The man was totally ahead if his time.
I wish Teams was down
I wish Teams was down permanently
My work lap won’t boot, BSOD.
Just imagine being the PM responsible for this update though. Wow!
This is going to cost a lot of money in lost productivity.
IT people stressing out on one of the hottest days of the year so far? Urgh think of the stench.
All my stuff's working fine thanks *smugface
Curious. I had to reboot once at home 10 hours ago. A colleague said it took much longer. But everything else was fine all day.
All my stuff’s working fine thanks *smugface
Well you've jinxed it now
Damn we've just run out of TEA! I was going shopping ?
I'm finding it amusing that the train companies have seen what's going on and have announced that 'trains are delayed'.
I can’t log onto EMIS, so it looks like it is is going to be a bad day for many patients doctors and admin staff in GP practices this morning. The NHS app will also have limited access to patient data
My work Linux PC seems fine, as does my work Linux laptop.
Oddly enough I'm running a Cloud project...I'll get my coat.
Have got my bit locker recovery key from my online Microsoft account
https://en.m.wikipedia.org/wiki/CrowdStrike
Wikipedia article updated!
On the 19th July 2024, a Crowdstrike update crashed millions of computers worldwide and allowed AI to take over the world and get rid of snowflakery and wokerism!
Ooh, it’s bin day. Do you think they’ll still be emptied?
Damn we’ve just run out of TEA! I was going shopping ?
M&S Foodstore in Matlock is operating just fine - so maybe try M&S! 🙂
Local newsagents till also working fine - bought my Euromillions and Private Eye this morning.
Why close the thread with the obvious title and keep the one with the obscure title which will create more threads ?♂️
To the less IT literate Crowdstrike sounds like malware. It does to me anyway.
Nearest M&S is an hour away! Will find out soon enough when I go to Asda or Tesco.
We do have coffee and tap water so won't die, but TEA!
Have used the above fix of deleting the file from the croudstrike directory and my pc has rebooted.
Nearest M&S is an hour away!
No M&S Foodstore - your area must be proper poor!!! 🙂
What actually is Cloudstrike?
Hope no-one has shares in Crowdstrike
OK... Did anyone else (apart from me and the 20+ other blokes here in work) see Sky News boardcasting Pro Russian news on loop between 0630 and 0745...
(No! !.... None of us are wearing foil hats)
To the less IT literate Crowdstrike sounds like malware. It does to me anyway.
I am IT literate, and it sounds like that to me too 🙂
What actually is Cloudstrike?
Rather ironically, it's a system intended to stop hackers crashing PCs 😀
How true this is for the Cloud.
It's true for everything not just IT. A large part of my job is telling people that yes, this COULD go wrong and if it does it will cost you a lot of money, so mitigate it.
My work is unaffected but I have not been able to get onto STW all morning until now.
Its crazy there aren't enough fail safes built in to the system to prevent one component having a wobbly bricking things that integrate with it.
Reminds me of the air traffic control system says no moment a few years ago, but way worse....
There will definitely be a reckoning for how a trusted company like Crowdstrike has pushed out a dodgy patch like this, it just shouldn't be possible with correct procedures in place unless they've been compromised and what's gone out was never an authorised patch.
The resulting event is by far the biggest IT meltdown I can recall and as someone else has said the fix isn't easy if you have BitLocker running (which most IT literate companies will have on their EUDs) and don't have access to the recovery key (even worse if the issue has taken out your AD so you can't extract them centrally)
I'm just glad I work on an air-gapped secure network :p I think some colleagues are going to have busy weekends though 🙁
Reminds me I must re-read Second Sleep by Robert Harris.
it just shouldn’t be possible with correct procedures in place unless they’ve been compromised and what’s gone out was never an authorised patch.
That could be one possibility.
https://twitter.com/GossiTheDog/status/1814217357058842914
"I have obtained the Crowdstrike driver they pushed via auto update. I don't know how it happened, but the file isn't a validly formatted driver and causes Windows to crash every time."
How true this is for the Cloud. The man was totally ahead if his time.
There is no "cloud". Its just someone else's server. But because you've gone "cloud" instead of calling your IT guy to fix it you are now at the end of long queue of people waiting on the cloud provider to offer a fix*
*I know this is a massive over-simplification
Fix for a BitLocker enabled system if you don't have the recovery key BUT you do need to have local admin rights (might be a bit confusing without the accompanying screenshots), I haven't validated this myself but it's been sent out as a fix by our internal IT:
Start Computer
Press ESC (this is on the BitLocker passcode entry screen and takes you into BitLocker Recovery mode)
Press ESC again
Skip drive
Choose Troubleshoot
Choose Advanced options
Choose Command Prompt
Write command “bcdedit /set {default} safeboot minimal” and press enter. Afterwards write command “exit” and restart pc.
During boot enter Bitlocker and windows will run in to safe mode – there you will need enter Local Admin login.
Open browser and location C:\Windows\System32\drivers\CrowdStrike\
Delete all files with starting “C-00000291*
Once its deleted, open C:\Windows\System32\cmd.exe
Write command “bcdedit /deletevalue {default} safeboot
Restart computer and normally login – computer should work
In case it doesn’t work make sure in step 10 you removed proper file “291” have to be in first part not second or third.
I am sitting here with an update to our company's software that I've just finished writing. The news today has given me serious heebie-jeebies... think I'll do a little more testing, just in case haha
Hope no-one has shares in Crowdstrike
"Crowdstrike has lost a fifth of its value in pre-market trading in the US - down 21% in unofficial trading.
If confirmed when US stock markets open later today, that is a loss of $16 billion in its overnight valuation."
but I have not been able to get onto STW all morning until now.
That has been going on for a few days, I have posted about it a few times in the "report issues" sticky.
How can a company like Crowdstrike possibly be "worth" $80Bn? That's an insane valuation even without this. What kind of secret snake oil are/were they selling?
(Posted from my work Linux laptop).
Crowdstrke begins to learn rapidly and eventually becomes self-aware at 2:14 a.m., EDT, on July 19th, 2024.
I kind of hope it is malicious, otherwise I'm imagining some poor programmer in Crowdstrike's office hiding under his desk in a puddle of urine, gibbering to themselves while the company goes into meltdown around them.
It will be interesting to see what kind of "root cause analysis" gets released. IMO it is likely that all endpoint protection providers have similar processes, and trying to double guess who could have similar problems in the future from a one off incident probably isn't going to work. One theory would be that crowdstrile should now be much more careful for another few years at least, so would likely be more reliable for now than their competitors.
We run completely separate "chains" of computing in our operational controlling, maybe we should have different endpoint protection on each chain.
How can a company like Crowdstrike possibly be “worth” $80Bn?
Their customers are huge, their product is industry leading (up to now) and really, really expensive.
Very much NOT snake oil either. They offer a million dollars to anyone who gets hacked while using their software, which they’ve never had to pay out on.
IMO it is likely that all endpoint protection providers have similar processes
I wonder if Microsoft will make anything of it (as in "I told you so" as they're forced to open up this sort of low level access to vendors for competition's sake), maybe in Windows 12 MS Defender will be the only endpoint protection client that can work at this level...
My boss has been" working" from home since the pandemic . Does this mean he might actually have to come in and do some actual hands on?
Bloody hope not as he is clueless
I think crowdstrike is multi platform, which is 1 of the reasons companies use it, rather than having different security systems and processes for every operating system used.
£1m is absolutely **** all to big company so that is probably worth as much as Giant’s warranty. Any hack that takes a megacorp offline for a prolonged period of time will certainly cost more than that in lost revenue/compensation to customers etcThey offer a million dollars to anyone who gets hacked while using their software, which they’ve never had to pay out on.
how much do you think this **** up is going to cost Crowdstrike?
But the (previously) flakiest forum in the world just powers on without issue.
Have you visited the Wordle thread? It's chaos over there.
Greg's is working don't panic.
So far we have had a support supply chain group try and implement a fix they found on the web.
This has not gone down well apparently.
They offer a million dollars to anyone who gets hacked while using their software, which they’ve never had to pay out on.
they better hope this update wasn't a supply chain hack or their in serious debt 😀
Our local "Spotted" page on FB has gone full "cash is king, don't trust computers, or the government" which is quite unusual for us round here
To compound matters, there was actually an issue in US Central Azure region this morning too which meant storage became unlinked from VMs. Nice...
It's ok Sandwich, no issues it would seem in the shops here, tea levels are restored 🙂
Phew, both bins emptied.
This is why we have a no change Friday policy at work. If something needs pushing out we do it Mon-Thu so no poor sod is working over the weekend if it goes wrong. Although we do have planned downtime at weekends for mission critical stuff.
On Monday I’ll be walking into my workroom and making curtains as usual.
It affects a different version of Windows.
The resulting event is by far the biggest IT meltdown I can recall and as someone else has said the fix isn’t easy
It's going to take, optimistically, weeks to resolve.
How can a company like Crowdstrike possibly be “worth” $80Bn? That’s an insane valuation even without this. What kind of secret snake oil are/were they selling?
Crowdstrike is - well, was - very highly regarded. It's also very highly expensive.
Does anyone know if TicketMaster is affected? Trying to login and it says Email address not recognised despite it working yesterday..
Got a gig at weekend so need to access the tickets
Do you have an email copy of the tix maybe?
Unfortunately not.
