Massive worldwide IT outage, hitting banks, airlines, supermarkets, broadcasters, etc. [19th July 2024]
- Thread starter cupid_stunt
- Start date
BassJunkie
Clams on the half shell, and roller skates.
Firms skip security reviews of updates about half the time
Complicated, costly, time-consuming – pick three
A report from yesterday. Crowdstrike warning about untested updates.
kalmatthew
Lurker Extrodinaire
We aren't hit very hard where I work and most disappointingly our end user devices seem to be working (I'm not in IT). Given the weather today it would of been an amazing day to turn on your PC and it just BSOD so I could sit in the garden all day with a beer!
Spymaster
Plastic Paddy
Buddy Bradley
Pantheistic solipsist
Sounds like it's primarily companies using MS Azure for their cloud solutions that are the ones affected. So if your company happens to be on Google or AWS you might be lucky.
Lord Camomile
Yipchaa!
'lucky'
Pretty sure we're on AWS, and was about to say "so presumably that's why I'm not hearing of any problems", but then I suppose as I'm WFH, if there were any problems, would I be hearing about them...?
MickiQ
In My Defence I was left Unsupervised
A lot of major corporations have outsourced their internal desktop support offshore. If the PC can't come up far enough to connect to the network there isn't a lot someone in India or the Philippines can do to help. They usually do have a few local people who can go round with a USB and the list of super secret passwords but it is going to take them ages to visit everyone.
Buddy Bradley
Pantheistic solipsist
Unless you're frontline IT Support you might not be hearing much about it anyway. Also depends a lot on your company's update policies.Pretty sure we're on AWS, and was about to say "so presumably that's why I'm not hearing of any problems", but then I suppose as I'm WFH, if there were any problems, would I be hearing about them...?
Lord Camomile
Yipchaa!
Budapest airport, earlier today 
Boris Sprinkler
Dont be scared
Oh. My. Word.Firms skip security reviews of updates about half the time
Complicated, costly, time-consuming – pick threewww.theregister.com
A report from yesterday. Crowdstrike warning about untested updates.
vinba
Member
AWS is affected.. A friend of mine has to do the following on 200 servers:'lucky'
Pretty sure we're on AWS, and was about to say "so presumably that's why I'm not hearing of any problems", but then I suppose as I'm WFH, if there were any problems, would I be hearing about them...?
Unmount the EBS volume
Spin up a new EC2 instance
Mount the EBS volume to the new instance
Use that to navigate to the crowdstrike folder and delete the broken file
Unmount the EBS volume from the resolver instance
Mount the EBS volume to the original instance
Reboot.
Ouch.
Silas Loom
Man with glasses
Budapest airport, earlier today
The sort of people who watch Sky News, obviously.
existentialist
Tired and unemotional
It's true. It's always been a problem with IT kit - the system has to get to a state where it can even communicate with the outside world, and if a screwup happens that prevents that, you're toast.
There are ways around it (machines in datacentres often have hardware to enable direct console access remotely without the need for the operating system to be running, for example). Or, just, like, you know, staging your updates in such a way that failure outcomes have been eliminated before rollout. Sometimes marketing pressure force companies to make unwise decisions - Microsoft (for one) is notorious for whacking out dubious updates that have to be quickly backed out again.
And, in the old days (bangs cane on floor), pretty much every mainframe would have had some kind of failover capacity to enable almost any eventuality short of hardware failure to be resolved quickly. Of course, there were (and are) a lot fewer examples of that kind of hardware around.
cupid_stunt
Merry fecking Christmas.
Tech fuckup.
vinba
Member
Courtesy of Crowdshite..
Karl Masks
Birds Angel Delight
What is actually happening?
I'm more confused by the explanations? Is this the work of Jeremy Corbyn?
I'm more confused by the explanations? Is this the work of Jeremy Corbyn?
Boris Sprinkler
Dont be scared
How is AWS affected?
The issue is with Crowdstrike and Windows hosts. It’s just unfortunate that your friend’s AWS hosts are windows vm’s running crowdstrike.
emanymton
A cat politely sat on the flaming gardener.
SameI keep checking for updates but my work pc keeps working.
existentialist
Tired and unemotional
[side note]
I used to manage some machines that were running in Dubai, acting as mailservers over dialup. Any time I did anything on those machines (which occasionally included an OS upgrade), I had to operate in maximum paranoia mode, because if I did anything that stopped the machine being able to get to the point of accepting dialin calls, someone was going to be flying to Dubai at very short notice, and very great expense.
I'm proud to say that it never happened, though we had a close call with some power cuts and consequent disk errors which caused a few bum-clenching moments while it (eventually) came back up.
And this was TWO machines, running some very simple software, not a cluster of network-connected boxes doing all kinds of fancy stuff.
I do wonder whether we've got to a point of increasing complexity and interdependency in our IT environment that warrants a good hard look at whether we haven't barked a long way up a rather dodgy tree.
I used to manage some machines that were running in Dubai, acting as mailservers over dialup. Any time I did anything on those machines (which occasionally included an OS upgrade), I had to operate in maximum paranoia mode, because if I did anything that stopped the machine being able to get to the point of accepting dialin calls, someone was going to be flying to Dubai at very short notice, and very great expense.
I'm proud to say that it never happened, though we had a close call with some power cuts and consequent disk errors which caused a few bum-clenching moments while it (eventually) came back up.
And this was TWO machines, running some very simple software, not a cluster of network-connected boxes doing all kinds of fancy stuff.
I do wonder whether we've got to a point of increasing complexity and interdependency in our IT environment that warrants a good hard look at whether we haven't barked a long way up a rather dodgy tree.
ska invita
back on the other side
aye, but what does "restarted and a fix applied" mean? ....and i forget who posted it but supposedly its not rebooting into safe mode on the bricked machines?
Similar threads
