Urban75 Home About Offline BrixtonBuzz Contact

Massive worldwide IT outage, hitting banks, airlines, supermarkets, broadcasters, etc. [19th July 2024]


The joke here is that this is obviously a joke tweet but being reported in all seriousness by the BBC.

At first they came for Gail's, but I do not like overpriced croissants so I kept quiet.

Then they came for Waitrose.

And that, my friend, is war. Worse than war. Like, I dunno, 2 wars or something.
 
How is AWS affected?
The issue is with Crowdstrike and Windows hosts. It’s just unfortunate that your friend’s AWS hosts are windows vm’s running crowdstrike.

Yeah should have said windows hosts running on AWS. At least not too complicated to recover and they're trying to find someone who might be able to script the entire process..
 
So far about 1% of flights worldwide. Which is in fact quite normal.

It'll go a bit above normal as the day goes on, probably. No big drama in the end though.

Less than optimal for drivers not to have signalling and platform data even by UK rail standards

IMG_9033.jpeg
 
My work colleague who's convinced that you should always use cash and have a petrol car because electronic payments and electric cars are actually a secret means of the state controlling you will be having a field day.
 
aye, but what does "restarted and a fix applied" mean? ....and i forget who posted it but supposedly its not rebooting into safe mode on the bricked machines?
Either the machine will be able to be booted, and the boot process interrupted to allow it to boot into "safe mode", which doesn't load all the drivers, and might get the machine to a state where a fix can be applied.

If that can't happen, the machine can be booted from a USB stick, so that changes can be made on the original disk to ensure that the native boot can proceed again.

In higher-security environments, neither of these is possible, and it will be necessary for someone in possession of the codes and passwords to gain privileged boot access to intervene, quite possibly in person.

Once the machine is running, fixes can either be manually applied, or the machine may be able to be prompted to apply another update which corrects the erroneous one and allows it to operate normally again.

The word "boot" comes from "bootstrapping", from the idea that a computer, when it starts up, "pulls itself up by its bootstraps", first running basic hard-coded routines that get it to be able to do things like receive keyboard input, display stuff on the screen, etc. Then another layer gets loaded which takes the machine up to a higher level of capability (network connections, peripherals, stuff like that). This security stuff is a further layer that (clearly) gets loaded during the operating system startup phase, and clearly has the potential to render the machine unusable (and unfixable) via the usual means.
 
Either the machine will be able to be booted, and the boot process interrupted to allow it to boot into "safe mode", which doesn't load all the drivers, and might get the machine to a state where a fix can be applied.

If that can't happen, the machine can be booted from a USB stick, so that changes can be made on the original disk to ensure that the native boot can proceed again.

In higher-security environments, neither of these is possible, and it will be necessary for someone in possession of the codes and passwords to gain privileged boot access to intervene, quite possibly in person.

Once the machine is running, fixes can either be manually applied, or the machine may be able to be prompted to apply another update which corrects the erroneous one and allows it to operate normally again.

The word "boot" comes from "bootstrapping", from the idea that a computer, when it starts up, "pulls itself up by its bootstraps", first running basic hard-coded routines that get it to be able to do things like receive keyboard input, display stuff on the screen, etc. Then another layer gets loaded which takes the machine up to a higher level of capability (network connections, peripherals, stuff like that). This security stuff is a further layer that (clearly) gets loaded during the operating system startup phase, and clearly has the potential to render the machine unusable (and unfixable) via the usual means.


Worth adding that Windows has required bitlocker that encrypts all hard drives on its new operating systems. Without the right key you can’t get into safe mode whereas older OS you could just boot up and launch cmd

There’s going to a lot of IT depts who realise they’ve not got copies of these keys for quite vital machines
 
aye, but what does "restarted and a fix applied" mean? ....and i forget who posted it but supposedly its not rebooting into safe mode on the bricked machines?

I think the situation is where an end user can't boot into safe mode and remove the problematic file, they need to boot off usb stick. i.e. a tempory live OS instance. Then mount the disk to navigate to and remove the troublesome file.

If full disk encryption is being used though, they're not going to be able to do that themselves and need someone to turn up with the encryption keys and do the usb boot thing.
 
Sounds like it's primarily companies using MS Azure for their cloud solutions that are the ones affected. So if your company happens to be on Google or AWS you might be lucky.

A part of my work is hosted separately from the rest of our data on our IT Co's Azure server and its been fine.

Presumably the rest of our ms365/sharepoint is also on Azure.

Someon let me know when the stock exchange is back up. I need to do some trades.
 
Might be wrong but I read it as not an Azure problem as such. But where Falcon a Cloud Strike security system is being used, it has broken Windows on that machine. Which is probably a server. So your laptop is gonna work OK but might not be able to connect to services.

/reckon
 
It would be interesting to know how many computers in the world just bricked
Roughly
Crowdstrike report they have 24,000 Corporate customers. A global network maybe as large as 150,000 hosts.
But that’s big and a lot of large style IT estate has probably changed since the uptake of Cloud computing. But let’s say each customer has 10,000 windows clients. That’s 240 million affected systems. Likely more.
 
Just spoke to my brother, his employers (USA owned tech company with websites globally) got rid of crowd strike ages ago because it was too much of a security risk, he said many UK & EU companies did so around the same time for the same reasons, but it seems not all. He said, 'typical predictable IT disaster.'
 
AWS is affected.. A friend of mine has to do the following on 200 servers:

Unmount the EBS volume
Spin up a new EC2 instance
Mount the EBS volume to the new instance
Use that to navigate to the crowdstrike folder and delete the broken file
Unmount the EBS volume from the resolver instance
Mount the EBS volume to the original instance
Reboot.

Ouch.
My heart goes out to your friend, that sort of job would be straightforward to script but he won't have the time to do that. There will be some PHB jumping up and down shouting "now now Fix it NOW!"
And of course when it's over he will get told "Well it's a one-off it won't happen again so no need to worry"
 
Back
Top Bottom