Massive worldwide IT outage, hitting banks, airlines, supermarkets, broadcasters, etc. [19th July 2024]

[moochedit]

I don't know about kenya but i can confirm sainsburys in bedworth's card machines are working fine so panic over * eats sandwich *

 

[Artaxerxes]

Allianz looks to be suffering from this so that’s one item of ours broken and it’s not on our side

 

[Ax^]

hmm so got a call at 7 am from the job complaining that they could not see something as the work system was down helped them out.o. the phone pointed out I'd taken the day off have fun with that


have a smoke and fall back asleep wake up midday "Global airline fuck up"


good day to take off


would not want to be one of those "why do we need a in-house server when we can do it all on a cloud based structure"


what happens if that fail , never gonna happen

 

[xenon]

My work colleague who's convinced that you should always use cash and have a petrol car because electronic payments and electric cars are actually a secret means of the state controlling you will be having a field day.

Plenty of idiots making similar comments on FB. Seemingly not able to grasp the fact if the EPOS, stock control systems, logistic system somewhere in the transaction chain is effected, his cash isn't going to buy him much.

 

[Lord Camomile]

He said, 'typical predictable IT disaster.'

And of course when it's over he will get told "Well it's a one-off it won't happen again so no need to worry"

I say this as someone with a neurological condition that results in me failing to take the sensible precaution all the fucking time, but sweet christ on a biscuit I honestly think this is one of my most violent day-to-day frustrations with humanity.

 

[newme]

Unfortunately we seem to be having some difficulty gathering that data just at this precise moment...

A low-lever supervising program woke up a slightly higher-level supervising program deep in the ship’s semisomnolent cyberbrain and reported to it that whenever it went click all it got was a hum.

The higher-level supervising program asked it what it was supposed to get, and the low-level supervising program said that it couldn’t remember what it was meant to get, exactly, but thought it was probably more of a sort of distant satisfied sigh, wasn’t it? It didn’t know what this hum was. Click, hum, click, hum. That was all it was getting.

The higher-level supervising program considered this and didn’t like it. It asked the low-level supervising program what exactly it was supervising and the low-level supervising program said it couldn’t remember that either, just that it was something that was meant to go lick, sigh every ten years or so, which usually happened without fail It had tried to consult its error look-up table but couldn’t find it, which was why it had alerted the higher-level supervising program of the program.

The higher-level supervising program went to consult one of its own look-up tables to find out what the low-level supervising program was meant to be supervising.

It couldn’t find the look-up table.

Odd.

It looked again. All it for was an error message. It tried to look up the error message in its error message look-up table and couldn’t find that either. It allowed a couple of nano-seconds to go by while it went through all this again. Then it woke up its sector function supervisor.

The sector function supervisor hit immediate problems. It called its supervising agent, which hit problems too. Within a few millionths of a second virtual circuits that had lain dormant, some for years, some for centuries, were flaring into life throughout the ship. Something, somewhere, had gone terribly wrong, but none of the supervising programs could tell what it was. At every level, vital instructions were missing, and the instructions about what to do in the event of discovering that vital instructions were missing, were also missing.

Small modules of software-agents-surged through the logical pathways, grouping, consulting, regrouping. They quickly established that the ship’s memory, all the way back to its central mission module, was in tatters. No amount of interrogation could determine what it was that had happened. Even the central mission module itself seemed to be damaged.

This made the whole problem very simple to deal with, in fact. Replace the central mission module. There was another one, a backup, an exact supplicate of the original. It had to be physically replaced because, for safety reasons, there was no link whatsoever between the original and its backup. Once the central mission module was replaced it could itself supervise the reconstruction of the rest of the system in every detail, and all would be well.

Robots were instructed to bring the backup central mission module from the shielded strong room, where they guarded it, to the ship’s logic chamber for installation.

This involved the lengthy exchange of emergency codes and protocols as the robots interrogated the agents as to the authenticity of the instructions. At last the robots were satisfied that all procedures were correct. They unpacked the backup central mission module from its storage housing, carried it out of the storage chamber, fell out of the ship and went spinning off into the void.

This provided the first major clue as to what it was that was wrong.

Further investigation quickly established what it was that had happened. A meteorite had knocked a large hole in the ship. This ship had not previously detected this because the meteorite had neatly knocked out that part of the ship’s processing equipment which was supposed to detect if the ship had been hit by a meteorite.

 

[Boris Sprinkler]

Reddit has a good thread on the crowdstrike forums

 

[ska invita]

would not want to be one of those "why do we need a in-house server when we can do it all on a cloud based structure"
what happens if that fail , never gonna happen

Local servers would be screwed too though if they had this Crowdstrike antivirus stuff running, no?

 

[Artaxerxes]

Reddit has a good thread on the crowdstrike forums

11000 comments and tales of woe, there were 6000 3 hours ago

 

[Artaxerxes]

Local servers would be screwed too though if they had this Crowdstrike antivirus stuff running, no?

Yep

 

[two sheds]

If that's true then oh shit.

All I'm seeing is NOT FOUND . Was there anything or did it just mean twitter was down?

 

[Ax^]

at least you have someone on the office to give the daggar eyes to instead of shaking your fist at the sky

 

[Boris Sprinkler]

Everywhere a windows agent exists with the ability to get that particular update pushed to it will be in the bsod loop.
It will be interesting to see what an analysis of that update reveals: bad coding practice, lack of change control, lack of testing.

 

[Bahnhof Strasse]

 

[ska invita]

Reddit has a good thread on the crowdstrike forums

so clearly reports of servers AND workstations bricking....one poster there reports 170,000 computers bricking.
like i said upthread could be months to sort this out in some cases
could imagine certain businesses folding

 

[ska invita]

Everywhere a windows agent exists with the ability to get that particular update pushed to it will be in the bsod loop.

Just to be 100% clear, if a home user isn't paying CrapStrike for their products there's no chance of receiving the update of death?

 

[weltweit]

If one had a Win10 laptop that hasn't been started since Monday but which is set up to automatically download and install updates.

Say I returned to that machine on Monday coming, should I just start it up normally? or would that doom me to downloading the dodgy update?

 

[Boris Sprinkler]

Exactly. And crowdstrike isn’t cheap, so any Security guys who got their boss to pay for it, is seriously going to have to grovel and is going to be so pissed at their CS account manager

Just to be 100% clear, if a home user isn't paying CrapStrike for their products there's no chance of receiving the update of death?

.

 

[two sheds]

11000 comments and tales of woe, there were 6000 3 hours ago

From there:

I have 40% of the Windows Servers and 70% of client computers stuck in boot loop (totalling over 1,000 endpoints). I don't think CrowdStrike can fix it, right? Whatever new agent they push out won't be received by those endpoints coz they haven't even finished booting.

 

[Boris Sprinkler]

If one had a Win10 laptop that hasn't been started since Monday but which is set up to automatically download and install updates.

Say I returned to that machine on Monday coming, should I just start it up normally? or would that doom me to downloading the dodgy update?

Have you paid for crowdstrike?

 

[weltweit]

Have you paid for crowdstrike?

Not that I am aware of no.

 

[Boris Sprinkler]

Then you will not be receiving any update.
This is a corporate security product.

 

[two sheds]

Other posts from reddit:

a fk up of this magnitude is the result of culture and process, not 1 engineer. Blameless PIRs are critical for things like this, it's how you understand the true cause that led you there, rather than just firing Gary because he was the one that pushed the button.

with several posts expressing sympathy for Gary, and

I mean, you cant say its not protecting you from malware if your entire system and servers are down.

 

[moochedit]

If one had a Win10 laptop that hasn't been started since Monday but which is set up to automatically download and install updates.

Say I returned to that machine on Monday coming, should I just start it up normally? or would that doom me to downloading the dodgy update?

It's an update to crowdstrike not a windows update so if you don't use crowdstrike as your anti virus you'll be fine.

 

[Lazy Llama]

I believe the broken file has been removed from updates so no more machines should download it. Not that that will help those who already have.

 

[Lazy Llama]

All I'm seeing is NOT FOUND . Was there anything or did it just mean twitter was down?

The Tweet said something along the lines that each broken machine was going to require manual intervention to fix. Must have been deleted now, not sure why because it seems to be true.

 

[two sheds]

Local NextDoor conspiracist responded to a thread about NHS IT system being down with "interesting"

 

[Artaxerxes]

Firms skip security reviews of updates about half the time

Complicated, costly, time-consuming – pick three

www.theregister.com

Crowdstrike released a press release about updates yesterday


I’m going to be laughing about this for ages

 

[Karl Masks]

Pornhub's ok, right?

 

[teuchter]

Less than optimal for drivers not to have signalling and platform data even by UK rail standards

View attachment 433969

Signalling is entirely unaffected. It's some of the customer info systems that are not working.

As far as I can see trains haven't really been disrupted much at all in the UK.