Massive worldwide IT outage, hitting banks, airlines, supermarkets, broadcasters, etc. [19th July 2024]
- Thread starter cupid_stunt
- Start date
Chz
Stark Raving Sane
Well thank christ I missed out on that one. We only have ~50 Debian hosts, but they run some pretty important stuff and I'm really the only senior tech with any skill in it.
CrowdStrike broke Debian and Rocky Linux months ago, but no one noticed
CrowdStrike recently caused a widespread Blue Screen of Death (BSOD) issue on Windows PCs, disrupting various sectors. However, this was not an isolated incident, CrowdStrike affected Linux PCs also.www.neowin.net
Artaxerxes
Look out, he's got a gnu!
If he'd accidently gotten salt and vinegar instead of ready salted out of the machine because of it we'd be on 200 pages already
scifisam
feck! arse! girls! drink!
moochedit
I'm not feeding seagulls
Wait til he finds out his electronic toilet door lock isn't working
stdP
One Swiss dinosaur in Polly Gosling's anorak hood
Potentially dodged a bullet there ourselves; had to build some boxens directly facing t'interwebz some years back, and I chose Debian due to personal familiarity (running it for 20 years now more or less). Despite me and the in-house red team[1] spending months working on hardening the things, because of aforementioned procedural idiocy they failed the internal pen test report because they didn't have an antivirus installed (company didn't then and hasn't now an officially sanctioned AV for linux but the guidelines for the pen testers hadn't really wholly caught up with the heavy shift to *nix within the company). Thankfully the go-live decision is left up to the manager and, because the red team still couldn't get in even when three firewalls were disabled and they were given an ssh key for root, I was given the green light without having to install any AV. God bless ye SELinux.
Had some of the same madness at an insurance company I worked at some years back. Company standards mandated some form of AV regardless of the OS or exposure level; think they picked some McAfee bullshit for linux. Uptimes previously measured in years were reduced to days thanks to McAfee shitting on the kernel every week or so. In the whole time I remained there it didn't catch one virus but caused £$€ due to downtime on systems that were previously bulletproof.
Trending back in the general form of the thread... security is a process, not a product, and as much as the sales droids will like to tell you as such, you can't just press a single button to Make Things Secure. I've been of the impression and experience for years that most AV is largely useless, because by the time an AV notices something's gone wrong you're normally fucked anyway. Understanding of the nature of the exploits, in combination with defence-in-depth, is key. There's a load of vulns out there that flat out don't work when tested on our systems because we've got X, Y, and Z in place; luckily I work in a place where these can actually be tested if and when anyone cares enough. But that still doesn't stop the braindead security automation tools from flagging on the simplest of metrics; the recent openSSH vuln was a classic in this regard;
- This system is running SSH below v9.6! It's vulnerable.
- The vuln was only introduced in v8.6. This system is running v7.4. It isn't vulnerable.
- Computer says it is!
- Computer is a fucking idiot that can't understand a difference between less than X but more than Y.
- This system is running SSH below v9.6! It's vulnerable.
- System is indeed running v8.2. But it's 64bit Linux, not 32bit which the vuln was announced for. As of yet there's no exploit for 64bit openSSH, given that the exploit relies on the exceedingly small address space of 32bit so much that not even ASLR can save it. ASLR under the 64bit addressing means that under current understanding you'd need to keep the computer online for 37 years for the same vuln to work. It isn't vulnerable.
- Computer says it is!
- Computer's fucking wrong.
- This system is running SSH below v9.6! It's vulnerable.
- System is indeed running v9.3, but its running BSD as the OS. As you'd have understood if you'd have read the very well-written CVE report, the vuln is only present when openSSH is compiled against the linux-centric glibc. BSD doesn't use glibc and instead has a thread-safe version of the vulnerable function. It isn't vulnerable.
- Computer says it is!
- Computer's a fucking idiot that needs reprogramming with a very large axe.
My point, if it's even discernible at this point, is that there usually a huge pressure to patch stuff and install whatever update to whatever product Right Now! Because We Care About Security And I'm Sure The Vendor Has Done Their Own QA (RNBWCASAISTVHDTOQA for short); almost no-one, from vendor to client, bothers with QA, regression testing, testing patches in test environments or even a staggered patch rollout these days. Most people assume it'll just always work, because newer is always better, and someone above my pay grade must have tested this already, right?
[1]For those unfamiliar with the term at least in IT circles - I think it's descended from US military jargon in war games but "red team" is what they call "team who are on your side really but will use the known tactics of the enemy against you in a (hopefully) non-destructive manner in order to show you your weaknesses"; they characteristically use the exploit du jour against us to see if we're actually vulnerable to it, or whether our other mitigations we've put in place help protect against it, so we can gauge the impact correctly rather than knee-jerking to install a patch on day 1 instead of regression testing it. Fellow greybeards will understand it in the context of "white-hat" hacker (as opposed to "black-hat" [evil] or "grey-hat" [chaotic neutral]) terminology. Most places I've worked at or know about don't use this strategy, because it's probably very expensive in the real world and it's easier to just patch everything without question always. Apart from when the patch hasn't been properly tested and ends up killing you and feasting on your sorry carcass.
Last edited:
bcuster
Well-Known Member
7 things you need to know about the huge CrowdStrike IT outage
The outage, linked to a rogue update by cybersecurity firm CrowdStrike, affected operations around the world — with disruption likely to continue.
www.businessinsider.com
PR1Berske
Alligator in chains by the park gates.
It's a source of fascination, genuinely, which events pass people by or don't filter through.
_Russ_
Never not Knowingly Misunderstood
Though this was an accidental meltdown it's another reminder to vulnerable we are to a cyber attack.
The widespread use of Internet connectivity right down to using it as convenient remote administration and control of vital infrastructure is going to bite us hard one day
The widespread use of Internet connectivity right down to using it as convenient remote administration and control of vital infrastructure is going to bite us hard one day
Boris Sprinkler
Dont be scared
Yes. There’s an entire industry of people trying to navigate that. And it’s a thankless task, as everything is run according to growth and no consideration is given towards sustainability other than lip service and PR.
You know what’s really scary? Snmp/MIBS - udp scan Raw internet. Find devices. use default credentials. Profit.
You know what’s really scary? Snmp/MIBS - udp scan Raw internet. Find devices. use default credentials. Profit.
Last edited:
andysays
Love and solidarity
As it turned out, I managed to collect my prescription yesterday after all - the request from my GP had gone through before the shit hit the fan.
But the woman I spoke to at the pharmacy confirmed that their systems are currently down and they have no clear idea when they'll be up again.
8ball
Merr xmas...
I’m often a bit shonky keeping track of my med inventories so could easily have come a cropper here.
Hope someone has workarounds or a lot of people might start missing essential meds.
teuchter
je suis teuchter
This was one of my thoughts when I first read about this - do these things really get rolled out simultaneously everywhere and surely that's not a very good idea.
Boris Sprinkler
Dont be scared
Well that’s the thing. It’s the responsibility of each organization to be accountable for their estate.
I would expect that if they did have clauses for testing software before rollout to production, there probably would have been policy exceptions granted for security software running at such an executive level within the system.
There will be a shitload of work for corporate lawyers reviewing the contracts, procurement departments etc. there’s gonna be a lot of finger pointing at the security teams who insisted on this product and probably some firings as a result.
I would expect that if they did have clauses for testing software before rollout to production, there probably would have been policy exceptions granted for security software running at such an executive level within the system.
There will be a shitload of work for corporate lawyers reviewing the contracts, procurement departments etc. there’s gonna be a lot of finger pointing at the security teams who insisted on this product and probably some firings as a result.
keybored
Well done. You remember cat good.
Also, I don’t think this has been given enough consideration, for the devastating impact it had, I had to pay for my takeaway delivery with cash yesterday. Neither Apple Pay or card payments were working.
Regarding antivirus on Linux. Anyone using ClanAV? Tried it out a few years ago but I’m not knowledgeable enough to say how good it is. I don’t use it now on the handful of Linux machines I have access to. Used to mess around with this stuff a lot more when I was trying to get into IT security. Snort, tripwire. I stepped away from that for boring accessibility reasons. But it is still interesting to me.
Last edited:
If it’s a patch that fixes a zero day exploit…
High Voltage
In the top 97% of Urban's most interesting posters
The better of the two Chinese takeaways that we had one from yesterday has always been cash only for years and, to my certain knowledge, four changes of ownership
Special fried rice and special Foo Yung if you're interested
stdP
One Swiss dinosaur in Polly Gosling's anorak hood
Without going in to too much detail, as Boris Sprinkler points out there's usually the assumption that security software like crowdstrike will be continually updating 24/7. Many of these utilities don't even offer the possibility of deferring updates so you can test in tranches. And like xenon says, if there's a patch for a 0-day (chrome is utterly notorious for this) then it'll typically be rolled out ASAFP.
Assuming you mean ClamAV - I've only found it particularly useful as a box-ticking exercise, it didn't actually do much to detect intrusion attempts; TBH most of the really nasty viruses these days sail right through almost every detection engine[1] so I regard them of comparatively low utility. Defence-in-depth (mentioned it before but SELinux and other forms of mandatory access control are brilliant for blocking whole classes of exploits) in combination with things like snort and tripwire and other laundry lists of IDS and IPS are far more reliable indicators when you might have rogue code running.
[1]The recent backdooring of libxz (a compression library) is a great read for the way they managed to hide the malicious payload in "plain sight", and also a great demonstration of why I dislike systemd so much
moochedit
I'm not feeding seagulls
two sheds
Least noticed poster 2007
I've got ClamAV installed and it is updating the definitions ok - just ran a scan and it's not found anything. As I recall the only viruses it's detected are in in old mail programs/emails, presumably dodgy attachments.
An introduction to that here from Crowdstrike: Defense in Depth [Beginner's Guide] - CrowdStrike
anything here that can be installed on a desktop to improve security?
I presume not much to be done if systemd is already installed?[1]The recent backdooring of libxz (a compression library) is a great read for the way they managed to hide the malicious payload in "plain sight", and also a great demonstration of why I dislike systemd so much
BigMoaner
What A Load of Old Bollocks
I am sorry. I am so so sorry.
PR1Berske
Alligator in chains by the park gates.
That account appears to blame a "DEI engineer" for the issue. I think we can safely dismiss his warbling.
Artaxerxes
Look out, he's got a gnu!
No impact
Boris Sprinkler
Dont be scared
Lower impact than a worm or similar. As the difference between this and security incidents is here the facts were relatively easy to ascertain. There were no unknowns. Compare to the absolute mayhem sql slammer caused (due to not many places been on top of vulnerability management, nor were backs up routine then on all systems. Or the total wack a mole attempts to contain conficker on a a large global windows network.
Maybe this will prompt organizations to look at their 3rd party risk. It’s often way down the list.
Maybe this will prompt organizations to look at their 3rd party risk. It’s often way down the list.
Throbbing Angel
this is no longer a place of honour
TL;DR
Q: Should I be updating my Win11 laptop tonight or deferring them for a week or so?
Q: Should I be updating my Win11 laptop tonight or deferring them for a week or so?
Update it. I presume you’re not running cloud strike or it would already be broken. with a caveat there might be a Microsoft bug all of their own. But I don’t know about.
Similar threads
