A current worldwide #CrowdStrike issue causing #BSOD. Seen reports from AU, NZ ,Japan, India. And Europe. The global computer outage affecting airports, banks and other businesses.
CrowdStrike’s cybersecurity software — used by numerous Fortune 500 companies, including major global banks, healthcare and energy companies — detects and blocks hacking threats. Like other cybersecurity products, the software requires deep-level access to a computer’s operating system to scan for those threats. In this case, computers running Microsoft Windows appear to be crashing because of the faulty way a software code update issued by CrowdStrike is interacting with the Windows system.
This issue is not impacting Mac- or Linux-based hosts
Some servers on perm and cloud and devices are not resuming correctly and are getting stuck in boot loops that have #Crowdstrike.
Some seen successful reboots which work for about 15 mins and then they stop and then go back into a boot loop.
Technical Breakdown
1. Crowdstrike publishes a content update for their threat feed, which is basically a list of patterns of “bad things”
2. Software agents get this update and apply the controls to block things that match this pattern
3. The update has a pattern which matches a critical Windows process but the software blocks it anyway
4. Windows crashes with a Blue Screen of Death (BSOD) and reboots
5. On reboot, CrowdStrike kills the process again and Windows reboots
6. And it’s now a loop… There are various ways of fixing this but for most systems it will involve physically visiting every affected system, booting into “safe mode” and fixing the problem manually.
For some cloud systems though, such as AWS, “safe mode” is not even possible so this fix doesn’t work. The virtual servers will need to be shut down, their disks cloned, attached to another server, edited to remove the offending files and then finally reattach to the original server.
BUT, if you’re protecting your data and using encryption at rest, you need to manually decrypt the disk with a BitLocker Recovery Key, which is probably - for most companies
Updated workaround steps:
Boot Windows into Safe Mode or the Windows Recovery Environment
Navigate to the C:\Windows\System32\drivers\CrowdStrike directory
Locate the file matching “C-00000291*.sys”, and delete it.
Boot the host normally.
Crowdstrike published a post with updated details for quering machine and how to fix here
https://www.crowdstrike.com/blog/statement-on-falcon-content-update-for-windows-hosts/
This is really just a good reminder of how MANY systems are dependent on IT.
Technology is engrained in every part of our lives. We don’t notice it when it’s working well. We only notice when something goes wrong. No one talks about how many millions of attacks were stopped, or upgrades that went smoothly. Everyone remembers the ones that didn't.
#Crowdstrike #update #BSOD #EDR #outage #ITissue
