On Friday last week, CrowdStrike’s update caused ‘Blue Screen’ errors on Windows systems globally, affecting Microsoft’s ecosystem.
Microsoft reported that this issue impacted 8.5 million Windows devices, less than 1% of all Windows machines. Despite the small percentage, the disruption affected critical services.
David Weston, Microsoft’s Vice President of Enterprise and OS Security, stated that Microsoft has been in constant communication with customers, CrowdStrike, and external developers to gather information and expedite solutions. They have taken several steps:
- Collaborated with CrowdStrike to develop a workaround and provided instructions on the Windows Message Center.
- Deployed hundreds of engineers to work directly with customers to restore services.
- Coordinated with other cloud providers, including Google Cloud Platform (GCP) and Amazon Web Services (AWS), to share impact awareness and inform ongoing discussions.
- Posted manual remediation documentation and scripts.
- Updated customers through the Azure Status Dashboard.
Microsoft emphasized their continuous efforts to support affected users and developed a scalable solution with CrowdStrike to accelerate fixes. They also collaborated with AWS and GCP on effective approaches. Significant incidents like this are rare, noted Weston.
Recovery Tool
Microsoft released a new recovery tool to address the CrowdStrike issue on Windows endpoints. The tool, available in the Microsoft Download Center, offers two repair options:
- Recover from WinPE: Creates boot media to facilitate device repair.
- Recover from safe mode: Creates boot media to boot impacted devices into safe mode, allowing login with admin privileges and running remediation steps.
This tool creates a bootable USB drive for quick recovery, bypassing the need for Safe Mode or admin rights. It accesses the disk directly and deletes the problematic CrowdStrike file. If the disk is protected by BitLocker, it prompts for the recovery key.
Separate recovery steps are available for Windows Virtual Machines on Azure and for all Windows 10 and Windows 11 devices on Microsoft’s support site.
CrowdStrike Guidance
CrowdStrike also provided new guidance for dealing with the Windows outage. The Falcon sensor update, issued between 04:09 UTC and 05:27 UTC on July 19, 2024, caused the system crashes.
CrowdStrike reverted the problematic update. Key details include:
- Affected systems: Falcon sensor for Windows version 7.11 and above.
- Problematic configuration file: “C-00000291*.sys” with a timestamp of 04:09 UTC.
- Reverted file: “C-00000291*.sys” with a timestamp of 05:27 UTC or later.
- Symptoms: Bugcheck/blue screen error related to the Falcon Sensor.
- Non-impacted systems: Hosts brought online after 05:27 UTC on July 19, 2024, or those installed after this time.
CrowdStrike assured customers that their Falcon platform systems remain operational and unaffected, including Falcon Complete and OverWatch services.
CrowdStrike also said that continuous updates are available on the CrowdStrike Support Portal and urges users to contact their representatives for additional support.
CrowdStrike CEO George Kurtz stated that the issue was due to a defect in a Falcon content update for Windows hosts and was not a cyberattack. They are working closely with impacted customers to restore systems.