Global Outages Due to CrowdStrike Falcon Update’s BSOD (Blue Screen of Death): Crucial Steps and Recommendations

July 22, 2024
|In Security
|By Security Team

CyberSRC is committed to keeping our clients informed of critical cybersecurity issues. We are actively monitoring and advising on the recent issue with the CrowdStrike Falcon Sensor update, which has caused significant disruptions to Windows hosts. A faulty component in the latest update is causing widespread crashes in Windows systems, impacting various organizations and services worldwide. This glitch has led to massive outages for airports, TV stations, hospitals, and emergency services across the globe. The widespread impact of this faulty update highlights the critical dependence of various sectors on reliable cybersecurity solutions. The disruptions have underscored the need for robust contingency plans and prompt communication to mitigate the effects of such unforeseen technical issues. As affected organizations work to restore normal operations, this incident serves as a stark reminder of the potential ripple effects of a single software glitch on a global scale.

Impact on Critical Services

Emergency Services:

  • United States and Canada: Some 911 emergency service agencies in states like New York, Alaska, and Arizona, as well as parts of Canada, experienced significant disruptions.
  • Illinois: A 911 telecommunicator mentioned they were “working off of paper until things come back.”
  • Spain: In Catalonia, the health hotline was affected, prompting authorities to advise citizens to call 061 only in emergencies.

Aviation Disruptions

Airports Affected:

  • Netherlands: Schiphol Airport experienced disruptions, grounding several KLM and Transavia flights.
  • Australia: Melbourne Airport faced a “global technology issue” affecting check-in procedures, particularly for Jetstar and Scoot airlines.
  • Switzerland: Zurich Airport reported delays and cancellations, with no departures to the U.S.
  • Other Impacted Airports: Berlin, Barcelona, Brisbane, Edinburgh, Amsterdam, and London.

Airlines and Flight Cancellations:

  • Global Impact: Over 3,300 flights canceled worldwide.
  • United Kingdom: Long queues at London’s Stansted and Gatwick airports. Ryanair canceled several flights, advising passengers to check their accounts for options once the system was back online. British Airways also canceled multiple flights.
  • United States: American Airlines, United, and Delta grounded their flights worldwide for much of Friday, with assistance from the Federal Aviation Administration.
  • Japan and India: Airports in Tokyo and Delhi were also impacted.

Impact on Other Sectors

Payment Systems and Banking:

  • The update has disrupted payment systems and banking services globally, affecting numerous financial transactions and operations.

Healthcare Providers:

  • Netherlands: Scheper in Emmen, Slingeland Hospital in Achterhoek, and emergency posts in Hoogeveen and Stadskanaal were impacted.
  • Spain: Terrassa University Hospital and the Catalan Oncology Institute experienced issues but have started to return to normal activity.
  • United States: Bellevue Hospital in New York and NYU Langone Hospital were also affected.

Media Outlets:

  • Sky News: Off the air for several hours on Friday morning, unable to broadcast.
  • ABC and Other TV Stations: Experienced disruptions as computers crashed.

Transportation:

  • Railway Companies: Britain’s biggest operator, running Southern, Thameslink, Gatwick Express, and Great Northern, warned passengers of delays.

A Global Issue

The widespread impact of the faulty CrowdStrike Falcon update highlights the critical dependence of various sectors on reliable cybersecurity solutions. The disruptions caused by this update have underscored the need for robust contingency plans and prompt communication to mitigate the effects of such unforeseen technical issues. As the affected organizations work to restore normal operations, the incident serves as a stark reminder of the potential ripple effects of a single software glitch on a global scale.

 

Technical Details

  • Technical Details on the outage can be found here: Read the blog Published 2024-07-19 0100 UTC
  • CrowdStrike assure their customers that CrowdStrike is operating normally and this issue does not affect our Falcon platform systems. If your systems are operating normally, there is no impact to their protection if the Falcon Sensor is installed. Falcon Complete and OverWatch services are not disrupted by this incident.
  • CrowdStrike has identified the trigger for this issue as a Windows sensor related content deployment and we have reverted those changes. The content is a channel file located in the %WINDIR%\System32\drivers\CrowdStrike directory.
    • Channel file “C-00000291*.sys” with timestamp of 2024-07-19 0527 UTC or later is the reverted (good) version.
    • Channel file “C-00000291*.sys” with timestamp of 2024-07-19 0409 UTC is the problematic version.
    • Note: It is normal for multiple “C-00000291*.sys files to be present in the CrowdStrike directory – as long as one of the files in the folder has a timestamp of 05:27 UTC or later, that will be the active content.
  • Symptoms include hosts experiencing a bugcheck\blue screen error related to the Falcon Sensor.
  • Windows hosts which have not been impacted do not require any action as the problematic channel file has been reverted.

Non-Impacted Hosts

  • Windows hosts which are brought online after 2024-07-19 0527 UTC will not be impacted
  • Windows hosts installed and provisioned after 2024-07-19 0527 UTC are not impacted Updated 2024-07-21 1435 UTC
  • This issue is not impacting Mac- or Linux-based hosts

 

How do I Identify Impacted Hosts?

How do I Identify Impacted Hosts via Advanced Event Search Query? Updated 2024-07-22 0139 UTC

The queries utilized by the dashboards are listed at the bottom of the appropriate dashboard KB articles.

How do I Identify Impacted Hosts via Dashboard? Updated 2024-07-22 0139 UTC

An updated granular dashboard is available that displays the Windows hosts impacted by the content update defect described in this Tech Alert. See Granular status dashboards to identify Windows hosts impacted by content issue (v8.6) (pdf) or log in to view in the support portal. Note that the queries utilized by the dashboards are listed at the bottom of the appropriate dashboard KB articles.

How do I Remediate Impacted Hosts?

If hosts are still crashing and unable to stay online to receive the Channel File update, the remediation steps below can be used.

How do I Remediate Individual Hosts? Updated 2024-07-21 0932 UTC

  • Reboot the host to give it an opportunity to download the reverted channel file. We strongly recommend putting the host on a wired network (as opposed to WiFi) prior to rebooting as the host will acquire internet connectivity considerably faster via ethernet.
  • If the host crashes again on reboot:
    • Option 1 – Manual
      • Please see this Microsoft article for detailed steps.
        • Note: Bitlocker-encrypted hosts may require a recovery key.
      • Option 2 – Automated via bootable USB key

How do I Recover Bitlocker Keys? Updated 2024-07-21 1810 UTC

S No. Bitlocker Recovery Guidance for Knowledge Base (KB) Articles Resources
1 Updated 2024-07-21 1810 UTCMicrosoft Azure (PDF) or log in to view in the support portal.
2 Updated 2024-07-21 1810 UTCSCCM (PDF) or log in to view in the support portal.
3 Updated 2024-07-21 1810 UTCActive Directory and GPOs (PDF) or log in to view in the support portal.
4 Updated 2024-07-21 1810 UTCIvanti Endpoint Manager (PDF) or log in to view in the support portal.
5 Updated 2024-07-21 1810 UTCManageEngine Desktop Central (PDF) or log in to view in the support portal.
6 Updated 2024-07-21 1810 UTCBigFix (PDF) or log in to view in the support portal.
7 Updated 2024-07-21 0023 UTCBitlocker recovery without recovery keys (PDF) or log in to view in the support portal.
8 Workspace ONE Portal Omnissa article
9 Tanium Tanium article
10 Citrix Citrix article

 

Steps for Regaining Access to Windows PCs, AWS & Azure

For Windows PCs

  1. Boot Windows into Safe Mode or the Windows Recovery Environment.
  2. Navigate to the C:\Windows\System32\drivers\CrowdStrike directory.
  3. Locate and delete the file matching “C-00000291*.sys”.
  4. Boot the host normally.

For Cloud Environments Customers can revert to a snapshot taken before 04:09 am UTC.

For AWS (Amazon Web Services)

  1. Detach the EBS volume from the impacted EC2 instance.
  2. Attach the EBS volume to a new EC2 instance.
  3. Fix the CrowdStrike driver folder.
  4. Detach the EBS volume from the new EC2 instance.
  5. Attach the EBS volume back to the impacted EC2 instance.

For Azure (Microsoft Azure)

  1. Log in to the Azure console.
  2. Go to Virtual Machines and select the affected VM.
  3. In the upper left of the console, click “Connect”.
  4. Click “More ways to Connect” and then select “Serial Console”.
  5. Once SAC has loaded, type in ‘cmd’ and press Enter.
  6. Type ‘ch -si 1’ and press the space bar.
  7. Enter Administrator credentials.
  8. Type the following commands:
    • bcdedit /set {current} safeboot minimal
    • bcdedit /set {current} safeboot network
  9. Restart the VM.
  10. To confirm the boot state, run the command: wmic COMPUTERSYSTEM GET BootupState.

 

Cloud-Based Environment Recovery Resources

Cloud Environment Guidance
AWS AWS article
Azure Microsoft article
GCP (PDF) or log in to view in the support portal
Public Cloud/Virtual Environments Option 1:

  • ​​​​​​​Detach the operating system disk volume from the impacted virtual server
  • Create a snapshot or backup of the disk volume before proceeding further as a precaution against unintended changes
  • Attach/mount the volume to to a new virtual server
  • Navigate to the %WINDIR%\System32\drivers\CrowdStrike directory
  • Locate the files matching “C-00000291*.sys”, and delete them
  • Detach the volume from the new virtual server
  • Reattach the fixed volume to the impacted virtual server

Option 2:

  • ​​​​​​​Roll back to a snapshot before 2024-07-19 0409 UTC

 

 

Third Party Vendor Information Updated 2024-07-20 2259 UTC

Third Party Vendor Guidance
Intel vPro technology remediation guide Remediate CrowdStrike Falcon® update issue on Windows systems with Intel vPro® technology
Recovery for Rubrik customers CrowdStrike & Rubrik Customer Content Update Recovery For Windows Hosts
Cohesity Support Cohesity’s support for CrowdStrike’s Falcon Sensor updates

 

Actions Taken by CrowdStrike

  • The problematic update has been reverted.
  • CrowdStrike Engineering is fully mobilized to ensure the security and stability of customer systems.
  • Continuous updates and detailed guidance are available via the CrowdStrike support portal and public blog.

Recommendations from CyberSRC

CyberSRC recommend organizations to:

  1. Verify Updates: Before pushing any OEM updates directly into the production environment, system administrators should verify the potential impact. Updates and patches should be thoroughly tested in a staging environment prior to deployment in production.
  2. Review Disaster Recovery Plans: Ensure your disaster recovery (DR) plan is strong and effective. Organizations should review and update their DR plans to ensure they can handle business operation disruptions effectively.
  3. Communicate through Official Channels: Ensure communication with CrowdStrike representatives is through official channels to avoid misinformation.
  4. Implement Mitigation Steps: Follow the detailed steps provided in this advisory followed by CERT-IN, CISA & Crowdstrike to resolve the issue and restore system functionality.
  5. Remain Vigilant: Be aware of potential phishing and malicious activities exploiting this incident. Verify instructions from legitimate sources and educate employees on avoiding phishing emails and suspicious links.
  6. Stay Updated: Regularly check the CrowdStrike support portal for the latest updates and guidance.

 

Collaboration with Authorities

The Cybersecurity and Infrastructure Security Agency (CISA) and the Indian Computer Emergency Response Team (CERT-In) are aware of the issue and are working closely with CrowdStrike and other partners to support remediation efforts.

Contact CyberSRC for Support

CyberSRC is here to assist organizations in navigating this incident. If you require further security help or have any concerns, please do not hesitate to contact us.

Stay informed and stay secure with CyberSRC.

Note: Please note that the above recommendations are based on our understanding and assessment from various public and internal sources. Users are advised to conduct their own due diligence before implementing any recommendations.