CrowdStrike Incident: What Went Wrong and Lessons Learned

CrowdStrike, a leading cybersecurity company, recently experienced the biggest technical slip-up of the century. This hiccup did not just affect CrowdStrike. It unfortunately sent ripples through the whole security scene, making folks wonder about the safety of their digital fortresses.

It is a wake-up call that has got everyone from tech whizzes to business bigwigs talking. Buckle up—it is going to be an eye-opening ride! 

Table of contents:

I. Understanding CrowdStrike’s technology and the incident

• The technology
• The incident

II. Why were Apple and Linux unaffected by the CrowdStrike incident? 

III. CrowdStrike's crisis management: Hits and misses

IV. Lessons for business leaders

Understanding CrowdStrike’s technology and the incident

The technology

CrowdStrike specialises in endpoint security, focusing on prevention, detection, and response to advanced threats. Its platform leverages cloud-based technology, artificial intelligence, and machine learning to provide real-time protection against a wide range of cyberattacks. 

Endpoint security is a cybersecurity approach that focuses on protecting individual devices, such as laptops, desktops, and mobile devices, from malicious attacks. These endpoints are often the entry points for cybercriminals, making their protection crucial. 

However, endpoint security played a central role in the recent CrowdStrike incident. 

The incident stemmed from a faulty update to CrowdStrike's Falcon sensor. Falcon is a core component of their endpoint protection platform, designed to combat a wide range of threats, including:

• Malware: Viruses, worms, ransomware, and other malicious software. 
• Exploits: Zero-day vulnerabilities and other software exploits. 
• Fileless attacks: Threats that operate in memory without relying on traditional malware files. 
• Ransomware: Attacks that encrypt data and demand payment for decryption. 
• Insider threats: Malicious actions by employees or contractors. 
• Supply chain attacks: Compromised software or hardware used to infiltrate networks. 
• Advanced persistent threats (APTs): Sophisticated, long-term attacks by nation-states or organised criminal groups. 

Read more: Cybersecurity Threats Loom Large for Vietnam's Financial Sector 

What does CrowdStrike’s Falcon platform have to do with Windows systems?

Falcon integrates seamlessly with Windows systems through its lightweight sensor. The sensor resides in the Windows directory and uses configuration files called "Channel Files" to control various protection mechanisms. These files are regularly updated to respond to new threats and tactics observed by CrowdStrike. 

These features work together to provide comprehensive protection, whether the endpoint is online or offline¹.

The incident

On July 19, 2024, at 04:09 UTC, CrowdStrike released a sensor configuration update to Windows systems as part of their ongoing operations. The update targeted Channel File 291, which controls how Falcon evaluates named pipe execution on Windows systems. Named pipes are used for normal interprocess or intersystem communication in Windows. The update aimed to target newly observed, malicious named pipes. 

The configuration update triggered a logic error, resulting in an operating system crash. The crash affected Windows hosts running Falcon sensor version 7.11 and above that were online and received the update between 04:09 UTC and 05:27 UTC on July 19, 2024². 

The Falcon sensor program is installed locally but serviced from the cloud. It operates in conjunction with and regularly connects to CrowdStrike's servers. With this setup, users do not have to check for or trigger the update installation manually. Unfortunately, this is exactly what accelerated the spread of the defective update. 

Microsoft estimates that the faulty update affected around 8.5 million Windows computers. This is a small percentage (less than 1%) of all Windows computers globally. The impact of the catastrophe remains significant nonetheless, impacting client computers and servers across various industries³. 

Fortunately, CrowdStrike was able to release a fix for the issue relatively quickly. The process of restoring affected systems took several days for many organisations.

This highlights several key points about endpoint security and the incident: 

• Centralised management: The incident underscores the potential risks associated with centrally managed endpoint security solutions. While they offer efficiency benefits, a single point of failure can have widespread consequences. 

• Update management: The importance of rigorous testing and quality assurance processes for software updates is evident. Even established security vendors can be susceptible to errors. 

• Dependency on third-party software: The incident emphasises the reliance of many organisations on third-party security solutions, highlighting the need for robust contingency plans and disaster recovery procedures.

It is important to note that while this incident involved CrowdStrike, it serves as a reminder for all organisations to carefully evaluate their endpoint security strategies and ensure they have appropriate safeguards in place. 

Read more: What is Cybersecurity? Everything You Need to Know 

Back to the top

Why were Apple and Linux unaffected by the CrowdStrike incident? 

Apple and Linux users were unaffected by the incident. This discrepancy can be attributed to several key factors: 

• Operating system architecture: Apple's macOS and Linux distributions have more restrictive environments for third-party software, limiting their ability to interact deeply with the operating system. This reduced dependency makes them less susceptible to system-wide issues caused by software conflicts. 

• Update and deployment mechanisms: Apple and Linux distributions often use more gradual update processes. New software versions are rolled out to smaller user groups before wider deployment. Their users typically have more control over when and how updates are installed, reducing the risk of unexpected system disruptions. 

• Ecosystem differences: Apple maintains a tightly controlled ecosystem for hardware and software, reducing the likelihood of compatibility issues and security vulnerabilities. Despite being open-source, Linux's distribution models and package management systems often incorporate rigorous testing and quality control processes. This allows Linux to minimise the risk of widespread issues.

While this specific CrowdStrike incident did not directly impact Apple and Linux systems, they are not immune to cybersecurity threats. A robust security posture, including regular updates, user education, and additional security measures, remains essential for all operating systems. 

Back to the top

CrowdStrike's crisis management: Hits and misses 

Swift acknowledgement and transparency 

When the outage hit, CrowdStrike CEO George Kurtz stepped up to the plate right away. He took to social media and other channels to keep everyone in the loop. By being upfront and clear about what was going on, Kurtz managed to keep the rumour mill at bay. 

He was straightforward, stating that the issue was a technical glitch rather than a security breach. This straight talk was key to keeping clients on board⁴. 

Rapid deployment of fixes 

CrowdStrike's tech team quickly pinpointed the issue in the Falcon Sensor software and got to work on a fix. Their speedy response helped limit downtime and showed they have an established process to handle crises like this4. 

Brody Nisbet, CrowdStrike's Director of Overwatch, shared a workaround on social media that involved booting Windows machines into safe mode, deleting a specific file, and rebooting. This hands-on approach helped some customers get back on track faster⁵. 

Read more: How to Implement A Disaster Recovery Plan to Protect Your Financial Data? 

Areas for improvement in crisis response 

CrowdStrike's slip-up highlighted the need for more rigorous testing before updates go live. The fallout was significant, with critical industries like aviation and healthcare taking a hit. Delta Airlines, for instance, faced major operational issues, leading to flight delays and cancellations. This domino effect showed just how vulnerable our interconnected systems can be when IT goes haywire. 

The financial impact was hard to ignore too. CrowdStrike's stock took an 11% nosedive, and analysts started lowering their ratings. This financial hit underscores the importance of keeping operations running smoothly to maintain market confidence 4. 

Back to the top

Lessons for business leaders 

The CrowdStrike incident serves as a stark reminder that even the most advanced cybersecurity solutions are not infallible. Here is what business leaders can learn: 

Robust incident response plan

• Prioritise incident response planning: Develop a comprehensive plan outlining roles, responsibilities, and communication channels. 

• Regular testing and updates: Conduct simulated attack scenarios to identify vulnerabilities and refine response procedures. 

• Communication strategy: Establish clear communication protocols for internal and external stakeholders during a crisis. 

Read more: Key Elements of a Disaster Recovery Plan

A diversified security strategy

• Risk assessment: Identify critical systems and data, and prioritise protection efforts accordingly. 

• Layered security: Implement multiple security controls to create a defence-in-depth approach. 
• Vendor diversity: Avoid overreliance on a single security vendor.

Continuous monitoring and evaluation

• Real-time threat monitoring: Utilise advanced threat detection tools to identify potential attacks early. 

• Regular security audits: Conduct thorough assessments of your IT infrastructure to identify vulnerabilities. 

• Employee training: Educate employees about cybersecurity best practices to prevent human error. 

Business continuity planning

• Develop contingency plans: Outline steps to maintain critical operations in case of a cyberattack. 

• Data backups: Regularly back up critical data to ensure business continuity. 

• Disaster recovery testing: Conduct drills to validate your response capabilities. 

The CrowdStrike incident highlighted the importance of careful planning in cloud-based systems. However, this should not overshadow the many benefits the cloud offers, such as scalability, flexibility, and enhanced security.

The vital lesson is taking a proactive and layered approach to cybersecurity. Instead of avoiding cloud solutions, businesses should focus on selecting reputable providers with strong security track records and implementing robust security controls. 

This setback can be a chance for the security industry and businesses to create stronger, more reliable systems together. By learning from this experience, we can build a safer digital world for everyone.

Back to the top

References

1. https://www.crowdstrike.com/blog/falcon-update-for-windows-hosts-technical-details/

2. https://siliconangle.com/2024/07/24/crowdstrike-reveals-cause-faulty-update-led-windows-crashes/

3. https://www.isc2.org/Insights/2024/07/After-the-CrowdStrike-Outage-What-Can-We-Learn

4. https://www.forbes.com/sites/goldiechan/2024/07/23/crowdstrike-crisis-big-brand-lessons-for-what-went-right-and-wrong/

5. https://www.lightedge.com/blog/resources/lessons-from-the-crowdstrike-it-outage/