CrowdStrike outlines fixes in root cause report after global outage
CrowdStrike has released its root cause report of the faulty software update that caused one of the biggest IT outages in history. In its new post-mortem report, the cybersecurity firm investigated the error that led Windows machines to crash in July, admitting that there were issues with the testing process.
In its Root Cause Analysis (RCA) report, described how its Falcon sensor “delivers AI and machine learning to protect customer systems by identifying and remediating the latest advanced threats.”
The sensor, released in February, was produced to enable “visibility into possible novel attack techniques that may abuse certain Windows mechanisms.
“On March 5, 2024, following a successful stress test, the first Rapid Response Content for Channel File 291 was released to production as part of a content configuration update, with three additional Rapid Response updates deployed between April 8, 2024 and April 24, 2024,” CrowdStrike said. These “performed as expected” in production.
However, the sensor expected 20 input fields, but the update provided 21 input fields, causing a mismatch. This resulted in an out-of-bounds memory read, crashing the system.
The company stated that “this scenario with Channel File 291 is now incapable of recurring,” adding that what happened is now informing how it tests its systems going forward.
In a post on X, the firm wrote: “We apologize unreservedly and will use the lessons learned from this incident to become more resilient and better serve our customers. To any customer still affected, please know we will not rest until all systems are restored.”
This morning, we published the Root Cause Analysis (RCA) detailing the findings, mitigations and technical details of the July 19, 2024, Channel File 291 incident. We apologize unreservedly and will use the lessons learned from this incident to become more resilient and better…
— CrowdStrike (@CrowdStrike) August 6, 2024
CrowdStrike promises new test methods in root cause report
Based on the findings in the report, CrowdStrike said it will upgrade its Content Configuration System test procedures, including updated tests for Template Type development, with “automated tests for all existing Template Types.”
It is also incorporating deployment layers and acceptance checks into the Content Configuration System.
Meanwhile, it will block the creation of problematic Channel 291 files by adding validation for the number of input fields.
CrowdStrike plans to introduce more checks in the Content Validator and improve bounds checking in the Content Interpreter for Rapid Response Content in Channel File 291.
It will also enlist “two independent third-party software security vendors” to perform additional reviews of the Falcon sensor code as well as the quality control and release processes.
Last week, investors filed a lawsuit against the company. CrowdStrike and Delta’s CEO were also engaged in a public dispute after the airline blamed $500 million in losses on the security firm. The company’s chief executive, George Kurtz was called to testify before Congress last month.
Featured image: Ideogram / Canva
The post CrowdStrike outlines fixes in root cause report after global outage appeared first on ReadWrite.