Talking to the Bank of England about systemic risk and systems engineering

Patrick McKenzie opened his Bank of England seminar—and the public derivative released on 2025-10-23—by arguing that financial stability policy cannot stay in the clouds: regulators must talk to engineers and understand implementation details. He framed the talk around single points of failure (SPoFs) and horizontal vs vertical scaling, then used the July 2024 CrowdStrike Falcon outage as a concrete case study of how a technical incident at one vendor propagated into broad operational and systemic effects across the US financial system. McKenzie repeatedly emphasized that implementation choices and organizational incentives determine how failures cascade.

The CrowdStrike episode is the centerpiece. McKenzie recounts reporting (Wall Street Journal) that five of the top ten U.S. banks were heavily hit on a random Friday in July 2024: teller PCs presented Blue Screens of Death, preventing teller-driven withdrawals and branch services for much of the day. Although CrowdStrike framed the outage as about 90 minutes of defective behavior, McKenzie documents why the real operational disruption lasted across the business day and required teams working through the weekend to fully recover. He attributes the technical root cause to a pushed signature/definition file with 21 parameters where the 21st was unexpectedly non-wildcard; interpreted by kernel-mode EDR code this caused out-of-bounds memory access and machine crashes. That kernel-mode decision—chosen for deeper visibility—was later criticized by Microsoft because boot-time kernel failures are hard to recover from.

Beyond the immediate bug, McKenzie focused on systemic drivers that multiplied harm. He explains how FFIEC guidance (IT Examination Handbook, Sept 2016), regulatory expectations, and vendor sales playbooks resulted in a near-monoculture of endpoint monitoring within certain regulated sectors. Even if multiple EDR vendors exist globally, vendors had effectively split markets such that US banking ended up heavily concentrated on CrowdStrike Falcon; CrowdStrike’s automated push model for signature files left customers unable to intercept or roll back a definition that propagated to hundreds of thousands of endpoints. McKenzie also called out engineering-process weaknesses: automated testing emphasized code review but not data validation for signature blobs; organizational politics meant high-status groups (trading) resisted EDR installs while customer-facing, lower-status machines (tellers) were prioritized and—ironically—became the most visible casualties.

Operational responses, he said, both mitigated and revealed fragility. Many banks had printed branch phone lists, telephone trees, and standing instructions (e.g., "do not close branches") that prevented mass panic and allowed diversion to ATMs or digital rails like Zelle. But these stopgaps exposed other faults: ATMs ran low on cash in cities like Chicago, some automatic fraud heuristics flagged unusual withdrawals and failed to clear because web-based confirmation subdomains were also affected, and smaller local banks ran out of physical cash quickly when large banks could not service customers. McKenzie contrasted the FCA’s post-mortem claim of "minimal consumer harm" with on-the-ground anecdotes of payroll and cash problems, calling the episode a near miss. He recommended regulators ask for blameless incident postmortems, collect supervised firms’ incident reviews, insist on data-as-code testing for pushed artifacts, consider enterprise-level opt-outs for pushed definitions, and prefer architectural controls (diversity of OS and out-of-band control-plane communications) to reduce correlated failures. Finally, he warned that a coincident market-stress event would have magnified these technical failures into a full systemic crisis—so policy should aim to reduce monocultures and improve operational transparency now, not after the next near miss.