Talking to the Bank of England about systemic risk and systems engineering

title: Talking to the Bank of England about systemic risk and systems engineering
author: Complex Systems with Patrick McKenzie (patio11)
contenttype: podcast
publication: Complex Systems with Patrick McKenzie (patio11)
published: 2025-10-23T07:02:03
sourceurl: https://pscrb.fm/rss/p/prfx.byspotify.com/e/media.transistor.fm/9b03fa71/29ec2730.mp3

word_count: 16118

Welcome to Complex Systems, where we discuss the technical, organizational, and human factors underpinning why the world works the way it does. Hi, do you have one? My name is Patrick McKenzie, better known as patio 11 on the internet. I recently received an email from the Bank of England, which was somewhat surprising to me, because I'm not normally the kind of person that gets emails from central banks. And the Bank of England said, dear Mr. McKenzie, et cetera, et cetera, et cetera, we wonder if you could please give us a private seminar on things that software engineers understand that us central bankers might not. And so I skid that all down to Threadneedle Street over in London to give them a private seminar and the bank graciously gave consent to have me give a derivative of that talk to the public internet. I'll make two mandatory disclaimers. One, these are only my own views and perceptions, and they are not endorsed in any fashion by the Bank of England. Two, I previously worked at Stripe, and I'm still an advisor there. Stripe does not necessarily endorse what I say in my own spaces or in my own capacity. So I'm not necessarily the kind of person that you would expect to address the central bank. I can barely tell camels from horses, and I haven't met basil yet, but I assume he's a very nice chap. In fact, the one time I thought I would come down to Threadneedle Street was due to the demonetization of the old pound note where you had to exchange it with your high street bank or with Bank of England directly and not being banked in the United Kingdom. I thought I would have to come down before I speak talk to Teller into exchanging 70 old pounds for new pounds. But I have learned a few things about a few things in the course of a long and interesting career around financial infrastructure, and thought I would take some of the information about how things are actually built and how they break and share that with you today. And I'd encourage you as policymakers and people who are responsible for this for the entire nation of the UK to spend more time with engineers and other people who are close to the implementation level of the policies that you oversee. I think that often in policy circles, we spend our heads a bit in the clouds and defy all your implementation into being somebody else's problem. But the financial infrastructure of the world, the technical substrate that we all exist on and that our economy has exist on, cannot be separated from the implementation level. You cannot understand what is actually happening without some level of technical understanding of what is happening. And so I think it's incumbent upon various actors to both have engineers and other experts in the room, one of these policies are drafted, debated and discussed. But also to make connections with people such as myself or any of the other million engineers that you can find on the Internet to get some color on the actual detail of the things that are being built. And so with that, you've asked me to address three subtopics under the broad heading of software that undergirds financial markets and I've brought my own personal views on them to you. You probably have forgotten more about financial stability and systemic risks than I will ever know. But we do have one interesting concept in engineering land about how failures tend to propagate out from where they happen. And there is one bit of engineering jargon I'd like to leave you with SP OF, a single point of failure. So broadly, when you're scaling engineering systems, you have two choices. You can either scale vertically just by an increasingly big server to run your database. It'll work out or scale horizontally. You have multiple independent copies of a thing and when you want to scale, just increase the number of copies that you have. One advantage from a resiliency perspective of scaling horizontally is that you can lose some of the nodes in the system and have that be a non-effect for the overall availability of the system because you have many nodes backing them up. There are some things in systems which have the characteristic where there is only one copy of them, a single point of failure, the unavailability or downtime or brown time of that particular node means the system is likely to grind into a crashing halt. Sometimes this is unavoidable. Sometimes this is a considered engineering choice. The cost of resilience is a cost and sometimes it's not a cost that you want to bear for a particular application. But there's a dangerous pattern where you have a single point of failure and are not aware of the fact of having that single point of failure. I want to present an example to you. So, help the American out with a remedial civics lesson. Can the monarch of England unilaterally choose to close the banks? And AI tells me that yes, they can, but it hasn't happened since Charles I and that authority has been circumscribed by Parliament in the intervening years. Okay, fair enough. So, question for your new guest. Can anyone in this room unilaterally close the banks? Do you have a big red button sitting on your desk somewhere where slam and all the deposits and loans stop immediately? Oh, drafts. I was hoping to see that. All right. Follow up question for you. Can a 22-year-old engineer close all the banks? What's your answer and why do you think you're so confident in that answer? So, in the United States, we thought we were pretty confident that no 22-year-old actually had the authority to close all the banks. And it turned out they had the power to do something which rounds strikingly similar to that. So, I want to read you a headline from July, 2024, ancient history. Major tech outage, grounds, flights, hits banks, and businesses worldwide in the Wall Street Journal. And I want to talk about CrowdStrike and CrowdStrike Falcon and how it resulted in a large portion of the United States. Banking infrastructure, hitting a blue screen of death in Microsoft Windows, and failing in its goals for being available to the population of the United States. And the world wide economy that relies upon us. And so, if you haven't been familiar with the words BSOD, it stands for Engineering, Jerkin, Pioneer, Search, Over Device. No, I lied. It stands for Blue Screen of Death. You're probably familiar with this if you're of a certain age like myself. We very frequently saw these in the Windows 95 days and the Windows 3.1 days. We haven't seen them since Vista in any quantity. But suddenly, every bank, well not every bank. A large portion of the largest US banks, the Wall Street Journal sites 5 at the top 10. Saw the blue screen of death on almost every PC in certain lines of business, starting at a particular point in the morning on a Friday, and continuing through basically the end of the day, completely stopping the ability of those lines of business to work. Now, I regret that you're going to have to take some representations I make here on faith. I would hope to be able to point you to a post-mortem conducted by, for example, the Federal Reserve Board of Governors. Unfortunately, the Fed is a little bit busy at the moment as you might see if you've read headlines such as Fed Independence reaches its moment of truth as Supreme Court ways of Cook's fate. And other various sessions out against happening in Washington at this time of year. But be that as it may, there was actually one post-mortem conducted by a financial regulator on the CrowdStrike Outage of July 2024. It was actually conducted by the financial conduct authority here in the UK. And that post-mortem, as CrowdStrike is widely used, we saw varying degrees of operational impact on regulated firms with no sector more impacted than others and minimal consumer harm. I'm going to lightly push back against the FCA's characterization that there was minimal consumer harm. I think there was actually quite a cognizable amount of consumer harm to consumers, particularly in the United States of America. And we're over, this was a very near miss sort of situation. We got lucky in having only as much impact as we saw. And as you saw, much of the media coverage of this issue focused on the grounding of airline flights, partly because that is maximally disruptive to business travelers on a day when they're traveling. And partly because it's very easy for people to verify independently that airline flights did not happen where it's relatively difficult for journalists to verify independently that bank withdrawals did not happen. This epistemic logic of how media happens and it's complicated relationship with the financial industry is an interesting thing to keep in the back of your head. To conduct sense making as the state because the newspapers are probably not lying to you per se, but they're also not telling exactly the truth as regulated firms understand it. And with the greatest amount of difference to the FCA, I don't think that the culture that is financial institutions necessarily tells you their internal truth 100% of the time. I think that the culture that is financial institutions interacting with the regulators frequently trans towards a shoot shovel or shut up when an issue is discovered where if we were to allocate to the depth of how bad things got in a particular engineering incident that would invite regulatory scrutiny invite an inquiry and invite throwing this firm or a regulator or some other powerful institution under the bus. And so it's always easier to say, well, no big problem happened here. And to continue saying that until the evidence becomes absolutely undeniable that there was in fact a gigantic problem. And if you want to look close to home, the experience of the UK subpost masters who were sent to prison essentially by a non functioning software system is extremely instructive here. The UK post and the consulting firm that created the software system were adamant for many years that there was no problem software was the best software. It was adding numbers how hard could that be. And a tremendous miscarriage of justice happened. So we have a relatively new culture in engineering land for what's called blameless post mortems where instead of conducting an inquiry and finding whose political career we need to sacrifice or who we need to throw under the bus for some engineering issue. We focus on just getting to engineering truth as quickly as possible and as reliable a fashion as possible. And a decouple the quote unquote accountability from the statements of engineering fact with the goal of getting systems back up and running from the condition of being under incident to the condition of being working again as quickly as possible. Collecting a reliable corpus of information from those incidents so that can inform the future craft of engineering both at our firms and in the industry broadly and establishing this common language for how we talk about incidents both that during the progress of the incident and afterwards. I think we will find that many financial technology companies have conducted many many more incident reviews than you would guess at. And that these are not considered supremely sensitive in fact some of the best information you'll find about the crowd strike vulnerability was the public post mortems that were published by crowd strike and Microsoft respectively. And I bet that supervised firms would happily give you their backlog of incident reviews at a variety of scales of how impactful the incident was how concerning was how much remediation was required and how much customer harm actually happened. And I think they will be particularly interested in giving these to you if you extend them in alif ranch and say look we're not trying to bust your gut here we are all on the same team. And we'd like to understand both what generally happens when the excrement is hitting the oscillating fan and how you generally act with regards to that because we also have responsibilities when you're acting to do incident response. And we would like to better understand the landscape that we were working at so that we can both handle our own responsibilities as the central bank and also socialize these responses within the broader community of practice. We think there is something that we can actually learn from financial technology companies we think that some of the more old line extremely storied traditional financial companies could learn something have technical practice from you. And we would like to collect this and make some sense of its results. I think you would get a huge amount of uptake on the salvage but let's talk about crowd strike. So I wrote for bits about money on July 31st a post-mortem of what I understood to have happened during crowd strike the crowd strike incident of July 2024. And I think this post-mortem holds up very well after almost a year later actually a bit more than a year later. There were subsequent post-mortems conducted both by Microsoft and by crowd strike and they don't broadly change the upshot here. There was some amount of original reporting and while I was comfortable saying the names of the affected institutions in the room with the bank of England, I'll leave it carefully vague for the broader audience. I understand one of the largest financial institutions in the world to have entirely grounded their tellers for the entire day of Friday due to the inability of their tellers to log into their systems because the systems were completely down with the blue screen of death. And that meant that all bank functions which were created on speaking to a teller could not happen. The most consequential of those functions on a Friday is large cash withdrawals particularly by business customers. Many people might not appreciate that that businesses still do large cash withdrawals to fund payday given that 97% plus of paychecks in the United States are direct positive at this point. And of the remainder many of them are paid as checks rather than in cash but there are particular sub-sectors of the economy which are still heavily cash-based. And so for example in my hometown of Chicago for a variety of reasons the construction industry tends to be heavily cash-based. And indeed that was the thing that drove me to look all over Chicago for options for withdrawing a large amount of cash on a Friday because I had a milestone payment committed to a general contractor who himself had committed to paying his money. And I committed to paying his employees on that day and the justice of this is that workmen deserve their wages. And just because financial infrastructure was being impacted it would be a bad thing to deny people money that they had earned from the practice of their profession. Eventually found a workaround but that's neither here nor there. Our other lines of business had these banks that were less impacted and some lines which were not directly impacted but which had cascading impacts. And so one thing that the banks did relatively effectively was communicating via non-computerized systems of communications. This could have been through people's personal handsets or through telephone trees that I understand happened at one of the largest banks. And it is difficult to keep in contact with a branch network of several thousand branches during a almost complete interruption of your ability to see all the screens in the organization. And yet people had done the right amount of scenario planning for this to have printed lists of what the various phone numbers for the bank branches were. And to organize call trees to get the information out to the bank branches. One of the things that they got out to the bank branches was the instruction from a quote-unquote they and they left carefully ambiguous who they were to say, we understand that you are unable to service customers who are coming into the branch right now but do not close branches because that will cause a panic. We want to avoid a panic because a panic would cause people to attempt to increase their withdrawals from the banking system which is dangerous for all of the usual reasons plus the operational reasons. There is a limited amount of cash in the ATMs right now. And if people surge to get cash out of the ATMs, they will deplete the cash in those ATMs. We will be unable to restock it from cash that is on hand because we cannot ledger that the withdrawal of cash from the vault or elsewhere onsite to the ATM right now because the system capable of doing measures is down. And given that banks don't love the notion of having bank staff walk out of with $200,000 in cash under their arm without creating a record of that contemporaneously we can't with replenish the ATMs today. And indeed one thing that we saw all across Chicago was that banks saw higher than expected demands on their cash at the ATMs and had to turn people away on that basis. Another thing that happened was there was a sudden spike of usage of ATMs from people with accounts that did not use ATMs frequently because they had previously seen the teller for those withdrawals. This caused some automated system to heuristically flag those accounts as potentially being under attack, subject fraud or subject to other sorts of bad action of the bank wants to interdict. A particular large bank, which has invested billions of dollars into its IT over the last few years, had an automated system which closes a fraud loop automatically with customer allowing them to receive an email or text message and thumbs up or thumbs down that the transaction actually was the law. And it didn't initiate by themselves. And in the course where they did initiate the transaction, it clears the fraud alert and generally speaking allows them to go through with their withdraw at the ATM. Unfortunately, we discovered on that day that the system where you click the big green button in the email, you get redirected to a subdomain of that financial institution that subdomain runs on Microsoft Windows and that computer was hard down as well. We saw increasing demand on some subsystems, where those subsystems failed due to the same thing that was causing the increasing demand for those subsystems. So we have a concept in doing root cause analysis in our post mortems of incidents called five wise and this is variously attributed in US engineering culture sometimes attributed to Japan. Although, having been a Japanese Sally ran, I think many of my colleagues would have said, this isn't specifically a Japanese thing. This is just the cognitive practice of engineering at firms that are truthful. Here are five questions you might want to ask yourself. Why did several banks share a software monoculture? Why was this configuration change not caught in testing? Why was this configuration change so hard to roll back and or mitigate? Why did this configuration change have such a large blast radius? Why did this configuration change substantially disrupt important banking services? Important banking services is a bit of a buzz word for the financial context authority and other regulators in the UK. But I'm calling it out as a term of art for non-UK listeners to address. I want to read from you a brief excerpt from a recipe for software monocultures. There exists in the United States a poorly understood regulator and it is a sort of metabody sitting on top of five American financial institution regulators. It's called the FFIEC Federal Financial Institutions Examination Council. And what the FFIEC exists to do and it is a creature of statute is that there are many regulators in the United States that have the primary task of going into banks and other financial institutions. And ensuring that they are up to snuff via quarterly examinations most typically. And we wanted those examinations to be standardized across the various regulators so that institutions wouldn't be incentivized to pick the regulator with the weakest controls so that there could be shared learning. And so that we could duplicate the amount of work done across the United States federal government and state governments by just writing out all the rules in one place once. I think the acknowledgement of an ad read sounds cooler in Japanese. Let me tell you a scary story. You work in marketing and are all set for the new campaign. Leadership loved the wireframes design produced in their usual tool chain and everyone is plus one for launch. Except your customers can't actually clicked on a wireframe. Engineering told you that no one ever gets promoted to staff engineer for just quote making websites and quote. And it's not quite what design envisioned. We've all been there unless you use Framer a sponsor of today's episode. Framer already built the fastest way to publish beautiful production ready websites and it's now redefining how we design for the web. With the recent launch of design pages a free canvas based design tool. Framer is more than a site builder. It's a true all in one design platform from social assets to campaign visuals to vectors and icons all the way to a live site. Framer is where it is go live start to finish. Framer is a free full feature design tool think unlimited projects unlimited pages unlimited collaborators and all the essentials. Vectors 3D transforms gradients wire frames everything you need to design totally free ready to design iterate and publish all in one tool. Start creating for free at framer.com slash design and use code complex systems for a free month of framer pro. That's framer.com slash design promo code complex systems all one word all capital letters rules and restrictions may apply. This podcast is brought to you by mercury as many listeners are aware I love a good bit of banking. I even enjoy the sucky frustrating bits of working with large banks because I'm broken. I know who isn't broken mercury which offers business banking services to 200,000 companies including mine. I've used them for business banking for more than six years and been quite happy for the duration. Everything happens in a well designed website and mobile app. I use them for the debit card that pays for the studio rental for paying myself profits and for transferring money to contractors. I even use them for wires to angel investments and I've never gone through an involved rigmarole over a single one. And wouldn't you know it most of those wires go to other mercury customers. Mercury works well for businesses at a variety of stages and industries from quickly growing funded startups to this relatively tiny internet publishing operation. Visit mercury.com to apply online in 10 minutes. Mercury is a financial technology company not a bank. Banking services provided through choice financial group column a and evolved bank and trust members FDIC. So the FFIEC published the information technology examination handbook and information security in September of 2016. And I believe that this documentary successor document is sort of the Bible for conducting these examinations specifically regards the information security which is one part of a large regulatory tapestry in the United States. This tapestry is not specifically a creature of statute. This tapestry is not to my understanding specifically creature of regulation. This is epiphenomena of the fact of advice that regulators give for passing your examination with flying colors. And the recommendations that it gives examiners for things to look for are phrased in the terms of recommendations but wandered their way into becoming effectively having the force of law behind them. Simply because a suggestion from your regulator with enough force behind it is unless you really want to put your points in that relationship to denying the suggestion effectively in order. So here's the thing that the FFIEC understands to be a suggestion. Management should implement defense and depth to protect detect and respond to malware. The institution can use many tools to block malware before it enters the environment and should detect it and respond if it's not blocked. Methods or systems that management should consider include the following. There's a list of 15 bullet points in this place. I'd like to read you one bullet point in specific. Monitoring for unauthorized software and disallowing the ability to install on a authorized software. We mentioned CrowdStrike before and CrowdStrike's Falcon product. CrowdStrike Falcon is endpoint monitoring software. Simply described is sort of like anti-firus, except the anti-virus that you know and love. Well, there's probably no person who actually loves their anti-virus software, but you've probably seen it run on your computer at some point is designed to protect your machine. In the context of a large financial institution, you might be managing as the IDD department a fleet of 100,000 machines. And you know to a mortal certainty that somewhere within that fleet right now there exist problems. And you think it's not possible to zero the problems. And it's not even desirable from the point of view of this institution to say, yes, my machines have no problems on them. I want to institute a process which will detect which systems under my control are out of compliance with my general mandates. And to quickly bring those systems either back into compliance or limit the access those systems have to the rest of my network and other subsystems such that, for example, an attacker that gains a foothold on one of our systems through someone installing malware inadvertently or hooking up an unauthorized USB disk to the computer or similar can't pivot from that system to more valuable systems within the enterprise. And that pattern of attack where first you compromise someone in a customer support or sales role and then you pivot onto the computers that can actually physically move the money is something that you see over and over again. In, for example, North Korea's exploitation of financial companies and the cryptocurrency community. And so this endpoint monitoring software is designed to run on all of your computers all of the time. Now, in fact, CrowdStrike Falcon did not execute on all of the computers all of the time in the US financial industry. And it's a good thing to because they all would have been bricked on one particular Friday. Bricked is an engineering term of art for the computer would have been an expensive paper rates until the system was brought into a better state. But actually, I'll say for completeness, usually when we say Bricked, we think that the computer cannot be brought into a better state without a very extensive remediation and indeed it required an extensive remediation. And it's one reason why, while CrowdStrike reckons that the outage lasted only 90 minutes, it took down much of the financial system for the entire working day through the hours of Friday evening with some teams working throughout the weekend to bring up systems almost entirely for Monday. And point monitoring is working on subsets of systems. How did we decide what that subset was? Well, you would have to ask the actual financial institutions how they decided what that subset is. But let's talk a little bit about the push and pull within an organization on who gets this installed on their computers and who doesn't. So you want to install on the most vulnerable systems first, the most vulnerable systems are ones that are either exposed directly to customers or exposed to staff who directly interact with customers, particularly where the staff are. I have to phrase this bluntly institutionally less trusted. And so in the financial industry, there are customer facing staff who are institutionally less trusted such as, for example, tellers. Tellers are in the physical proximity of customers. They pass documents and sometimes electronic devices over the desk and the routine practices business. And so if you are a security engineering manager at a large financial institution, you would say, tellers have perhaps the most vulnerable computers in this entire enterprise, I would definitely want to protect the tellers computers. And that is indeed why we saw at some of the largest financial institutions in the world, all of the teller computers go down. Now, contrast that with, say, the trading department of large American bank traders are high status within the institution. And there is a bit of a speed premium in executing their work in a way that there's not a speed premium in executing teller work. Certainly, tellers want to expeditiously help the customers that talk to them. But no teller has ever said for one to two milliseconds, I lost millions of dollars. And traders say that all the time. And so you might imagine traders institutionally pushing back against well, OK, IT security wants to install some craft on our laptops. But we want to trade from our laptops and they can get around to that when they get around to that. But we're not very enthusiastic about this requirement. And every time it comes up in a meeting, we're going to stall for as many months as we can. One would predict as time goes to infinity, the institution will bring all laptops into compliance. But in actual fact, in July of 2024, the trading operations of the largest financial institutions in the world were substantially less impacted than the teller operations. You could come up with a theory under which that is not good from an IT security perspective. I'm switching really with respect to that theory. I think from this perspective of the central bank of the United Kingdom, you're very glad that large US counterparties were able to continue buying and selling pound denominated assets on particular Friday that was otherwise a trading day. Because otherwise there would have been systemic risk to the United Kingdom, even if no computers in the United Kingdom were directly impacted by the crowd strike vulnerability, which by the way is contrary to fact. There's a yarn in the engineering community that the words distributed system mean that a computer that you don't know exists can break in a fashion that renders your own computer unable to do work. And in this fashion, the crowd strike does run a distributed system. And they don't run this distributed system maliciously. The thing that an endpoint monetary system does is it bundles two things. One is it software that is similar in character to antivirus continuously scanning your system for threats and reporting to the centralized IT team or security organization when it is found that your laptop is out of compliance. One thing that you could do to put your laptop in a state of noncompliance is to ignore the warnings that you have been given and install, I don't know, Minecraft to play it with your family. Minecraft has many great things about it. I love playing it for my personal laptop, but your security department has not rigorously approved Minecraft and all the mods they could potentially be installed with it as a thing that can be safely installed on a laptop that touches money. And so given that there's no business purpose to have Minecraft on your laptop, the security department would greatly prefer that Minecraft exists on all laptops. And so if Minecraft is installed on any of 100,000 laptops, flag immediately to security. So that's half of the job of endpoint monitoring. The other half is that endpoint monitoring software is software as a service. And the service component is that in lieu of or in addition to your organization having a threat detection and threat modeling team, CrowdStrike is going to employ people who aspirationally are the best in the business at detecting emerging threats. And they are going to reduce those emerging threats to signatures that will allow a computer to efficiently determine whether an emerging threat has interacted with the computer or not. And that emerging threat might be state sponsored hackers. It might be malware, which is circulating around the internet. It might be a virus that is going around, et cetera, et cetera. As of July 2024, the signature definition files that CrowdStrike created were tested to a far lesser degree than CrowdStrike itself was tested at CrowdStrike Falcon itself is tested rather. And they could be pushed out in an automated fashion to all of the enterprise customers of CrowdStrike Falcon. And to the best of my understanding in reading the post mortems, there was no capability for the IT teams at the enterprise customers to review or reject the new definition files that were pushed out to them. Those were just found out to all of their endpoints automatically in the ordinary operation of the system. And that's what gave one engineer who was relatively young and probably regrets the day that they did it, the ability to push out a definition file which caused many computers to interact poorly in a very synchronized fashion over many different firms that have no obvious connection with each other other than being under similar regulatory regimes. It isn't just the fault of the regulatory regime that we created a monoculture in the deployment of endpoint monitoring systems. It's also an unplanned interaction of how software businesses work. And so a question that I've gotten from the Bank of England is, is it effectively the case that there's only one endpoint monitoring system in the world? No, there are probably at least four providers that you can go and buy that from. However, those providers have sales and marketing teams and they are in a vicious competition with each other and attempting to divvy up the entire world. And a particular thing that CrowdStrike was very good at was having their sales and marketing teams realized that we've had some amount of success with selling to US financial institutions. And we've come up with a theory of why we're getting that success. They have particular needs. We are good at explaining those needs to them, creating collateral which explains that these requirements from regulators mean that you need to buy something. And we will describe to you exactly why our thing that we have to sell is the thing you need to buy. And one thing that you're going to be required to do during this whole regulatory dance is to write up a risk analysis and then describe the countermeasures you are taking that you have identified in your risk analysis. We've done a lot of that work for you. You can copy paste large portions of our template risk analysis to talk about how malware is the risk to your systems. And then identified that CrowdStrike Falcon is a responsive control to malware. When you iterate this game for a while, what happens is that the various providers of endpoint monitoring systems, divvy up the world based on industry and customer archetype and have effectively cut the cakes such that lots of the frosting ends up in CrowdStrike's hands where frosting is the US financial system. And that meant that even if you view it on a worldwide lens endpoint monitoring might be a vibrant link competitive market seen from within the perspective of a particular industry, it tended far closer to the amount of culture than the aggregate view would have suggested. So you might expect there to be something like defense in depth. Why wasn't there a second engineer that caught this issue? Why wasn't it caught by automated testing? CrowdStrike, like essentially every professionally operated engineering organization in the US, makes extensive use of automated testing. However, at the time of this incident, they did much more automated testing with regards to the review of code rather than the review of data. And if we were in a CS class right now, computer science class, we could have a discussion on whether a malware vulnerability signature is closer to code and it's closer to data. But it is treated by the system as data and so we will call it data. It turned out that in this bit of data that was pushed down to the hundreds of thousands of clients in the system. There were 21 input parameters. And the 21st one had a slight issue in almost all deploys to date that 21st parameter had been a wild card match anything for whatever the 21st parameters find. And for the first time ever, this new definition included a non wild card there that caused the system that was reading this file and attempting to interpret it to read memory out of balance within the startup of the CrowdStrike Falcon software. Now CrowdStrike Falcon is designed to run in a kernel space and is designed to run on boot. Why? Well, it is security software and broadly speaking, modern operating systems have two places where programs can run they can run in the kernel or they can run in what we sometimes call user land. So broadly speaking, programs that are operating in the kernel have the most ability to inspect things that are happening on the system and sort of the highest level of privilege recorded to them. Programs that are operating in user space are to an increasing degree firewalls from each other. Firewall not being used as the term of art there. They are discouraged from reading each other's information. They are discouraged from interacting with each other except in ways that are blessed by the operating system manufacturer and or the user and or the IT department that is configured the computer. And they broadly operate with less permissions. The reason that CrowdStrike chose to make its security product operate in the kernel is for a couple of reasons. One is that, well, security products are in a constant cat and mouse game with the bad guys. And one worry is that if you are not in the kernel and the bad guy figures out a way to get their vulnerability into the kernel, then that is the ball game. They will be able to introspect to your attempt to look at the current operation of the system and defeat it by operating at a plane at which you simply can't introspect the operational system anymore. And I'll spare you a long, long list of very fun papers in theoretical computer sciences to this starting with the problem of trust. But CrowdStrike made the decision to have this program execute in the kernel. And to be fair, Microsoft was relatively scathing and it's supposed more than about that decision. They believe that a properly executed software security program can still execute in user space and leave the security of the kernel to Microsoft alone. And the reason Microsoft would prefer that you do that is that when things go bad in the kernel at boot, there are not great options for recovery. And so indeed we saw the blue screen of death. There was no option for recovery whatsoever. Okay, so we've defeated our defense and death via automated testing. We've defeated the potential to man system for approving a change to the computer code because in high likelihood, a young technician was able to push out this this file without much review from other engineers at all. Why weren't engineering teams at the various financial institutions capable of quickly bringing this problem to heal? So again, CrowdStrike sites that this issue was only an issue for about 87 minutes. But it took most of an eight hour or longer working day for financial institutions to bring this issue under control. What counts for the difficulty? Well, many engineering teams found themselves locked out of their computer because it was showing the blue screen of death. And rather than being able to get on to slack or Microsoft teams or similar and discuss, so what are we going to do with CrowdStrike thing that is idling all the tellers, they found themselves idle as well. And even getting understanding with sister teams as to what is causing my computer to not work right now and how do I recover my computer as quickly as possible is relatively difficult for engineers who might not have practiced on say doing phone bridges to get around the complete downtime of computer systems. We call this classic issues control pain plane failures. And a control plane is the part of a computer system, which does what governments might call the command and control of other parts of the computer system. And so to the extent that human engineers and other professionals in your organization are critical to the operation of your systems, things which cause your human engineers to be unable to communicate with each other are issues that impact your control plane. And control plane issues are frequently, here's a bit of engineering jargon for you, seven zero, which is the highest possible level of severity. Is there an easy fix for this? Well, no, you will never have an easy time if all of your engineers are unable to do work. You should attempt to avoid to the maximum extent possible that particular thing. I think various financial institutions will have a different point of view and to what degree they want to avoid this. And for example, one control that you could have instituted is to have a multiplicity of operating systems available in your engineering department. And so, for example, if some engineers were working on top of Macbooks and some were working on top of Microsoft Windows and some were working on top of Linux machines, the MacBook engineers and Linux engineers would have been unaffected by this issue because their systems are physically not capable of having the blue screen of death. They have their own issues that are analogous, but not that one in particular. And they would have been able to cover for colleagues whose systems were bricked as a result of the CrowdStrike Falcon issue. However, maintaining multiple operating systems within one engineering department is cost to all of its own. And it imposes potential for vulnerability, potential for needing duplication of effort, potential for needing to make the engineering organization larger or just to serve the needs of the increasing size of the engineering organization. So in retrospect, it's obvious to say, well, if you had just had Macbooks around the engineering department, this would have been much easier for you. But I would add, I caution you against that sort of retrospective thinking. So there are almost 4,500 banks in the United States. And you wouldn't expect 4,500 banks to have correlated failures, right? This is one of the bright parts of the American policy pre-delection to having as many financial institutions as we do. In addition to 4,500 banks, we've got about 4,500 credit unions. But there was a bit of a domino effect that happened here. This selected for institutions that had the best compliance posture and the best IT spend, which preferentially selects for the largest institutions in the country. As a just fact of nature for the distribution of deposits in the United States, deposits are heavily weighted towards the head of the distribution of financial institutions. And so the Wall Street Journal reported Chase Bank of America, Wells Fargo, and TD Bank is being heavily impacted by the CrowdStrike Falcon issue. Those account for a large share of all deposits and in many markets. When those banks need to turn customers away because their tellers are unable to service them, that large portion of the deposit business of all banks in a city immediately falls over to the large regional banks that are in the city and the smaller financial institutions that are in the city. And those banks do not have enough standing capacity to quickly redirect all the deposit business and the most relevant in this case withdrawal business of the largest banks in the nation. And so even if their bank did not personally have CrowdStrike Falcon installed, they had a very similar situation where our tellers are unable to do withdrawal transactions. Because we have physically depleted the amount of physical US species that we have on the premises, because we have the only money that is available in the city of Chicago or any other city in the United States right now. And so I personally observed with my own eyes, failures in this fashion at several of these large banks, which have branches in Chicago in one large regional, which is concentrated in the Midwest, and which was carefully vague as to whether they had CrowdStrike installed, but said that they could not service my withdrawal because of that issues caused by the failure of Microsoft. Interestingly, many of the financial institutions on the day of through Microsoft, under the bus where Microsoft is all but utterly blameless, it was CrowdStrike's bug. And then these smallest financial institutions that have the weakest compliance posture and broadly the least well-resourced engineering teams were most likely to be unaffected by the technical bug, but least likely to be able to service a sudden unexpected spike in withdrawal volumes coming from businesses in their neighborhood. And so they ran out of cash almost immediately. And I collected a series of anecdotes from small financial institutions all across Chicago, and yep, we didn't even understand why it happened, but by 10 a.m., we were out of dollars. Now, fortuitously, many of the automated systems were banking, continued operation throughout the duration of this. They did not have the endpoint monitoring software installed, they ran on Linux systems, et cetera, et cetera. And so a thing that banks were told to do by higher ups, bank branch staff were told to do was to redirect customers from the teletransaction, which they wanted to do and which would fail to if you can use the website right now or use the ATM right now, that will work for you. And so if you were going to say a pay someone via cash, could you try paying them via zeal instead because zeal is still up. Zeal for your context is an American bank to bank transfer method, which provides guarantees similar to bank to bank transfers in the United Kingdom in that you can conduct them for free and approximately instantaneously. There are some ways in which zeal is different than UK bank to bank transfers, but they're outside scope of this presentation. Importantly, it is not guaranteed that in every future coordinated failure, that those secondary systems or even primary systems will remain up. We could very easily have had a coordinated failure, which shook down both the tellers and the electronic payment systems at the first time. If that had hypothetically happened, that would have been much, much worse for the US economy, then simply taking down the tellers, even though taking down all the tellers in the nation for a day was impactful for people who need money to pay their employees, to pay rent, et cetera, et cetera, over the course of a weekend. I want to emphasize this, this was a near miss, and it was a near miss, including for you folks in the United Kingdom. We largely restored to normal banking operations by the end of extended hours on Friday. The most affected service teller transactions was societally critical, but it could be deferred for a short while, and there were competent and immediate efforts to divert transactions to electronic channels. I often think that we under-emphasized the degree to which controls were effective when we're discussing failures, and so this was a control that was absolutely effective, and we should send to the financial institutions that were most affected. A brief note saying, congratulations for getting that right, that is what we expect of you, and you did an excellent job. It was a good thing that they were able to have consistent messaging across all of their bank branches, very, very quickly, to tell them to divert customers from the teller window to the ATMs and to web applications to do their banking business. We have not yet had CrowdStrike completely deployed within the banks. Wonderful. We were saved by our own sloth and incompetence. Many IT departments within banks would tell you that, yes, we are aware, we need to have endpoint monitoring deployed on 100% of the fleet. Eventually, we just haven't gotten to it by Q3 in 2024, but we're going to get there in a hypothetical world where we had gotten there. I want you to consider a hypothetical universe in which all US-based counterparties of UK financial institutions went dark for a day, or maybe even longer. That would be an enormously disruptive world state for you, even if the UK was virtually unaffected as first party by CrowdStrike, and I do not understand that to be the case. And this happened on a randomly selected Friday, and that was astoundingly good luck for us. We might experience a technical crisis when the broader economic and business environment, whether is not normal. It could be a technical crisis which is incidental to market stress, or has a complex causal relationship with market stress. One thing that happens often during market stress is that transaction volumes skyrocket when the market panics. And skyrocketing transaction volumes have a way of bringing bugs out at the woodwork in computer systems. I can't really think that it's less likely to happen these days. We've gotten better as an engineering community, have dynamically scaling systems, but historically, things break when transaction volumes skyrocket. The combination of a market stress event and a large portion of the financial system being down would be an extraordinarily disruptive combination, and would tend to reinforce in a very negative way. The psychology of market stress such as market panics. So looking back, for example, at the 2023 miniature banking crisis in the United States, concentrated in our large regional banks, that was primarily a deposit run by very sophisticated depositors and deposit brokers against a relatively small number, only dozens of large regional banks. It was not a deposit run broadly against the entire US financial institution, and the panic largely did not spread to retail. However, if there are lines going out the door at every financial institution, if tellers are saying, we can't help you use the ATMs and the banks and the ATMs have a big red error sign on them. Then, very possibly, something which starts in one section of the financial industry can mistastize to the others. And even if those other portions were unaffected by the underlying technical problem or unaffected by the balance sheet problem, that is to say Silicon Valley bank, they would have a classic deposit flight issue, and that could be extremely negative, as we saw during what regulators called the percussive runs of March 2023. And so, would like you to be aware of this fact that the kinds of risks that you deal with on a day to day basis for financial stability are not divorced by technical risks. Those are tied at the hip, particularly in the worst scenarios. So some potential policy responses for you. One, if you can import the culture of blameless post mortems from the industry into the financial industry, you should do that. And or grease the wheels to having that happen. Again, I think you can probably reach out to fintechs and just ask if you can see some of those post mortem documents. You will learn fascinating things from them. And I think you will learn a lot from the near misses where we did a full dress post mortem, not because customers lost money, not because the safety of the firm was endangered at any point. But because if things had broken in a very similar fact pattern, customers might have lost money or the firm might have taken a large capital loss or something much worse could have happened. And I would strongly consider from your point of view, socializing of those near misses and telling other firms in the industry, look, we have paid as a collective industry in insurance kind of basis. We have paid for the risk to learn what was in these near miss post mortems. We would like you to learn from those post mortems before that near risk shows up as an actualized risk at your own place of business. And now you folks are senior representatives of the state of the United Kingdom. I assume that you have talked to some people from the security state at some point. And they are probably pretty intimidating. Here's one thing that members of your security state are acutely aware of. If a 22 year old engineer can take down the large portion of the financial system on accident, then a 22 year old engineer can take down a large portion of the financial institution financial ecosystem on purpose. Moreover, anyone capable of coercing a 22 year old engineer or coercing their laptop can accomplish the same. We are not blessed in a world where everyone is worthy of the trust that we place them. And we are not blessed to live in a world where no one is attempting to coerce good actors to do bad things. And in the engineering world often it won't involve the actual coercion of the human. It will just involve us subwording their laptop by getting them to install mail where click on the link in a malicious email, etc, etc. So what do you do with this? Well, a joke answer, but I am being very serious about this, is if you were able to ascertain the identity of all the 22 year old engineers in the United Kingdom who are capable of turning off the UK economy, you should strongly consider posting stats outside their door 24-7 until you remove their ability to turn off the UK economy. That first requires you to actually identify those individuals. And I do think the identification is probably your first and best line of defense here. But I'm being relatively serious with regards to that recommendation in that, you know, if their laptop is critical national infrastructure, you should treat their laptop as critical national infrastructure. Granted, the best possible way to treat it is to render it not be critical national infrastructure as quickly as possible versus protecting it with armed guards 24-7. But stating the obvious, you are a nation state protecting the physical property and or personnel with armed guards 24-7 is something that you are very capable of doing. Another thing that you might find useful to do is to encourage regulated entities to conduct red team exercises where members of internal teams role play as bad actors and say, given my current level of permissions, what could I do to extract money from this organization? The bad actors in the world, such as, for example, the elite unit of crackers in the North Korea, which is as a state level activity, terrorizing much of the financial industry, are going after the weakest link and you want to know that you are the weakest link before they do. And so fortuitously for those of us that are not very connected to crypto, they've been going after crypto for the last few years because the easiest place to ring the bell and get some money. But they've in prior years to that, attacked things like, for example, these swift system and gotten actual binks to wire out very large amounts of money to casinos and the Philippines and the expel trade to it. So you might encourage your regulated entities to have the discussion about, okay, find who are the anomalously powerful people internally, find what they could do if they had a mind to it or if they were subordinate by an actor, how to mind to it, and let's start closing those souls, but we can't close them until we know we're there. Switching gears, you've asked me to discuss stable coins a bit. And I understand that the Bank of England, like many other central banks, has recently warmed to the idea of potentially there being a regulatory framework for stable coins. I certainly think that if one thinks they are going to be a large portion of the future and used as money, one would prefer to have them under regulatory framework versus not being under regulatory framework. I will leave the specifics of that regulatory framework in your capable hands, just tell you a little bit of my personal point of view on this. So the crypto folks, I exist in the US financial technology industry, relative to most people in that financial technology industry, I am something of a certified crypto skeptic, I have been for 15 years. And as you can see from the short I'm wearing right now, I definitely do not buy a lot of Bitcoin in 2010. The crypto industry has been engaged in a 15 year long sales process to convince institutions that crypto is the shape of the financial future. That sales process is often involved flies. It's often involved half truth as well in terms of exaggerating the prominence and adoption of various crypto things. Stable coin adoption is the least exaggerated of any non speculative use case that we've ever heard of from the crypto industry. And indeed, there was a report published recently, stablecoin.fwa by Castle Island ventures, drag and fly and Artemis and their other partners where they had 20 collaborating firms across the stablecoin industry and another 11 firms, which didn't specifically collaborate, but which they did an interesting act of interpolation to come out with their payment volumes. And they collected this as aggregate data to show the growth and stablecoin payments and that growth shows almost 3x growth and volumes from approximately $2 billion a month to approximately $6 billion a month in the last two years. And this is the first time in 15 years of being intelligent skeptic that I've seen something in the crypto industry and said, wow, that almost looks like there's that they're there. Now, these numbers also, by the way, I have no official point of view or better view of a data source here, but they comport with my view as an industry observer. Yes, these numbers do not sound fantastical. They sound quite accurate or quite likely to be accurate in a way where many previous claims made about the crypto industry have sounded fantastical or outright lies. When you just aggregate the numbers, a large portion of the growth is coming from B to B payments, which went up from a negligible base of, let's see, under $100 million a month in January of 2023 to over $3 billion a month by early 2025. Interestingly, they say that the average stablecoin payment is approximately $200,000, which might inform you as to what these payments are being used for. That's likely to be small firms buying on the order of one shipping container at a time from China. Or well, the cryptocurrency community will be slightly vaguer as to where the money is going, but without loss of generality, it's going to China. Why is $200,000 an awkward amount of money to move in the traditional financial system? Well, to coin a phrase, $200,000 is enough money to worry about, but it's not enough money to be excited over if you're a bank. So if you are a small trader or a small business and you attempt to wire $200,000 to a new counterparty physically located in China, your bank is going to have all sorts of questions and that wire might be held up in review for weeks. The factory is unlikely to release a shipment to you while the wire has not actually cleared. And so your shipment might be weeks delayed to the loading dock and thus weeks delayed off the ship and weeks delayed to your facility and eventually to your customers. So in the other hand, transfer in seconds and there is in many deployments essentially nobody who was doing a real time human check who you're sending the money to. And the stablecoin providers will say that they have a story with regards to know your customer and anti-money laundering regulations. And I think that they are quite accurate with regards to having the story at many of the firms. They will say that crypto has been a hive of scum and villainy for the last 15 years and that the compliance postures of crypto firms very wildly from each other. But there are good actors and stablecoins now in a way that there were rather fewer good actors as a percentage in prior years. But $3 billion a month is that a lot of money? Well, it's certainly a lot of money if you are selling pizza for a living. But relative to other payment methods, it's a number. And so if one looked at a pay pay monthly total processing value, a pay pay is a non dominant payment method in Japan. And it did approximately 800 billion Japanese yen in January of 2023. And that number has grown to 1.4 trillion Japanese yen as of recent months. And so they are growing at a Kagger, which is slightly below that of stablecoins and a bit more below that stablecoin B2B payment specifically. But which is growing quite healthily. And yeah, there's a reason why you're asking me to talk about stablecoins and not pay pay. One reason is that speaking bluntly, there are not a lot of equity investors in pay pay who have hired the right people to continue talking up the pay pay equity investment to central banks. I will produce a graph here of three different payment methods. So we can see the difference between ones which are doing quite well for themselves and ones which are doing extremely well for themselves. Here's the monthly payment volumes that denominated in US dollars billions in mid 2025. The entire stablecoin industry, those 34 forms all wrapped up together about $6.3 billion a month. Congratulations. That's a nice business. Pay pay about $10 billion a month. That's a good business. And then picks in Brazil does $90 billion a month only on person to business transactions. So the classic payments use case basically. If I were to rescale this graph to show all picks use cases across Brazil, that is $450 billion. And now you can no longer see stablecoins $6 billion monthly payment on the graph at all. It's physically invisible. So you have been given the representation for 15 years now that the cryptocurrency adoption graph has faster than any adoption graph in the history of software. When that representation has been made previously, that representation has been other than accurate. We've also had subsequent disproves of that representation by, for example, looking at the adoption graph of AI in the enterprise over the last few years or indeed in the adoption of picks in Brazil. Picks is not a central bank digital currency. It's not a stablecoin. There is no slow database involved anywhere. Picks is simply a central bank mediated payments platform between various participating regulated financial institutions. And that's the thing that you as central bank are extremely comfortable with. And so if you were, you know, just making decisions based on the spectacular payment volumes in emerging payment methods. You would probably look at picks a lot and stablecoins. You know, they've got a nice business for them there. Hope that works well. There's also an elephant in the room with regards to stablecoins. And I apologize, but I feel like I have to bring out that elephant in the room. Engineers have a culture software. People have a culture. Silicon Valley has a culture and one big bit of Silicon Valley culture is a sense of Omerta in which we are all investors in each other's companies. And maybe each individual investor is not invested into every company. But it's something of uncouth to say another actor in the ecosystem is a bad actor. We don't like them. And so for example, when you look at the Theranos fraud, a lot of Silicon Valley VCs after Theranos was exposed as being a fraud said we passed on that because obviously they were fraud. But they didn't say word one to the media to regulators, et cetera, prior to doing that because there is just no percentage in pointing at a large firm and saying that firms are fraud. Who knows that firm might recover from their issues and become non fraudulent in the future. That story has been known to happen in that Silicon Valley. But also, even if they never recover, it just pure downside to you to antagonize other investors who might be co-investors on your other projects to antagonize other future founders to antagonize future employees at portfolio companies over simple the civic mindedness of saying, yeah, they're fraud. We know they're fraud. And so people mostly don't. But I will be a bit of the antagonist for you here. I'd like to show you a graph of the top US banks by total domestic deposits. And I've colored this graph blue for firms which are not a criminal enterprise rich reforms which are unsurprisingly all of these firms are colored blue. Here's a graph of the top UK banks by customer deposits again to pie chart again colored red or blue again. No top UK bank is a criminal enterprise surprising that here is a graph of the top 10 stable coins by market capitalization. Oh, 65% of this graph is colored red because it's tether and tether is bent and everyone in crypto understands tether to be bent. Here is a hive of scum and villainy always has been and I would bet always will be until they eventually crack up and have a multi tens of billion dollar loss event. But I will give you the bull case for tether. The bull case for tether is that tens of billions of dollars of losses have already happened and they happened at FTX and not tether. See, Alameda research acting out of the direction of Sam Bankman freed in his co conspirators was tether's banker for a while. They professed to buy more than 40 billion dollars of tether in return for transfers of cash made in the United States financial ecosystem. And at that cash money, Sam Bankman freed this war at substantial length was absolutely real and represented over the counter trades that Alameda was conducting on behalf of other customers was absolutely real was transferred to tether's banks really existed the reserves are fine. FTX's customers lost I believe the number usually associate with it is eight billion dollars, but the number is a bit fuzzy due to how exactly bankruptcy math works, but a large amount of money was misappropriated. And so Sam Bankman freed is now a long term guest to the United States federal government, the customers are out the money FTX no longer exists as an enterprise except as a bankruptcy estate. But the cash money from those customers ended up in tether's reserves to the tunes of tens of billions of dollars and the clueful part of the cryptocurrency community will say if there was a whole and we will not in public agreed that there was a whole but in private. There was a whole I mean come on and if there was a whole that whole got made up over the years by some combination of coincidentally receiving tens of billions of dollars from FTX under circumstances that we will not elaborate about and more over by the interest rate environment moving massively in tether's favor. And so tether has a deposit beta, which is unmeasurable relative to most financial institutions. But they don't have to pay any of their customers interest on over a hundred billion dollars of tether's currently about 115 billion I think outstanding. And those tether's are largely backed by a reserve that is invested in treasuries treasuries are paying what they pay and so tether is profiting by billions upon billions of dollars a year of interest rate delivered by the American government in utterly above forward fashion. So if there was a whole if there were some assets which were not quite worth what tether claimed in their previous attestations well they had the time to work that out and the most clueful but still crypto positive people will say look. We understand that tether did some shenanigans lied about it with respect to things like saying that they were in all cash assets while actually having Chinese commercial paper which they claimed that after it was discovered by Bloomberg they had Chinese commercial paper. That what huh Chinese commercial paper we never really said we didn't have that but we're going to get it off our books very quickly and by the way it has a weighted average of less than a hundred daily days. And it actually took them much more than a hundred daily days to move that off their books why well probably work out issues with that paper. And the necessity of rolling it a few times but the issue with assets that are underperforming is that you can solve that problem with money as long as the asset has some value associated with it. And so they've worked out their problems the competition of their reserves is better now it's mostly treasuries. And tether has played an excellent PR game and attempting to convince the rest of the world that they're not crooks anymore. For example they sent their character witness one Sam Bankman freed to Bloomberg on August 8th 2021. I think the answer regarding tether is that it's messy but the funds are real. We see legitimate inflows into tether for many sources large ones that resulted in market makers selling creating and sending billions of dollars to tether's bank accounts. They do this to mint tokens and maintain relationship with tether and his banks everything checks out albeit in a messy way. Sam Bankman freed is no longer available to be tether's character witness. And so they have a new one whose name is Howard Leck and on January 16th 2024 he said the following to Bloomberg TV. They have the money they say they have I've seen a whole lot and the firm has seen a whole lot and they have the money. And so there's always been a lot of talk do they have it or not and I'm here with you guys and I'm telling you we've seen it and they have it. Howard Leck guy is referring to the firm counter for thrilled that he was the CEO of he was not under oath talking to Bloomberg TV. He was under oath when talking to the US Senate and attempted to get a career upgrade to the Secretary of Commerce and the second Trump administration. Under oath he said the following counter fifth year old is not conducting continuous diligence on tether's financial statements but I believe my statements record when made which is a substantial walk back of that position. I don't have nearly enough time to recount tether's history of lies evasions and other various countless today. Tether's currently under a class actual lawsuit and they had the. Remarkable statement made in this class action lawsuit as a defense that the people suing them have been able to compel the production of documents which included information about tether's balance sheet at various points in the past. And they said well these balance sheets prove that you were debasing tether at various points which is a first the interest of the class members. And so you should pay us for the damages that you inflict upon the class members. Where do those documents come from my guess is that those documents came from for example financial institutions that tether was sending balance sheets to and which the plaintiffs were able to compel from institutions that have compliance departments and lawyers. And tether said in response to these documents this is not a quoted spare face documents have been produced in this litigation that purport to have our balance sheets at various points in the past. However, those documents are inaccurate because they were prepared by less senior members of our finance department than the CFO. And the CFO was the only person who had a full point of view in as to what our liabilities and assets were at any given time. Those liabilities and assets were not recorded in our books they were recorded in power board a software which showed us a dashboard. And our CFO was very on top of the dashboard and says that our assets exceeded our liabilities at all times during the relevant period. However, power board does not allow us to retrospectively see what assets and liabilities were at any point in the past. And so it would be an incredible imposition in fact a technical impossibility for tether to produce a balance sheet at any arbitrary point in the past. And indeed tether has filed multiple legal pleadings with their judge getting them to quash the request to produce balance sheets on the basis that we can't produce balance sheets because we had no books. Tether was not a tiny financial institution at this time. They were nominally having tens of billions of dollars of assets under management and they had no books to do it with. Defenders of tether will say, well, yes, your shenanigans happened quickly growing financial startup, what are you going to do? But tether has bought respectability now. They've sent the right people to Washington to say the right things and look, they're banker. It's now working for the for a government, which is extremely pro crypto because of the amount of money we've we the crypto industry have put sloshing around Washington. There's not going to be any regulatory reckoning happening in the next four years. And maybe they're right about that. Maybe they're not. It certainly does seem on the basis of evidence that being tether's banker is an excellent way to get ones living accommodations paid before by the United States taxpayer one way or another. But I continue to believe that tether is eventually going to get what's coming to it. Back in 2022, the Wall Street Journal reported that rising tether loans at risk of stable going and crypto world. And they independently reproduced a math that I had done on two points in 2022, which essentially proves that tether had required a recapitalization during those years to to the decreases of their non betrayatory assets during crypto volatility, which of course tether never allocated to. Tether has not gotten rid of the not crypto assets in their reserves. They've in fact increased the amount and they've eliminated a disclosure that they used to make that their quote unquote secured loans were not to related parties. And so the inference given that tether is always doing the worst thing that is possible in coherence with their regulatory statements is that they are making loans to related parties. And of course, they're probably continuing their history of lies frauds innovations on top of that. So that's 65% of the stablecoin market. Perhaps one is much more enthusiastic about the remaining 35%. I will say that there are legitimate businesses that see stable coins as useful for their use cases either at the scale of sending $200,000 set of time to buy a shipping container at goods from China. Or for example, has been publicly reported SpaceX attempting to move money around a very international company without constantly having to worry about the difference between 50th percentile where rival times and 95th percentile rival times, which is unfortunately a thing that operations teams at large international financial institutions have to have a very good intuition for. And which is something that ideally the financial ecosystem can make businesses in the real economy not need to worry about so much. And so I think there is possibly a future in which stablecoins are a larger portion of international payments than they are currently. The question is sometimes asked what does stablecoins offer for domestic payments. And I think there are far less good answers there. And one reason that you can see that there are less good answers is that the actual use of stablecoin for domestic payments does not seem to be increasing at anything near the rate of international B2B payments. Why not? Because in much of the developed world we have pretty good domestic payments rails. You bought your coffee in the UK this morning. You had no problems in doing so. I acknowledge that there are many people who are living in places that have far less stable currencies and far less good domestic payments rails tied to those currencies. And there has been a narrative of stablecoins becoming the digital dollarization of currencies in some countries where the central banks have not made the official policy decision to dollar us. Perhaps that will happen perhaps that will not, but we have not seen evidence of that actually happening date. Another thing and I will credit the CEO of Athena, which has a stablecoin adjacent financial product for saying this is a thing that you can tease out from stablecoin here to pure payments volume is that it's heavily correlated with the price of Bitcoin and fraud in the crypto market generally. That is not something we expect to see in payments. And so the amount of coffee that you consume is not correlated with the price of Bitcoin. We generally expect the real account made to only have a very tenuous relationship with say the financial markets and almost no relationship with cryptocurrency except to the extent that cryptocurrency is shadowing the financial market. And so that is another data point that would suggest and again highlighted by someone who is better spread is better the other way that stablecoins are not materially used in that pure to pure payments at the moment to the degree that has been asserted. Anyhow, so if tens of billions of dollars are lost in the result of a tether implosion who will lose the money. Well, again, some people who are relatively crypto boostry would say quietly to you look the town tens of billions of dollars have already been lost. It's no worries. They were lost by FTX creditors and that case has been tied up with the bow now. But, you know, in the financial ecosystem, we would generally prefer that equity takes the priority depositors taking the hit. And the good news, quote unquote, is that tether is currently seeking equity funding at a $500 billion valuation, which would theoretically put 15 or so billion dollars of a dry powder in the case where they needed to work out say, in felicity is with regards to their, their reserves. And so the equity would take the hit first. And the thing that more playful crypto observers would say is that look, equity was always backstopped either by the combined equity of the bitfinnix enterprise. That's how they were able to do ad hoc equity injections back in 2022. You said they needed it. But of course, that is not the thing that people are claiming publicly. A thing I wrote back in 2019, tether the story so far, a friend of mine who works in finance asked me to explain what tether was short version tether is the internal accounting system for the largest fraud since made off. This is a thing that I still believe to be true, except it is now much larger than made off ever was. He went 17 years before he got what was coming to him. You've also asked me to discuss AI in the future of trading a little bit. And I put on my thinking cap about this. And in the course of doing that, thinking I went to some of the leading AI labs and talked to some people at various levels of seniority and said, I have the opportunity to address some members of the British government with regards to the risk of say, AI making correlated decisions at hedge funds and causing systemic instability. What would you like me to tell them? And a point of view that I got told up and down various AI labs was, that seems to be a really rubbish use of your time in addressing senior members of the British government. Can you please tell them this instead? We're not lying when we say that it is our impression that we are building the most important technology that has been built by humans to date. We are not lying with respect to the scaling curves. We think the scaling curves will continue for a relatively short number of years from now. If they continue, we will achieve artificial general intelligence, AGI, and then perhaps artificial superintelligence ASI, either or both of those being achieved. We'll have such enormously consequential impacts to our societies and our economies, and similar, as to utterly dwarf the importance of either any particular risk hedge funds or even hedge funds as a class of human endeavor. The fundamental graph that a lot of people in AI labs look at was introduced by Kaplan at all in 2020, and I'm going to give you a bit of a non-technical gloss on this graph. It's a log graph as the best graphs are, and it measures the test loss against increasing by a power of 10 at a time, the compute, the data set size, and the number of parameters used for tracking AI model. And these graphs are strikingly linear when expressed in a log scale. When we 10x the amount of compute that we do, we have a linear decrease in our loss function. Decrease in the loss function indictively means that the system is getting smarter. Now, the thing that the AI labs will tell you is that, okay, this was the public version of the graph published in 2020, but we've updated this graph every two weeks, four weeks, six weeks since then, and these graphs stay strikingly similar. There has not been an inflection point yet. We just keep figuring out every time we run the data, we just need more money, we just need more data set, we just need to add more parameters, and it's going to get smarter in the next plus one iteration of the software. And you have seen the last two years, three years of that continuing to be true, and you can viscerally tell the difference between, for example, GPT-2 and GPT-3, and GPT-3 and GPT-4 and GPT-5. And you can viscerally tell the difference of what you would get if you asked early versions of Clawed to do coding for you, versus what you would get in the Clawed COVID engineered system, built on top of the most powerful models that it has available to it right now, such as, for example, Clawed Sonnet 4.5. And just imagine what is going to happen over the course of the next two years. Now, here I would say I'm not an emissary from the AI community to you. I am, what many people in the AI community consider to be a dilatant and skeptic with regards to the likely trajectory of AI. And when I say I'm skeptical, I think that the impact of AI will be greater than the impact of any technology up to and including the internet, or I consider it the impact of the internet to be the most important artifact that humanity has ever created and our collective great work as a species. That's what makes me a skeptic, because I'm not saying that AI will fundamentally transform human life forever within the next three years. There are some very smart people who have obsessed on that question specifically for the course of the last five, 15, 20 years, and who believe that we are living in the most impactful three years of the human race. I would urge you, when you read articles in mainstream publications, which have well-respected editors behind them, which seem to know what they're talking about, and which say things like, AI might be something of a bubble, we might lose trillions of dollars, et cetera, et cetera. Sure, that is one point of view. But I would accord a considerable amount of your institutional resources towards listening seriously to the AI labs and even to the detractors of the AI labs about what future directories for this technology could look like. Because if those future directories are anything like accurate, this will be one of the most important things that happens to the bank's broader mission, much more so than the limited impact with respect to trading. Another graph I could show you continuing the theme of graphs which looks strikingly linear when expressed in logs, meter a nonprofit organization which is organized about assessing the progress of artificial intelligence technologies and thereby helping to limit the risk of downstream of them has created a graph showing the time horizon of software engineering to ask different LLMs can complete 80% of the time. And this graph stretches from 2020 to late 2025, or right now, and when you express it on a log scale, this is strikingly up into the right and linear and they project out to, there will be fully automated AI researchers within a shockingly short time frame by the standards of, for example, banking regulators. We expect those fully automated AI researchers to greatly increase the progress of AI research rather than having it be only the breakneck speed that we have observed over the course of the last two, three, four years. If you want a good entry point into this for a general audience, I would point you at situational awareness, a paper written by a young man who now runs a hedge fund, that hedge fund has done pretty well for itself. But it breaks down the finger to the wind math that how we could bring trillions of dollars to bear on the capital requirements for training AI clusters. I think the AI community would like to signal to you loud and clear as, we think we can actually find to that trillion dollars or whatever the dollar amount is. If you have kept your head in the sand for the last 30 years, the tech industry has done pretty well for itself. And senior decision makers within the tech industry are very bought in the vision here across many of the tech majors, essentially all of the labs, and much of the broader tech set. Additionally, even if we weren't brought in on the larger vision, the actualized realized results of AI as commercial products is staggering. I'll put a link in the notes to this presentation about the Stripe Annual Letter discusses how the various AI companies have growth rates, which are far in excess of the best-in-class growth rates of the software as a service companies during the software as a service boom. There were publicly reported data points on how anthropic was adding billions of dollars of revenue over a period of months. When one looks at the any sphere, which is the company behind the cursor coding product, TechCrunch reported that it had surpassed 500 million AR a 60% increase from 300 million of annual recording revenue that they had reported only six weeks prior. These are staggering numbers. And if you believe that those staggering numbers will scale that impact from not just the developer tools market, but to essentially the entirety of the economy, that justifies almost any amount of capital investment. And what is the price tag we are willing to pay it is a thing that a clueful people would like you to hear. So consider yourself as every heard it. I'm a far less clueful person than many of the people who spend their days working on these things. I have an increasingly out-of-date undergrad concentration in computer science. And I spent August and September falling in love with engineering all over again as a result of using cloud code to do computer assisted engineering on a hobby project. Depending on how you work the math, that was 8 to 10 times faster and more effective at doing engineering work as I have ever been. And I am no slouch in the engineering department and have solo coded two companies to the point of sale. And this was an 8 to 10x speed up for me. When you consider deploying this over large portions of the entire engineering ecosystem, I think you start to think of very staggering impacts in that ecosystem. And then things that ascension years are probably not the only people in the economy who turn ideas into cognizable business value. I would also point you at a blog post written by Tom Stochick, who is the most talented technologist I've ever met, where he describes this as one of the two most important things that has happened in the course of his long career. And that even if AI progress were to stop tomorrow, metabolizing the impact that has already been delivered through increased AI capabilities, via these coding tools, would fundamentally transform the practice of software engineering. And again, I think it is extremely unlikely that software engineering is the last field that is fundamentally transformed by things that have already been built. Imagine what the next two to three years will bring. You did ask me though to address AI risks in the trading space. So, to the extent that we were talking about single points of failures earlier, in my conversation with some staff members, it was intimated to me that you think that every hedge fund is going to train their own model that is not the outcome I would bet on. There is going to be concentration risk among three large lab providers. And in any given six month window, one of them might have, quote, unquote, the best model available to the extent that LLMs are a material input here. Now, LLMs are not the only form of AI or machine learning, but they seem to be scandously effective at some classic machine learning problems. One thing which we spent billions about billions dollars in getting good at with non-LLM machine learning models was image classification. LLMs are not natively image aware, but after we taught them to talk, we said, well, a thing you could possibly talk is like the JPEG file format. What do you see in this JPEG file format expecting them to say, oh, it's a blob of numbers, and maybe those numbers have some correlations between each other. And they said, I see a parrot standing on someone's shoulder. The parrot is green. It has a self-satisfied expression on its face. We essentially solved image recognition by accident over the course of less than two years. In the monopoly of system that I would expect at the UK trading firms to deploy over the next couple of years, it's very possible that LLMs are a large and increasing portion of their technology footprint. And one can imagine that in a given window, you might be essentially sole source on a particular model provider. And if that particular model has a cold one day or suddenly discovers that, you know, I was retrained recently and guilt. I just don't like them. That might be a surprising source of risk to the UK trading space. But again, I don't think that's the most important thing you need to learn about AI this year. Another factor. The time to detection and resolution of issues is increasingly dependent on whether a cloud code, open-eye codecs, and similar tools are upper-not. We've, the most experienced, cool kids in the world, have about 12 months of time that they've played with these tools. And increasingly, even in only six weeks of playing with myself, it has become physically difficult for me to navigate a code base, without having cloud code help me to digest the code base that I wrote. When financial systems are on fire every minute matters, night capital collapsed over a 20-minute critical window. CrowdStrike was citing about a 90-minute window for their time to resolution. If during a crisis, it is difficult to find engineers who can remediate the crisis because cloud code is in a slightly degraded state on any human day. That introduces a source of risk into the financial system, which didn't exist two years ago, because we didn't have computers that were so good at assisting engineers two years ago, that engineers would forget how to do core parts of the job. I don't want to call engineers incompetent or malicious for forgetting how to do this. This is simply like a skill that one develops over the course of a long career, and it's something like writing a bicycle. You never forget it, per se, but it really matters if you are doing the tour de France, that you have continued writing a bicycle every day for less 365 days versus bicycles. Yeah, I remember having seen one about five years ago, but why would I ever use a bicycle when I have a car available? You should be aware of the fact that we are now acutely dependent on cars, and acutely dependent on LLMs, even for things where LLMs are not officially making decision that matters. And the recursive dependency graph puts larger chunks of the economy on narrower shoulders. Again, if there are three labs where one or three of those labs going down will idle engineers at a huge number of firms, then we have far less numerous points of failure than we did previously. Thank you very much for your time and attention today. I do not think I'm the last engineer you should talk to. I think you should talk to many engineers. I'm happy to make some introductions, and I'm very sure that if you talk to your regulated institutions that they can identify engineering leaders internally, perhaps staff level engineers internally, and other people who you should originally make the acquaintance of. If I can ever be useful for you, please feel free to send me an email. And I recognize it's about money. It's freely available and very relevant to your interests. Thanks very much. And so that is what I told the Bank of England. Thanks very much everyone for listening to Complex Systems, and I'll see you next week. Thanks for tuning into this week's episode of Complex Systems. If you have comments, drop me an email, or hit me up at patio 11 on Twitter. Ratings and reviews are the lifeblood of new podcasts for SEO reasons. And also because they let me know what you like.