Twitter/X

Paul Buchheit (@paultoo) wrote on 2026-08-07 that “bad AI is inevitable” and…

2026-08-07 · 19:52 UTC ·@paultoo ·1 min read

Brief

Paul Buchheit ( @paultoo ) argued on 2026-08-07 that "bad AI is inevitable" and called for more investment in defensive AI. He proposed a personal "User Agent" AI with attorney-client-like privilege to monitor accounts, detect fraud/hacking, auto-reference-check sources, advise on health/finance, and monitor physical security; he solicited startups and ideas, especially on defending against engineered viruses and gain-of-function risks.

Why it matters

Paul Buchheit (@paultoo) wrote on 2026-08-07 that “bad AI is inevitable” and argued we must invest in defensive AI to counter AI-enabled scammers, criminal super-hackers, deep-fakes, misinformation, and bioengineered viruses.

Key details

  • He proposed a personal AI called "User Agent" that would have attorney-client‑like privilege, monitor communications/accounts for fraud and hacking, perform automatic reference-checking (like community notes), advise on health and finances, and integrate physical security data.
  • He asked for recommendations on startups/teams building defensive AI and explicitly requested ideas for defending against engineered viruses and rising gain-of-function risks.
Source evidence

Bad AI is inevitable
Good AI is up to us

Paul Buchheit (@paultoo)

We need more investment/innovation in defensive AI. It is inevitable that AI will be used against us by scammers, criminals, governments, and other bad actors. As AI improves, so too does the sophistication and danger of these attacks. How many of our existing computer systems would remain secure if attacked by an AI super-hacker? How do we know if we are talking with a real person or a deep-fake? How do we know if our sources of information are accurate or reliable? How do we defend against the next bioengineered virus?

One product I would like to see is something I call "User Agent". It's a personal AI programmed to guard and defend our individual rights and interests. Like an ideal lawyer, it should always be working for and representing our interests, not those of corporations or governments trying to control and manipulate us. Ideally we would have the equivalent of "attorney-client privilege" with this AI, meaning that it keeps secrets and will not report us to the thought-police. It monitors our communications and accounts to guard against fraud and hacking attempts. It helps us to stay well informed by automatically reference checking the information we consume, providing helpful background and context (similar to community notes on X, but for everything). It's a trusted health and financial advisor that can warn us away from scams while surfacing more promising and beneficial options. Ideally it could even monitor our physical environment and integrate information such as crime reports to help maintain physical security.

What are the best startups/teams building defensive AI? I'm especially curious if anyone has good thoughts on how to defend against engineered viruses, since gain-of-function is only going to get easier and more dangerous.

— https://nitter.net/paultoo/status/1750316380220780795#m