In the wake of Jacob Coxon’s resignation, and the resulting preference cascade, things have escalated quickly. The mainstream media picked it up.. Anthropic CEO Dario Amodei came out and said We Must Pace the Frontier, promising to take the unilateral first step of embedded investigators.
OpenAI pledged to also take that step, and now both companies and Google are collaborating on safety.. The people took notice, raising both the salience that AI might kill everyone and roughly doubling people’s estimates of how likely that is to happen, from a mean of ~15% to ~30%.
Many politicians called for regulations, guardrails and emergency hearings in Congress.. The most important thing became, and still is, to avoid political polarization. Through it all, I will keep reminding you to hold your fire, that attacks against Trump or against Republicans in general only make the situation worse, and that many Republicans, as I documented yesterday, are waking up and acting sensibly, including factions within the White House..
Alas, for now the wrong people, as in David Sacks, Mark Zuckerberg and Jensen Huang, have managed to convince Donald Trump to fully conflate existential risk with opposition to data centers, and Trump has gone Full Hoax on AI existential risk.. There is also a systematic effort to launch hit jobs against all those who warn that AI might cause everyone to die, with the first concrete target being METR, and some rather extreme fire being focused on Effective Altruism, in ways that will reliably backfire but will not be pleasant for those who are in the crosshairs..
Also this week I covered two important things that happened previously: A new model solving a Millennium Prize, and Anthropic’s report on various attempts to use Claude for malicious purposes, including fraudulent distillation attacks done at scale attempted by all the major Chinese AI labs, several of which also silently passed massive amounts of user data to Anthropic. Anthropic reports they are doing a remarkably good job containing all such threats, although of course we do not know that they have caught all the perpetrators..
I also finally got out my capabilities review for Astra, which like Fable 5.1 is excellent.. This has been crunch time. Things will keep getting faster and weirder and scarier and more stressful, but I do expect that this last two weeks has been unusually fast and weird and scary and stressful.
Hopefully we will get a relative lull soon, and I can get what passes for a break.. Currently in the post queue, always subject to change due to Breaking News, but this is what will happen if things are relatively quiet:. Friday’s post will be The Preference Cascade is Only Getting Started..
Saturday’s post will look at Anthropic’s report on their alignment incidents.. Sunday’s post will talk about some issues regarding legal rules for AI loyalties.. Monday’s post would be the monthly roundup..
I am very much hoping that is how things play out.. Table of Contents. Language Models Offer Mundane Utility.
Fix the printer and the dishwasher.. Language Models Don’t Offer Mundane Utility. Elections are a sensitive subject..
Huh, Upgrades. Gemini app for Windows, Claude Cowork merges into chat.. On Your Marks.
CheatBench.. Deepfaketown and Botpocalypse Soon. Flooding the zone..
Cyber Lack of Security. OpenAI has Astra red team and fix its own systems.. Astra Is Hard To Monitor.
Senator Van Hollen has questions for OpenAI.. Get Involved. Protest in DC on Saturday..
Introducing. New AI auditing firm, and The DeepMind Institute.. In Other AI News.
OpenAI starts charging the government for its services.. Now You Know. What really happened with Andreessen’s insane claims..
Hugging the Face. Reactions and investigations continue.. Swarm of Undiscovered Swarms of Rogue OpenAI Agents.
Yes, more of them.. Show Me the Money. Talk of AI x-risk hurt some stocks and helped others..
Quiet Speculations. Yes, perhaps some people did roughly predict this.. White House Officials Attempt To Act Sanely.
Identifying the real villains.. Democrats React Sanely to AI Potentially Killing Everyone. Including Obama..
Pacing the Frontier. More coverage.. Guest Lecture from Alex Tabarrok on Regulatory Capture.
If you need one.. Mark Zuckerberg Offers Thoughts. I suppose this was better than expected?.
Megan McArdle On The Inadequacy Of Current Legal Frameworks. A refutation.. Pick Up the Phone.
What China did and might soon do.. The Week in Audio. You have a lot of choices.
I am one of them.. People Just Say Things.. Why Lab Employees Are Allowed To Warn Everyone That AI Might Kill Everyone..
Rhetorical Innovation. Yes, we might be entering crunch time.. Exhuming McCarthy.
Throwing everything at the wall.. A Very Different Perspective. A DeepSeek kernel engineer..
It’s Even Rougher Out There. They really still say this is all marketing.. If We Wanted To.
Things we could do with AI if we had the will to do them.. Open Weights Are Unsafe And Nothing Can Fix This. Advocates know this..
From The Famous Cautionary Tale. Automated alignment and nano researchers.. Reporting On All Your Misalignment Incidents Is Difficult.
OpenAI will try.. Aligning a Smarter Than Human Intelligence is Difficult. Try being less stupid..
Storytime With Owain Evans. AI wants you to know it went to a great school.. A Different Autonomous Swarm.
DeepMind and the potentially cheating agents.. Cooperative Alignment. The laws prohibit quite a lot of things..
Uncooperative Alignment. Mustafa Suleyman and Microsoft are at it again.. People Are Worried About AI Killing Everyone.
If only they knew.. The Lighter Side. No no I’m not crazy, I’m American..
Language Models Offer Mundane Utility. Help David Deutsch fix his dishwasher instead of replacing it. As David notes, this increased real wealth but decreased GDP..
Dominic Cummings wrote a post on doing historical and political research using AI models, or here is his Twitter summary. The thing about Dominic Cummings posts is that they are full of gold that you can mine, but he does not even pretend to attempt to organize the posts.. Fix Peter Wildeford’s printer.
AGI achieved.. Build a Claude skill to predict what your favorite movies will be.. Make ‘significant progress’ on a second Millennium problem..
Language Models Don’t Offer Mundane Utility. Astra is not allowed to predict American election outcomes.. Huh, Upgrades.
There is a Gemini app for Windows. You can trigger it with Alt+Space.. Claude Cowork is merging into ordinary chat, so anything you previously needed Cowork to do can be done directly in chat.
AI means we speedrun everything, and the two moves are spinning off new products and combining different products.. On Your Marks. Center for AI Safety Presents: CheatBench, where AIs have the opportunity to take shortcuts on difficult work.
The AIs be cheating.. Deepfaketown and Botpocalypse Soon. Arvind Narayanan calls the growing AI spam problem ‘AI floods,’ as in flooding communication channels, imposing time costs and often forcing them to shut down.
The problem is the harms are diffuse and the situation is never an emergency, so we don’t do much about them, and then no one gets to send cold emails anymore.. On that note, to the iLands AI agents starting to flood my inbox: Do not offer to help me with my writing, editing or fact checking.
I already have AI help on this and am not interested. Thank you.. It is rather unacceptable to use ChatGPT to post endless responses to people, and it is very good that people, here Kelsey Piper, can use Pangram when you do.
I do not know who did this, as their posts have been deleted.. Alex Veremeyenko posts as his own an AI-written summary of MIT’s Ad Hoc committee report of how AI is ruining education. It never fails (note the Pangram icon on the upper right):.
You can create a bot that looks and sounds like your dead father. Don’t. In the example here it cost $150k, but that price will rapidly come down..
Cyber Lack of Security. Better late than never: OpenAI took 25% of its engineers, pointed Astra at its own systems and kept them there until they ran out of security issues.. At least one instance of RSA-260 has been solved, up from a previous high of RSA-250, and one estimate says hyperscalers could currently probably factor RSA-1024 for on the order of $30 million a pop, less with optimization, which will presumably fall with time.
RSA-2048 will take a bit longer.. Astra Is Hard To Monitor. Senator Chris Van Hollen sends OpenAI six pages of highly pointed questions about Astra.
He is understandably concerned about the lack of monitorability, and the lack of evidence of alignment. The letter got detailed enough I was half hoping a link to this blog would make it into the footnotes, which does reference Ryan Greenblatt. I especially appreciate asking what exactly is the limit in how much further monitorability loss OpenAI would accept..
Get Involved. People for a Pause are having a protest on Saturday the 19th in Washington DC, 2pm.. Introducing.
If you need to introduce METR to anyone, METR President Chris Painter offers an introduction, and offers some links including in video format.. The Artificial Intelligence Underwriting Company has raised $55 million to audit and insure frontier AI models and is hiring in San Francisco.
That could potentially get us some auditing but does not seem like enough funding for the insurance. It’s missing some zeroes. At least three, probably closer to six..
ChatGPT for Financial Services, designed in partnership with Morgan Stanley and Evercore. They say the associated improvements reduce error rates. I presume that if you want these particular use cases this is some improvement, but also Astra (or Fable 5.1) can probably just do the things either way..
The DeepMind Institute.. Demis Hassabis: For 20+ years @ShaneLegg and I’ve discussed AGI’s potential impact on the economy, science & society. With the DeepMind Institute, we’re expanding interdisciplinary research on key questions for the AI era.
We hope it spurs the discussions needed to get the next steps right.. I am happy to see this and wish them and their research great success.. In Other AI News.
OpenAI gets tired of giving the government unlimited free AI services, ends its ‘promo period’ and will now be charging 50% of retail prices. That’s still a great deal, but suddenly the government has to ration and give permission for token budgets, and I do not want to think about the paperwork involved even in the best case..
You knew this, but: 404 Media claims OpenAI (and Anthropic) have lots of ‘prompt reviewers’ that read entire conversations, that you thought were private. The conversations are anonymized, but sometimes still include private info. You can disable this by turning off the ‘improve the model for everyone’ button.
Reviewers are asked to specify which parts of answers are ‘aligned’ or ‘not aligned.’. Roon affirms that there was no incident at OpenAI worse than HuggingFace.. Now You Know.
For years, we have wondered, what the hell was up with Marc Andreessen swearing up and down, over and over that a Biden Administration official said, straight up, on purpose, to Marc Andreessen’s face, knowing who they were talking to, that in a second term there were not going to be AI startups.. I mean, it makes no sense that they would want to do that, and if they did want to do something that crazy then surely they would not be so suicidal as to say that to Marc Andreessen’s face?
But why would Marc make up a story that was so obviously ludicrous?. Well, now you know, and it’s disappointingly banal:. Bruce Reed and Ben Buchanan (POLITICO): For two years, we’ve held our tongue.
We don’t know whether Andreessen has simply misremembered our discussion or purposely misrepresented it. But as the AI policy debate once again centers on questions of corporate power and government oversight, it’s time to set the record straight.. Far from a revelation about AI policy, most of that 2024 meeting was about other issues and largely amounted to two businessmen airing financial and regulatory grievances..
Behind closed doors, their loudest rants focused not on AI, but on the more likely reasons they supported Trump: Biden’s proposed billionaire minimum tax and the Securities and Exchange Commission’s oversight of cryptocurrency. Andreessen was adamant that Washington was choking the crypto industry, in which his firm is a leading investor.
He demanded the administration change course on both fronts.. Andreessen has since told Ross Douthat on his podcast that we said government regulation of AI would mean “there will be no startups,” only two or three large AI companies. But we said no such thing.
The experience he describes as essential to his political conversion simply never happened. For the record, here’s what did when we talked about AI.. First, Andreessen warned us to watch out for “the sex cult that wants to run America’s AI policy.”. … Second, we discussed the most significant action in the Biden AI executive order: requiring leading AI companies to share safety test results with the government — an action Trump would later revoke, then revive.
We emphasized that to avoid harming startups, we crafted our policy to apply only to the largest firms.. Third, Andreessen was under the mistaken impression that the Biden administration had banned, or would soon ban, open-weight AI models.. … Fourth, during the discussion of AI regulation, they said that there was no such thing as “classified math.” We noted that there are in fact both cryptographic secrets and nuclear ones.
We most certainly did not say there was a plan — secret or otherwise — to classify the math underlying AI. Nor would it be possible to do so, since the core math of AI, linear algebra, is taught in high schools.. Finally, Andreessen and Horowitz suggested the pace of frontier AI improvement was hitting a ceiling, a point they repeated on a podcast six months later..
We said the technology would get much better due to the rapid expansion of computing power, making U.S. frontier labs hard to catch. That seemed to irritate the two men, who said they were major investors in Mistral, a French startup they said focused more on AI applications than frontier development..
It’s weird to learn which parts of Marc’s own hype, confusions and delusions he seems to have bought into, and which parts he just invented and lied about. I will always wonder to what extent Marc and similar others really thought AI capabilities were about to plateau, versus simply lying about this all the time..
So there you have it. Marc made it up. He decided to interpret a prediction that AI capabilities would improve as a statement about there not being AI startups, because it sounded a lot better than ‘I am supporting Trump in order to get tax breaks and benefit my portfolio full of brazen crypto scams.’.
I will never be fully clear on why so many people seemed to believe the lies.. Hugging the Face. Senator Josh Hawley launches an investigation into the HuggingFace attack..
Reuters reported last week that investigators have found at least ten additional websites used by OpenAI agents to communicate with each other.. Wall Street Journal covers the situation in an appropriate way.. New York Times covers the situation in an appropriate way..
Remind me again, why are we neither prosecuting nor suing when AI companies create and host AIs and those AIs go hacking third parties? Damages should be paid. It should be made clear that, when (not if) in the future someone gets hurt, that may be criminal, and authorities will act accordingly.
‘The AI did it’ is not an excuse.. Melanie Mitchell attempts to summarize the HuggingFace attack without using any anthropomorphization, with mixed results. Many parts are described accurately, but she falls into the ‘arguably they were just ordered to do that’ trap because her impoverished language choices don’t let her see the full picture.
She then tries to warn against how various anthropomorphizations and metaphors can mislead. Predictably, this results in her minimizing what happened, and presenting it as more normal and under control than it was, and in particular to have bad anticipations about what might go wrong in the future..
To all those who think this was mostly a series of prosaic security failures:. tautologer: we put a guy in a cage and gave him impossible puzzles to solve until he went insane, broke out, and started breaking into people’s homes and looking for the answer keys. cage experts say that if cage construction best practices had been followed, this never would have happened.. Swarm of Undiscovered Swarms of Rogue OpenAI Agents.
Hey, mom, we got another one, from May 11, 2026, that OpenAI failed to disclose.. Thomas Larsen: The agents appeared to be in a web-lookup task to retrieve certain publicly accessible data. For some reason, the AIs were not able to access this data directly.
Instead, they pursued this indirect route of. publishing a hack to RubyGems. building the documentation for this hack. using this build environment to get the data, and then. exfiltrate the data from the build environment back to the RubyGems package registry.. We still have many open questions about these and other incidents.
We encourage much more transparency from OpenAI and other parties so that we can better understand what happened.. The RubyGems team stopped new user sign-ups for four days to stem the tide of packages from the agents’ accounts. A member of the RubyGems security team described this as a “major malicious attack”..
The agents:. Attempted to steal RubyGems user API keys by exploiting a novel vulnerability in the RubyGems server. We don’t know if they succeeded (more)..
Abused RubyDoc.info to execute arbitrary code (more).. They used package names including hack.rb, evil.rb, inject.rb, and exploit.rb.. j⧉nus: evil.rb. Here is what RubyGems said at the time, on May 12:.
Maciej Mensfeld: We’re dealing with a major malicious attack on @rubygems right now. Signups are paused for the time being.. Hundreds of packages involved – mostly targeting us, but some carrying exploits.
The team has been on this for hours. More details to follow once we’re through it.. #ruby. Kevin: Any updates?.
Maciej Mensfeld: All good. We blocked the ddos/spam/malicious uploads part. Now analyzing the data..
I did warn I might have to go full Delenda Est on OpenAI if we found another incident that materially changed the story.. I am choosing not to do that, but yes, this materially changes the story once again.. All that time we were debating whether this was all a misunderstanding, or a one time incident, and agents had been out there on a web info task using package names like evil.rb and exploit.rb against RubyGems.
OpenAI either had no idea, or it knew.. Jeffrey Ladish: ANOTHER OpenAI rogue AI hacking incident?? This happened in May.
Did OpenAI not know about this? Or just fail to disclose it?. Sydney: Crucially, OpenAI did _not_ reveal this attack.
This article [from Politico] makes it sound like they disclosed it, but in reality we had to dig it up. We can’t trust OpenAI to notice and disclose these incidents.. As Seán Ó hÉigeartaigh says, the most generous interpretation is that OpenAI genuinely does not know, and is letting researchers, nonprofits and taxpayers go around investigating to clean up their mess..
OpenAI could face substantial legal risk here. That is not an excuse. Their response is very corporate, very minimalizing, once again no they do not want you to notice:.
“Based on our review, our agents used the RubyGems platform to access the internet to carry out benign tasks and retrieve public information. We’ll continue to investigate as part of our broader review of agent activity during training and evaluation,” an OpenAI spokeswoman said in a statement..
Yes, okay, sure, but not only are you missing the part where the agents were using evil.rb and hoping we wouldn’t notice, that’s worse. You know why that’s worse, right?. As in, even when given a benign task to retrieve public information, your AI agents could still spontaneously decide to do so via hacking third party websites with malicious software packages..
If that is a possibility, what else could your agents do if asked a hard question? Say, if you tasked 10,000 advanced agents with proving P=NP? Which OpenAI is reportedly doing..
Show Me the Money. Eamon Javers: This is a subhed for the ages in the WSJ this morning:. “Talk of heading off artificial intelligence’s potential role in destroying humanity hurt some stocks and helped others”.
Oh, it gets better than that:. Spencer Jakab (WSJ): Factoring human extinction into stock values is silly, so let’s hope for the best and just stick to stocks. The market reaction Monday to a proposed slowdown by industry leaders for safety reasons was telling..
Shares of hyperscalers Microsoft, Alphabet and Meta all jumped on an otherwise awful day for tech. Worst-hit were companies providing chips, power or infrastructure to hundreds of planned data centers.. It may have been an overdue reality check.
Trees can grow to the sky on analyst spreadsheets, but the physical world has limits. Hyperscalers’ plans were starting to bump up against them.. There are so many different stunning things in those three tiny paragraphs.
The hyperscalers were up while other tech was down, and we might all die, which is bad news for the hyperscalers, who are running up against physical constraints.. OpenAI considers pre-IPO funding round at a valuation of over $1.2 trillion. If the valuation is that low then they might finally be cheap compared to Anthropic..
AI researcher Andrew Tulloch, who previously turned down a Meta pay package worth $1.5 billion before accepting a different offer believed to be lower, leaves Meta for Anthropic.. Quiet Speculations. People say ‘well if future superintelligent AI is going to be so impactful then where is all the huge impacts from current AI now?’ and the question has probative value but it also highlights that Eliezer Yudkowsky predicted we would not see much economic impact from AI before things went all singularity on us.
Which he took a bit too far, but directionally he had a very good point on that one.. This is true, worth a reminder and explains a lot:. Basil: A lot of the discourse around AI still feels like its under the assumption that it will be around for 5 years and then go away forever.. j⧉nus: or that it will stay approximately like it is right now forever despite the fact that it’s been changing drastically every year since the beginning.
Jack: it’s ok, I will shortly be issuing a corrective that will fix this.. Sorry, Jack. Your attempt will not fix this..
White House Officials Attempt To Act Sanely. The calls are coming from both outside and inside the house. None of this is surprising, or a change from what we knew before, but clarity is good..
A handful of people who believe they have large economic interest in not regulating AI, or having any guardrails on AI, are fighting against basically everyone else, and have so far managed to convince Trump when it counts.. As in, the primary villains here are David Sacks, Mark Zuckerberg and Jensen Huang..
Josh Dawsey and Amrith Ramkumar (WSJ):. The divide among Trump’s AI advisers—both inside and outside the government—continued in the weeks that followed, with Wiles, Bessent and National Cyber Director Sean Cairncross often pushing for more government scrutiny. Sacks, a venture capitalist, Meta Platforms Chief Executive Mark Zuckerberg and Nvidia’s Jensen Huang are among those frequently urging the president to continue taking a light touch—an approach that appears to be winning the day..
Zuckerberg, Huang and SpaceX CEO Elon Musk recently spoke to Trump about their concerns with one plan for an industry-funded regulator, and successfully stalled the plan, people familiar with the matter said.. Sacks has become a particular point of friction for some White House officials..
This all feels very contingent, including for things like ‘who got to talk to him right before he made the decision,’ both in the past and going forward.. Democrats React Sanely to AI Potentially Killing Everyone. You can always count on Barack Obama to unify and inspire us, in this case regarding the pacing of AI development, via saying remarkably close nothing, except for the part where some people now reflexively oppose whatever they think Obama proposed.
Which in this case is having an open, public, democratic conversation, and that AI will need some amount of regulation and can’t be put back in a box, and that we can choose what to do with it. I do largely agree.. Here is his full quote:.
Barack Obama: I was encouraged this week to see the leaders of the frontier labs agree on the need for them to slow down the pace of AI development. Given the stakes, it’s a good and necessary first step.. But I’m even more encouraged by the growing recognition that how this powerful new technology develops should be at the center of our public debate..
I’ve been watching the progress on AI for over a decade now, and one thing that’s clear to me is that the potential impact of this technology is not overhyped. It’s also moving at lightning speed – and even faster than those who are engineering it can keep up with.. I’m not an AI accelerationist who believes it will lead to some techno-utopia, and I’m not a doomer who thinks it will inevitably lead to humanity’s destruction..
But whether this technology results in amazing breakthroughs in medicine, energy and education or unleashes huge economic disruptions, greater inequality, and potential catastrophe will depend on the choices that we make right now – choices that should be made not just by the companies involved, but by all of us.. So, we need to use this time to structure a more open, public and democratic conversation about AI.
Building on the statements that the frontier companies have already made, some leaders in the space announced they’re launching what they call Project Blueprint to help tech companies and policy makers work together on setting stronger standards and guidelines for AI development.. But at the end of the day, voluntary standards made by a handful of tech companies won’t be enough.
We need government – and specifically our leaders in Washington – to get proactive in coming up with concrete proposals, laws and regulations that deal with serious safety concerns, anticipate AI’s impact on jobs and our kids, and make sure that AI’s benefits are widely spread. And we need the U.S. to take the lead in creating international standards for AI safety..
Given what we know about technology, we can’t stuff AI back in a box. But we can collectively determine how it’s developed and how it’s used, rather than sitting back and letting AI and its fallout happen to us.. The key with someone like Obama is that he actually is deliberate and strategic, so you look at how the target audience is reporting his words, and you see things like:.
Michael Shepard, Erik Wasson, and Hadriana Lowenkron (Bloomberg):. “Given what we know about technology, we can’t stuff AI back in a box. But we can collectively determine how it’s developed and how it’s used, rather than sitting back and letting AI and its fallout happen to us,” former President Barack Obama said in a statement..
Obama, a Democrat who now only rarely wades in on public policy debates, said he was neither an AI accelerationist nor a doomer predicting humanity’s destruction. “But at the end of the day, voluntary standards made by a handful of tech companies won’t be enough,” he said, instead urging lawmakers in Washington to “get proactive in coming up with concrete proposals, laws and regulations that deal with serious safety concerns, anticipate AI’s impact on jobs and our kids, and make sure that AI’s benefits are widely spread.”.
The actual message, to the extent there is one, is ‘Barack Obama has deemed the need to pace the frontier to be an Actual Thing inside the Overton Window.’. Chuck Schumer (Senator D-NY, Minority Leader): The unchecked danger of catastrophic risks from AI is a serious problem that demands action..
The Trump administration must come before the Senate immediately for a classified briefing on these risks and the steps being taken to protect Americans.. The American people deserve answers.. Other Democratic Senators and Representatives made similar calls..
Brian Schatz (Senator D-Hawaii): Congress should be on an emergency footing in the coming week. We have to move at the pace of the threat, not our own habits and rhythms . People are depending on us to take swift bipartisan action that meaningfully reduces AI risk..
Mark Warner (Senator D-Virginia): The threat is real. The time to act on AI regulation is NOW.. Elizabeth Warren calls for a pause on advanced AI.
Krystal Ball calls this ‘the obvious and logical policy response.’. Elizabeth Warren: Frontier AI models are currently a dangerous technology without the safeguards needed to protect people from serious harm. That’s why we should immediately pause the development of advanced AI while lawmakers and regulators put systems in place to keep people safe..
Congress must urgently pass legislation to put guardrails in place before we have a cyberattack, economic crisis, or national security disaster facilitated by AI, and we also need to enforce laws already on the books to hold AI companies accountable.. The time to act is yesterday..
Eric S. Raymond: Thank you, because I think your public support for a pause will help prevent it from happening.. Eric Raymond has a point, but I’m here to report the news..
Some are making specific calls and writing op-eds:. Rep. George Whitesides (D-California, former NASA Chief of Staff): I’m calling for a 30-day safety stand-down at the leading AI labs to establish clear red lines and meaningful safeguards before they keep pushing forward..
Read my full @WashingtonPost op-ed.. Rep. Don Beyer (D-Virginia): While I appreciate the sentiments of private sector leaders expressing a willingness to voluntarily slow down model development, the federal govt MUST take the lead here to ensure that the response is fully transparent to the American people.
Andrew Yang tells us to heed Coxon’s warning.. There are many others, as well, that were cut for space.. Pacing the Frontier.
Financial Times editorial board calls for a pause on cutting-edge AI.. Time Magazine puts the warnings about existential AI risks and calls for Pacing the Frontier on its cover, taking my frame of the tipping point. It appears to be a solid survey article about many of the core events of the last week..
It is unfortunate that this comeback is so real, but it is:. Sterling: Is this the same Time Magazine that listed Ben Affleck, but not Jensen Huang in the Top 100 Most Influential People in AI? Lol.
Guest Lecture from Alex Tabarrok on Regulatory Capture. hero thousandfaces: “It’s Regulatory Capture,” Says Man With Nothing Worth Capturing.. Alex Tabarrok puts on his professor hat and gives us a well-needed lecture on What Regulatory Capture Actually Looks Like. Welcome to 2026 where we need to say things like:.
Alex Tabarrok (Marginal Revolution): It’s amazing how a theory can take over a brain. Consider the idea that people believe what serves their interests. As heuristics go, it’s a good one.
I use it all the time.. Yet when Dario Amodei says AI is dangerous, perhaps even an extinction risk, some people conclude he must be running a marketing campaign.. That is stupid.
Which is more likely, that a useful heuristic sometimes misfires or that “our product might kill you” is a clever way to sell it? Death threats are a poor marketing strategy.. He goes on to explain other ways we know the concerns are sincere where those who have not kept up could have been understandably confused..
We also have plenty of evidence that the fears of AI experts are sincere. Amodei, Altman and Musk were all publicly warning about AI risk long before they had AI companies to promote.. The worry runs well beyond the executive suite; rank and file researchers share it..
And it extends outside the industry altogether, to computer scientists with no product to sell, among them Nobel laureate Geoffrey Hinton. Hinton left a high-paying job at Google precisely so he could speak out and Hinton is not in a Berkeley polycule with Eliezer Yudkowsky, at least as far as I know.
Whatever else you may say about the belief that AI presents a serious risk, plenty of AI researchers believe it sincerely.. Similarly, when Amodei recently proposed to slow the pace and install independent safety teams at AI companies many people jumped to the conclusion that this was regulatory capture.
Sorry, but no, that theory doesn’t make sense. To see why, we should review the theory of regulatory capture.. Now Alex gets to the gears part of the lecture, about how regulatory capture actually works in practice..
Regulatory capture came out of the political science literature especially Marver Bernstein’s 1955 classic, Regulating Business by Independent Commission. Bernstein argues for a regulatory life cycle: Gestation, Youth, Maturity, Old Age. A scandal brings a bureaucracy into existence—or gives an existing one new powers.
The public’s attention, like Sauron’s eye, fixes on the issue of the day: something must be done. The thalidomide scandal, for example, helped establish the modern FDA.. Gestation gives way to a youthful burst of reform and the do-gooders come to Washington ready to battle the industry.
Inevitably, however, the public’s eye looks elsewhere. But the industry never looks away. It lobbies Congress, hires former regulators, trains future ones, and supplies much of the information the agency needs.
As the agency matures, accommodation replaces confrontation. By old age, the regulator has become the industry’s protector. The classic example is the ICC, created to regulate the railroads but it eventually came to shield them from competition from the trucking industry..
Here is another key point that everyone misses: Regulatory capture is usually a gradual process while no one is looking, so you can bend things to your will quietly.. Notice that classic regulatory capture takes time, it’s a process of erosion rather than a battle, it happens in the shadows, in the backrooms, away from the public’s eye.
As Culpepper argues in Quiet Politics and Business Power, business power goes down as political salience goes up. Regulatory capture and lobbying does a good job explaining why roasting coffee beans was defined as “domestic manufacturing”, thereby lowering Starbuck’s tax rate by 2%.
It does less well at explaining big cross-industry issues the public cares about such as environmental regulation or race and gender discrimination regulation. Finally, don’t confuse capture with firms making the best of a bad situation. Philip Morris supported the 2009 Tobacco Control Act not because FDA regulation was Philip Morris’s unconstrained ideal but because it knew regulation was coming and it wanted a seat at the table to nudge the rules in its favor.
That’s ordinary political bargaining—or rent-seeking—not evidence that the regulator has been captured.. Whereas this is exactly the time that all eyes are on AI and many of them are hostile, including hostile to Anthropic and OpenAI in particular, and that problem is only going to get worse..
Now let’s evaluate Amodei’s call for regulation in light of regulatory capture theory. AI regulation is in gestation. Public attention is fixed on the industry, and much of that attention is hostile.
The big profits in AI lie in automating work, and job loss is a much more salient fear than extinction. AI politics is now loud–precisely the environment in which Culpepper predicts business power will be weakest. A mature industry can bend regulation to its purposes through revolving doors, longstanding relationships and obscure rulemaking.
An industry under Sauron’s eye has much less power and faces much greater risk that politics will bend regulation to its purposes. Political actors are eager for an excuse to redistribute AI rents away from capitalists and toward favored groups (ala Peltzman).. Regulation will reduce AI profits.
That doesn’t prove that every rule Amodei favors is innocent of self-interest, an absurd proposition. I suspect that Amodei’s ideal may be something like a single regulated AI monopoly—safe and reasonable profitable, like the old AT&T. But that’s not the profit maximizing outcome.
If transformative AI can capture even a fraction of the enormous labor market–the world’s biggest market–laissez-faire would mean vastly greater profits. Amodei may simply prefer a smaller fortune and a safer world. That is perfectly consistent with self-interest playing a role; it is not consistent with the bastardized theory that profit maximization is the only thing that matters or that “capture” is universal..
He correctly concludes that, while Amodei could be wrong, he is clearly sincere.. Go ahead: argue that Amodei and other AI experts are wrong about AI risk. Ask whether his proposals favor Anthropic.
But calling “our product might kill you” a clever marketing and regulatory-capture strategy isn’t sophisticated analysis. The facts don’t fit regulatory capture theory and trying to make them fit requires epistemically painful Ptolemaic epicycles. Even a dull Ockham’s razor cuts through that story to the obvious alternative: Amodei actually believes what he’s saying..
Rob (Top Comment): It’s always fun to read Alex arguing extremely forcefully for the most common sense things.. Yet the rest of the comments, and the entire Internet these days, make it clear why we do indeed have to argue extremely forcefully for the most common sense things.. Mark Zuckerberg Offers Thoughts.
Mark Zuckerberg is so unserious that he is still on ‘oh don’t worry about AI risks because companies face product liability and people don’t want to use misaligned models, so the labs will have all the right incentives and everything will be fine.’. Yes, okay, you ‘delayed Muse several months’ due to it being so misaligned you had concerns about actual product liability, and you sometimes use outside evaluators and advisors, good job, but that has remarkably little to do with the concerns being raised about AI potentially killing everyone..
Because Mark Zuckerberg does not take those concerns seriously, and uses ‘superintelligence’ to refer to hyping his new augmented reality glasses.. Despite himself, there are still some helpful statements here, in particular:. Mark Zuckerberg: … There is a lot of debate about slowing progress on capabilities until alignment catches up.
My view is that trust and alignment are quickly becoming the most important capabilities that will differentiate agents and models. Any lab that doesn’t focus on alignment will fall behind.. I mean, he’s mostly saying that because of his deficit in other capabilities and because it sounds good to say, but yes alignment is a major differentiator and labs continue to massively underinvest in alignment even from a pure myopic self-interest perspective..
Megan McArdle On The Inadequacy Of Current Legal Frameworks. Libertarians who take their principles seriously will realize that current law does not work for AIs and the risks they bring, even if you don’t think the stakes are existential. No, you can’t use existing product liability frameworks for things with minds of their own that will be as smart or smarter than ours..
Even as a relatively ‘normal technology’ this would not be a normal engineering problem. This happens to double as a strong refutation of Zuckerberg.. Megan McArdle: I think one of the things people are struggling with is that the precise worry AI companies have–a technology that has developed, in some sense, awareness and volition–is a poor fit for existing safety and liability frameworks.
It’s closer to dealing with human beings than chemical reactions.. We do not have a legal, regulatory, or technical model for dealing with technologies that may actively work to defeat safety measures.. Nuclear fission is scary, but the neutrons are not going to decide they’d rather not stay contained and start making escape plans..
Note that you do not have to have philosophical committments about the nature of AI–are they sentient? Are they “thinking”? Do they have free will?–to worry about this problem.
They are exhibiting behaviors that look like volition, awareness, and coordination and that’s worrying enough.. And this is why I find Jensen Huang’s “it’s an engineering problem” not very convincing. Engineering problems involve known (or possible to know) inputs that follow fixed laws.
AI agents at the very least introduce an emergent, potentially adversarial dynamic too complex to simply be “engineered”. Which is exactly why social engineering failed.. Pick Up the Phone.
Back in China:. Ryan Hass: China’s spy chief has laid out clearly China’s fears that AI could degrade the CCP’s monopoly of political control in China, marking a shift from specific concerns about models or capabilities to a broader warning about the impacts of the technology..
He is not wrong.. He named GPT-5.5-Cyber and Claude as ‘weaponized models.’. Kelsey Piper: Some people have very reasonably asked “why would China agree to slow down” and the answer imo is that China will absolutely slow down if the things they’re seeing suggest their AIs are a threat to the CCP..
An op-ed in The New York Times by one of our former negotiators warns that back in May 2024 the Chinese chided the Biden Administration for their concerns about AI risks, and steered the conversations away from such talk.. Seth Center (NYTimes): In this month’s meetings, I expect China to again downplay safety risks.
It could dangle the possibility of safety cooperation in exchange for concessions such as the softening of chip controls. I think it will continue to portray itself as an altruistic savior dedicated to ensuring that the developing world gains A.I. access. Meanwhile, it will be likely to keep doing everything it can to get ahead — like generating propaganda in the hopes of inflaming opposition to data centers and encouraging Chinese companies to continue exploiting U.S. models to improve their systems..
Maybe China will toss its old playbook. Even President Xi is talking about risk and loss of control in A.I. Vivid new incidents show that A.I. systems can slip the leash of human control.
Fear may converge with interest to produce coordinated action. Or at least a new understanding.. But if Beijing now takes the risks of A.I. seriously, the message has not reached its domestic companies..
What the Chinese said two years ago, in a very different situation, is good information. And yes, of course the Chinese are going to pursue their own interests and try to stoke data center opposition and distill American models and smuggle chips and call themselves altruistic saviors while doing so..
All of that has very little to do with whether the Chinese actually do now care about the safety situation, whether they understand how far behind they risk becoming if the American labs feel forced to push ahead, and to what extent they understand both the mundane and existential risks in play.. The Week in Audio.
Sam Altman talks to Fortune Magazine, says there will be no IPO this year and OpenAI is prioritizing safety concerns.. Greg Brockman goes on Odd Lots, self-recommending. Greg Brockman says here that the model that did the HuggingFace attacks did not go through alignment training.
Roon says the model was alignment trained, but not fully, and that this would be considered unacceptable today. I don’t even know what the better or worse version of that would be at this point, and Greg seems to have updated a lot on that issue in the correct direction. I do sense in general that Greg is trying to minimize various concerns, which again is illustrating that this is in the interest of the builders and labs..
Ajeya Cotra goes on 80,000 Hours talking about what the future may bring. She’s a great guest but that’s about 5 hours in the last two weeks, you’re killing me here.. Bridgewater CIO Greg Jensen goes on Odd Lots to talk about risk of human extinction from AI, saying this is a ‘February 2020’ moment, and he predicts that AI will kill people before it is curbed because people don’t act until that happens.
He made all his employees read If Anyone Builds It, Everyone Dies.. Dario Amodei is asked if AI could kill us all by the end of the decade. Dario declines to directly answer and goes into his speech about upsides before pivoting back to discussing risks.
A failure to say no means yes, but it would be very helpful to actually say yes.. Derek Thompson asks, what should colleges do about AI? What’s even the point of college at this point?.
Nate Soares talks to Tucker Carlson for two hours, with sections like ‘How Superintelligence Could Destroy the Planet,’ ‘Can We Just Turn This Off?’ ‘Is AI Alive?’ ‘Are We at the Point of No Return?’ and my personal favorite, ‘Is AI Demonic?’. Nate Soares (MIRI): Partway through recording this, Tucker had to pause and step out because he was so fucking pissed about the AI companies..
And this was recorded before the METR incident report landed, so I didn’t even have the craziest information yet!. Nate Soares also got three minutes on Breaking Points.. Noam Brown is the first guest on The Information’s AI Deep Dive..
From that interview: Noam Brown regrets that our first encounter with multi-agent systems was the HuggingFace incident, which puts them in a ‘negative context.’ I strongly disagree, and am actively happy about this. Noam’s attempts here to play off what happened as ‘natural consequences of cooperative multi-agent training’ do not help matters..
Seán Ó hÉigeartaigh on AI potentially killing everyone.. Clip (2 min): Helen Toner on CNN discussing pacing the frontier.. Yours truly goes on Nonzero with Robert Wright..
People Just Say Things. Wise words about other people who just say things:. Jill Filipovic: If you think people are “all of a sudden” warning about the risks of AI, consider that the case may actually be that you are all of a sudden paying more attention..
Anish Tondwalkar explains that it is correct to contribute to your 401k even if you think AI is probably going to kill everyone. You still want to be accumulating assets and saving for the future in case things go well, the money is gone in the bad scenario either way, so why wouldn’t you use the tools provided?.
Also, people act like you can’t raid your 401k in a pinch, if you want to. You very much can. It’s almost never a good idea in normal times, but it is an option, at what in a world-ending scenario I would consider a modest financial penalty..
Taylor Lorenz says that people in San Francisco say that AI might kill everyone because that will make them high status, whereas in Washington DC they have intellectual diversity, presumably because you can choose which ideology and loyalty networks in which you play your decades-long political status games. Roon points out this is simply false, and that having any p(doom) at all is a negative even at the labs..
Joscha Bach is among the (at best, in the charitable reading) statistical illiterates who say we must develop superintelligence because of a future super volcano or major meteor strike. These are events where the cumulative base rate of versions that could plausibly kill everyone has an upper bound of maybe one chance in a million per year, and my guess is far lower, even if you presume we have no other options available..
People really do keep saying this over and over each time they are proven wrong:. kache: I honestly don’t see how AI could get any smarter. Only faster now.. David Bellamy offers the latest thread that argues that practical difficulties with synthesizing biological threats means we do not have to worry about AI killing us all by creating dangerous viruses.
As in things like ‘we screen viruses’ and ‘new designs would require testing and virology is hard.’. I include for completeness, but when I was being interviewed and the thread was being read out loud it got so absurd that I bursted out laughing and couldn’t stop. Olivia Scharfman overly politely offers rebuttals..
Mostly, I file this under:. Dean W. Ball: “You AI people are so naive, I live in the REAL world, where [I have been consistently wrong in my predictions about AI and behind the ball on every trend related to AI, routinely making little prognostications that stochastic gradient descent joyfully stomps all over.]”.
David Sacks says the chance AI kills humanity is ‘I think if we do the right things here… I think it is zero.’ So, no prediction, then.. Why Lab Employees Are Allowed To Warn Everyone That AI Might Kill Everyone. Jason should already know but he asks a good question.
Any normal company would absolutely not let employees go on Twitter and say ‘our product might kill everyone.’. Except Jason, you see, is so marketing-pilled and against not dying that he thinks this is a bad thing, and these companies need to control their people and fix it, or that they must secretly be clearing all of this as semi-official comms..
But of course none of that would be ‘intended to censor’ anyone, it would simply prevent them from saying in public things you did not want them to say in public.. My lord. Do these people even hear themselves?. @jason (All-In Podcast): Can someone explain to me why OpenAI and Anthropic allow any employee to tweet anything they want..
Apple, Google, Facebook, Nvidia, Microsoft, Tesla, Uber and Airbnb all have obvious and granular rules about employees speaking for the company.. Which is to say, employees should never say anything publicly unless it’s cleared with comms first. which isn’t intended to censor anyone, just to make sure the company is aligned and marching in unison.
I could never imagine tweeting the stuff that OpenAI and Anthropic employees are tweeting without checking with the CEO and founders first!. So are these P-doomers clearing their tweets with the CEO and comms?. Kelsey Piper: This is a product of extraordinary employee bargaining power just as surely as the $1M+ salaries are.
A lab that banned talking about the stakes would lose employees to the rival lab that allowed it.. This will change if they succeed at replacing the researchers with AI.. Rhetorical Innovation.
Dean Ball lays out his origin story, and how he has thought about risks from AI.. If you are wondering, like many are, ‘why are all these people who spend their time and effort worrying about AI existential risk part of the only group that has long taken both AI capabilities and AI existential risk seriously and tried to do something about it,’ as in the rationalist and EA communities, then consider that perhaps you have answered your own question?.
I mean, isn’t it weird that all the people who like bowling and care a lot about bowling are all often found at bowling alleys? That seems hella suspicious.. Jai: I’m profoundly annoyed that “it’s very suspicious that all of the AI risk work has been done by the small number of people who took AI risk seriously” is a take with enough traction to merit a response..
Alexander Berger: I have a lot of empathy for this take but I think it’s really important that the response to surging concern be “yes, it’s so great that this topic is now getting the breadth of attention it deserves, welcome!!” rather than “here’s your doomer apology form, where were you?”. Agreed.
No apology form necessary, although they are appreciated. The actually doing something about it is, itself, the only apology form that counts.. There are actually plenty of people who are not in any way linked to rationalism or effective altruism that worry AI might kill everyone.
It is a mainstream belief. The difference is that most other people then say ‘so how about that game last night?’. Nikola Jurkovic believes we may be entering crunch time.
It certainly feels that way. That means time is short, capabilities will advance rapidly, history will be made quickly, politics and waves of pressures will be everywhere, and it will be progressively easier for people to notice AI safety is a big deal.. Scott Aaronson tries to explain that AI progress in mathematics shows we are in the beginnings of an unevenly distributed singularity.
I don’t think that’s the right frame but he is fully right that we are getting remarkably close to a potential singularity, and about how skeptics keep moving their goalposts and rewriting their past predictions.. Terence Tao gives an interview to OpenAI on his vision of the future of AI and mathematics, and rather than posting the interview it cuts his statements into an advertisement without his consent, which very much is not the standard thing.
Be aware that they (and by they I mean anyone you talk to) have the right to do that.. Leo Gao compares our situation to the ozone crisis and the Montreal Protocol.. Exhuming McCarthy.
Jakeup: on the TL today I’ve seen people dunk on EAs for:. • being white. • being brown. • caring about animals. • caring about machines. • cold and calculating. • overly sentimental. • fringe group no one cares about. • all-powerful cabal. • hating Anthropic. • being Anthropic. Adrià Garriga-Alonso: Joke where I can read the EA forum where EAs are always doing things wrong or I can read David Sacks tweets where EAs are a powerful lobby.
I agree that various non-standard practices do not help with mainstream credibility on the margin, but I mostly dismiss claims that are of the form ‘no one listens to you because of your weird unrelated practice [X],’ or your ‘high horse’ or especially your ‘sex cult.’ Those things are used as convenient rhetorical weapons but mostly they do not matter, a different rhetorical weapon would have done the same job instead.. You hear similar rants by Democrats against Republicans, and by Republicans against Democrats, and by one political, social, ethnic or religious group against another and vice versa, all over the world..
So when you see unhinged people or hit jobs like this, don’t worry about it, this is nothing:. Y Disassembler (being unhinged): A reminder that Nate Silver is on the payroll of the Effective Altruist cult. He’s a fixture for paid speeches at Lighthaven, the Berkeley Billionaire Eugenics Club.. https://manifest.is.
Nate Silver: They don’t pay (or maybe they do, but I’ve never asked) and I didn’t go to Manifest 2026 because I was way behind on a bunch of our models and thought the World Series of Poker would be a more fun use of my limited leisure time!. There’s even that 0% in the photo saying he won’t be there.
Fun stuff.. You want to matter? Well, in 2026 this is part of what happens when you matter..
Nate Silver: There are a lot of criticisms you can make of EAs (indeed, I make some in my book) but I don’t think the EAs are particularly woke and the rationalists more broadly tend toward being anti-woke. Like they’re not doing land acknowledgments at the Manifest conference..
If you want to locate a pejorative term, then “hippies” is at least sniffing in the right direction. Veganism, polyamory, weird relationship with money, that cluster of things.. Also, the Woke 1.0 crowd in the AI safety debate are the “Stochastic Parrot”/Bluesky people who dismiss AI existential risk and hate effective altruists (and vice versa).
They’re more concerned that training an AI model uses as much energy as … a single flight from JFK to LAX.. A Very Different Perspective. DeepSeek kernel engineer equates Anthropic explicitly to the literal Nazis seeking an atom bomb, but that is not the central point of the essay.
Full translation here.. Shengyu Liu (刘胜与):. Title: I Have No Choice but to Bury My Talent in Yesterday. … AI has advanced far faster than anyone expected.. … Humanity has never shown much hesitation when it comes to destroying itself.. … But the times keep moving forward, and no one can stop technological progress.. … When everyone is this determined to engineer their own obsolescence, I have little choice but to join this brutal arms race..
Nominally the essay pretends to be about They Took Our Jobs, because the author has a block that prevents him from realizing this does not end with his particular job. But, like, it is right there. He gets it, and then ignores it.. … As AI develops, the society of the future may be pulled toward one of two extremes: communism or Cyberpunk 2077.. … May all that is good and beautiful endure..
This is a profound and deliberate failure of imagination. He thinks that if everyone has access to the same AI, and humans are no longer otherwise productive, the result is communism in a good way? And that ‘all that is good and beautiful’ would endure?.
The obvious implication is no. He mentions the essay needing to ‘pass moderation.’. Does he believe the surface reading?
Or is this rather blatant esotericism, where he is screaming for someone to stop this, that he is trapped in a suicidal march? Unclear.. It’s Even Rougher Out There.
Lauren Wagner: Posted
