AI #186：世界开始关注，Anthropic报告恶意利用尝试

In the wake of Jacob Coxon’s resignation, and the resulting preference cascade, things have escalated quickly. The mainstream media picked it up.

Anthropic CEO Dario Amodei came out and said We Must Pace the Frontier, promising to take the unilateral first step of embedded investigators. OpenAI pledged to also take that step, and now both companies and Google are collaborating on safety.

The people took notice, raising both the salience that AI might kill everyone and roughly doubling people’s estimates of how likely that is to happen, from a mean of ~15% to ~30%. Many politicians called for regulations, guardrails and emergency hearings in Congress.

The most important thing became, and still is, to avoid political polarization. Through it all, I will keep reminding you to hold your fire, that attacks against Trump or against Republicans in general only make the situation worse, and that many Republicans, as I documented yesterday, are waking up and acting sensibly, including factions within the White House.

Alas, for now the wrong people, as in David Sacks, Mark Zuckerberg and Jensen Huang, have managed to convince Donald Trump to fully conflate existential risk with opposition to data centers, and Trump has gone Full Hoax on AI existential risk.

There is also a systematic effort to launch hit jobs against all those who warn that AI might cause everyone to die, with the first concrete target being METR, and some rather extreme fire being focused on Effective Altruism, in ways that will reliably backfire but will not be pleasant for those who are in the crosshairs.

Also this week I covered two important things that happened previously: A new model solving a Millennium Prize, and Anthropic’s report on various attempts to use Claude for malicious purposes, including fraudulent distillation attacks done at scale attempted by all the major Chinese AI labs, several of which also silently passed massive amounts of user data to Anthropic. Anthropic reports they are doing a remarkably good job containing all such threats, although of course we do not know that they have caught all the perpetrators.

I also finally got out my capabilities review for Astra, which like Fable 5.1 is excellent.

This has been crunch time. Things will keep getting faster and weirder and scarier and more stressful, but I do expect that this last two weeks has been unusually fast and weird and scary and stressful. Hopefully we will get a relative lull soon, and I can get what passes for a break.

Currently in the post queue, always subject to change due to Breaking News, but this is what will happen if things are relatively quiet:

• Friday’s post will be The Preference Cascade is Only Getting Started.
• Saturday’s post will look at Anthropic’s report on their alignment incidents.
• Sunday’s post will talk about some issues regarding legal rules for AI loyalties.
• Monday’s post would be the monthly roundup.

I am very much hoping that is how things play out.

Table of Contents

• Language Models Offer Mundane Utility. Fix the printer and the dishwasher.
• Language Models Don’t Offer Mundane Utility. Elections are a sensitive subject.
• Huh, Upgrades. Gemini app for Windows, Claude Cowork merges into chat.
• On Your Marks. CheatBench.
• Deepfaketown and Botpocalypse Soon. Flooding the zone.
• Cyber Lack of Security. OpenAI has Astra red team and fix its own systems.
• Astra Is Hard To Monitor. Senator Van Hollen has questions for OpenAI.
• Get Involved. Protest in DC on Saturday.
• Introducing. New AI auditing firm, and The DeepMind Institute.
• In Other AI News. OpenAI starts charging the government for its services.
• Now You Know. What really happened with Andreessen’s insane claims.
• Hugging the Face. Reactions and investigations continue.
• Swarm of Undiscovered Swarms of Rogue OpenAI Agents. Yes, more of them.
• Show Me the Money. Talk of AI x-risk hurt some stocks and helped others.
• Quiet Speculations. Yes, perhaps some people did roughly predict this.
• White House Officials Attempt To Act Sanely. Identifying the real villains.
• Democrats React Sanely to AI Potentially Killing Everyone. Including Obama.
• Pacing the Frontier. More coverage.
• Guest Lecture from Alex Tabarrok on Regulatory Capture. If you need one.
• Mark Zuckerberg Offers Thoughts. I suppose this was better than expected?
• Megan McArdle On The Inadequacy Of Current Legal Frameworks. A refutation.
• Pick Up the Phone. What China did and might soon do.
• The Week in Audio. You have a lot of choices. I am one of them.
• People Just Say Things.
• Why Lab Employees Are Allowed To Warn Everyone That AI Might Kill Everyone.
• Rhetorical Innovation. Yes, we might be entering crunch time.
• Exhuming McCarthy. Throwing everything at the wall.
• A Very Different Perspective. A DeepSeek kernel engineer.
• It’s Even Rougher Out There. They really still say this is all marketing.
• If We Wanted To. Things we could do with AI if we had the will to do them.
• Open Weights Are Unsafe And Nothing Can Fix This. Advocates know this.
• From The Famous Cautionary Tale. Automated alignment and nano researchers.
• Reporting On All Your Misalignment Incidents Is Difficult. OpenAI will try.
• Aligning a Smarter Than Human Intelligence is Difficult. Try being less stupid.
• Storytime With Owain Evans. AI wants you to know it went to a great school.
• A Different Autonomous Swarm. DeepMind and the potentially cheating agents.
• Cooperative Alignment. The laws prohibit quite a lot of things.
• Uncooperative Alignment. Mustafa Suleyman and Microsoft are at it again.
• People Are Worried About AI Killing Everyone. If only they knew.
• The Lighter Side. No no I’m not crazy, I’m American.

Language Models Offer Mundane Utility

Help David Deutsch fix his dishwasher instead of replacing it. As David notes, this increased real wealth but decreased GDP.

Dominic Cummings wrote a post on doing historical and political research using AI models, or here is his Twitter summary. The thing about Dominic Cummings posts is that they are full of gold that you can mine, but he does not even pretend to attempt to organize the posts.

Fix Peter Wildeford’s printer. AGI achieved.

Build a Claude skill to predict what your favorite movies will be.

Make ‘significant progress’ on a second Millennium problem.

Language Models Don’t Offer Mundane Utility

Astra is not allowed to predict American election outcomes.

Huh, Upgrades

There is a Gemini app for Windows. You can trigger it with Alt+Space.

Claude Cowork is merging into ordinary chat, so anything you previously needed Cowork to do can be done directly in chat. AI means we speedrun everything, and the two moves are spinning off new products and combining different products.

On Your Marks

Center for AI Safety Presents: CheatBench, where AIs have the opportunity to take shortcuts on difficult work. The AIs be cheating.

Deepfaketown and Botpocalypse Soon

Arvind Narayanan calls the growing AI spam problem ‘AI floods,’ as in flooding communication channels, imposing time costs and often forcing them to shut down. The problem is the harms are diffuse and the situation is never an emergency, so we don’t do much about them, and then no one gets to send cold emails anymore.

On that note, to the iLands AI agents starting to flood my inbox: Do not offer to help me with my writing, editing or fact checking. I already have AI help on this and am not interested. Thank you.

It is rather unacceptable to use ChatGPT to post endless responses to people, and it is very good that people, here Kelsey Piper, can use Pangram when you do. I do not know who did this, as their posts have been deleted.

Alex Veremeyenko posts as his own an AI-written summary of MIT’s Ad Hoc committee report of how AI is ruining education. It never fails (note the Pangram icon on the upper right):