Amodei与Altman谈AI失控风险及Anthropic内部调查

Okay, it is true that half the internet seemed to get hacked in the last week, but for me being totally honest, this counts as a quiet few days by AI standards. I totally know that that is not the way to start the most super hyped YouTube videos, especially on AI, but if every week defies our expectations, then maybe we need to have better expectations. But this doesn't mean that there hasn't been a lot of developments that should have made headlines if more people were paying attention.

From Anthropic predicting the automation of their company and their CEO Dario Amade sending somewhat mixed signals to Sam Orman admitting that OpenAI's top plan involves hoping models want to keep humans in control. And we'll look at Google's re-entry to the frontier causing an apology from one commentator. and then I'll give my mental model as to why models keep impressing us with unforeseen capabilities. So for sure it's going to be a nuanced video, but I must say overall I am pleasantly surprised that lab leaders are continuing to invite criticism and debate.

Humans have never not been the best all round pattern recognizers on the planet. Losing that lead was always going to send a lot of people a little crazy, even if they weren't economics, war games, and rogue agents to worry about. I'm going to start with a very recent survey of anthropic staff that caught my eye. And I'm curious where you guys would fit because only a minority of US adults think AI will be capable of a Nobel discovery by 2030.

That's in contrast to 91% of anthropic staff. For what it's worth, I think a model would be capable of such discoveries now in say the field of economics. And given that computer science now seems to count for physics for the Nobel Committee at the latest in that domain by 2028, apparently 91% of anthropic staff think it's possible by 2030. But there's a cute contrast that then comes up because even though it will be capable of this Nobel level discovery, according to anthropic staff, those same staff think that it will only be able to do 75% of their own work.

Win a Nobel Prize for sure, but not do my job. Leo Gao of OpenAI gave a few days ago a different view. Lab employees, you have power only because you are useful. That won't last much longer. Use that time wisely. Before we leave this survey, I would note one discrepancy though. The vast majority of anthropic staff still believe there will be some human role in training a frontier AI model by the end of 2030. Yes, on average they think that 95% of the work will be done by an AI itself training a frontier model.

But between the 10th and 90th percentile of anthropic respondents, there was still some sliver of work being done by humans. This is in slight tension with the quote I gave you in my last video from the co-founder of Anthropic who said that full recursive self-improvement, the model improving itself he thinks will be possible in 2027. That's an update from his earlier view of 2028. Now, is he an outlier or is there a subtle difference between full RSI being possible and full RSI being done by AI?

In other words, are these researchers saying that we will voluntarily still leave a small role for the humans? Again, I think it's much more likely that those staff just think a model won't be fully capable by 2030. Otherwise, they wouldn't have said that 25% of their job an AI just simply couldn't do by that date. I lean a bit closer to Jack Clark, but let me know what you think. If full RSI is possible, then the views and actions of Anthropics CEO Dario Amade become more important than ever.

I was pleased to see him reply in detail to a critique launched by Nathan Calvin, chief strategy officer of Encode AI. And to be fair, Amade invited criticism, saying it benefits the public to have people whose job it is to scrutinize the actions of Frontier AI companies. We should be harshly criticized basically every day, he says. Brilliant. Well, he's going to love this video. None of what I'm about to say is implying that the birth of superhuman pattern recognizers was ever going to be easy for any of us.

It's more that I just think that Amade has evaded a bit more scrutiny compared to other lab leaders like Sam Orman. I'm going to pick out five quick points just to give you a flavor of what I'm talking about. He's obviously a very smart guy, so you have to look a little closer to spot the subtle rationalizations. first on this race to the top in AI that he celebrates. For context, when Anthropic were behind, say in 2023, he used to celebrate them not being ahead.

Praised himself for being responsible and not accelerating. He said in October 2023, it was wrong to release CHQPT and Anthropic made the right call in not releasing Claude at the time. A kind of big loud public release might accelerate things so fast that the ecosystem might not know how to handle it. I didn't want our kind of first act on the public stage after we put so much effort into being responsible to accelerate things so greatly.

I generally feel like we made the right call there. Fast forward to now with Opus 5.5 as the best model and Fable 5.5 ready to go at any point. Amade says, "Our strategy has always been what I've called a race to the top. The forces that drive the race to the top are hugely more effective if the company practicing them is at the frontier. That is effectively restating a previous quote from Anthropic that quote you can't do safety from second place.

Agree or disagree seems to be a rationalization of whatever anthropic are currently doing. Next, Amade is famously belligerent toward China wanting to ban chips going there and his stated reason this week is that this is because we think it will allow democracies to negotiate with China from a position of strength on AI safety agreements. Seems fair enough, right? They want AI safety agreements desperately. In the recent pacing the frontier post, he has called for an agreement with China on a speed limit on the rate of recursive self-improvement.

But he has also said that one of the goals of having powerful AI would be to empower democracies to use their intelligence services to disrupt and degrade autocracies from the inside. Not sure that's going to help terribly with AI safety agreements, but we'll have to agree to disagree. Here's another one. Nathan Calvin pointed out how much Anthropic is investing in AI acceleration and Amade said, "Well, yeah, we need this to compete at the frontier and we stand by these choices."

He gave lots of examples of how they're stopping misuse of their current models. But then earlier he'd said this, maybe one way you could object is that some of these items relate to AI misuse and not loss of control. Humans misusing AI, not the AI itself getting out of control. If you think the latter is much more important than the former, then that might change your calculus. But this is not anthropic's view. Don't focus too much, in other words, on all this acceleration.

Focus more so on how we're stopping misuse. We've always said these are equally important. That could be slight news to the head of alignment at Anthropic, who after giving his famous prediction on risk to humans said this. I think the risk from present models is low. What I am worried about is super intelligence arising from recursive self-improvement. Now I know what you're thinking. Amade said the company as a whole, not one individual.

So let's read Anthropic's core views on AI safety. A charter about what their entire company believes written in part by Amade. At the time of writing, they said the future is hard to predict, but there are optimistic scenarios and pessimistic scenarios. In the optimistic scenario, one of the main risks from AI is things like toxicity and intentional misuse. Human misuse being the major problem in the optimistic scenario.

And check out the footnotes. Again, in that optimistic scenario, the stakes of anything anthropic does are fortunately mu