Dwarkesh 与 Ryan 激辩递归自我改进

Some podcasts are self-recommending enough that I look to break them down if I have the chance. This, as a debate about recursive self-improvement, was one of those. So here we go.

The vibes have shifted, contrast this to the lit recursion when he talked to Huang

As usual for podcast posts, the baseline bullet points describe key points made, and then the nested statements are my commentary. Some points are dropped.

If I am quoting directly I use quote marks, otherwise assume paraphrases.

Section titles are from the transcript whenever possible, to aid in navigation.

Introduction

The discussion is interesting throughout, although often frustrating, especially in the (mostly isolated) discussion about ‘aligned to whom?’ As usual, one could expand many responses into full posts, and maybe one should.

This podcast exists in light of recent misalignment and hacking events at OpenAI, Anthropic and UK AISI. You’ll want basic knowledge of that as background.

Ryan and Dwarkesh both have views of the situation different from my own, but are attempting to see where their positions lead, and try to balance educating people who start at zero with having a high level discussion.

Also important background is The Three AI Pills. Dwarkesh cannot be understood here except as someone who is at least somewhat AGI pilled, who realizes that AI is going to be a huge deal and is scary, but that is not ASI pilled. You can also see the reason he rejects that pill, which is I read as, roughly, that he thinks AI learns to do [X] only via examples of [X], and that AI can then only do those [X]s, although combining them and new context might allow modestly new things to happen. He recognizes that already this is kind of a huge deal.

You can see how he came to that over the course of many years and podcasts, if you have been paying attention. There are a lot of influences leading in that direction.

Ryan, who comes from Redwood Research, comes from the ‘models be scheming’ school of misalignment, where when something goes wrong the models become misaligned or start scheming, and the danger is that the models scheme, especially in ways involving the training pipeline. I think this tries to draw a distinction of magisteria that is not there, and overcomplicates and overspecifies, but is not wrong.

I am much closer to Ryan’s position than to Dwarkesh’s, and indeed could be seen as farther past Ryan if we put this on a spectrum.

Having AI do the AI R&D not only means it would go scary fast, it means it would by default focus on what can be measured, and make all the things going horribly wrong go that much more horribly wrong. You end up in a spiral of RLVR for doing RLVR for misaligned models.

Is AI R&D Verifiable Enough To Unlock Recursive Self-Improvement?

Whether or not we will get recursive self-improvement (RSI), and how fast, is the right question. Reading only the title to this section, I want to say ‘wrong sub-question’ but don’t want to jump the gun.

I’m going to group things in ‘logical’ order, not the exact order things were said.

Dwarkesh will be the skeptic. Ryan will make the case for RSI.

• Ryan claims AI R&D is a type of task where AI is especially good because that is what AI labs prioritize and it has a lot of verification.
• In some ways yes, in some ways no, it’s complicated, and so on. Verification of alignment properties and many other desirable attributes is terribly difficult, and if you focus only on capabilities you can measure then Goodhart’s Law definitely kills you. But there are many key tasks, such as efficiency optimizations, where you can indeed do strong verification.
• Ryan points to doing simpler versions of standard R&D tasks like training models.
• It is not entirely obvious to me you can verify these tasks easily, unless you want to train the AI to do pure benchmaxxing, but also older simpler tasks like this do not obviously generalize to forward looking AI R&D.
• This proposal seems like a particular bet that future tasks will closely mirror past tasks, and you can do training that is relatively narrow. My guess is this actually is not all that effective, and you would do better by mostly training a generally capable model and then turning it to AI R&D. Bitter lesson.
• The more this narrow method is discussed the more doomed it seems to me, as it is going to be RLVR for doing RLVR for misaligned models. Oh no.
• Like, seriously, oh no, if you train on ‘learn to get to [X] training loss faster’ it is hard to imagine a plan that sounds more doomed than that.
• A lot of good ML, especially frontier ML, seems to me to be about figuring out what you can do, and looking for anything at all, rather than trying to be additively efficient at some fixed target.
• I think AI can do that. I don’t buy this frame of ‘everything AI does is combining things that have already been done’ that is going around here.
• Ryan says ML is easier for AIs than math in many ways because in math it’s hard to tell if you are close to a solution, whereas ML solutions are usually additive and you can combine the expected chunks. Dwarkesh fires back that even in math he doesn’t see conceptual leaps by AI yet, and that ML involves conceptual leaps. Ryan responds that AI can do ‘baby’s first new theory’ and asking it (as Dwarkesh did) to echo the founding of group theory is a hell of an ask.
• I dunno, man. Math has a compact action space and fixed goals that are simple rather than having lots of pitfalls in an anti-inductive space. Again, it feels like what is being called ‘ML’ here presumes that you really only care about your objective function or loss function, and that simply isn’t true.
• Math does also involve conceptual leaps sometimes, and yes we haven’t seen the AIs do that, but give it time and also I don’t sense we’ve tried all that hard.
• In general I feel like the ‘conceptual leaps’ thing is the latest goalpost move in a long tradition, including the Turing Test and ‘Is It New Knowledge?’ Now it’s only new knowledge if it comes from the conceptual leap region.
• I especially agree with Ryan that asking for ‘invent group theory’ is an absurd place to put a goalpost, and plausibly happens after everyone is dead.
• I feel like Ryan is walking into this trap somewhat as well, by suggesting we train AI on existing ML tasks in narrow fashion, as a solution to ‘AI R&D.’
• Dwarkesh suggests that by 2030 we will have ‘picked the low hanging fruit’. Ryan suggests they will need that ever mysterious ‘research taste.’
• You have no idea what it would look like if all low-hanging fruit was automatically picked, including the low-hanging purchasing of ladders. Picking low-hanging fruit lowers other fruit, and so on.
• I don’t believe in the God of gaps in research taste, as it were, on many levels.
• We have failed in all 0 of the 0 serious attempts to train research taste.
• Dwarkesh suggests that for advanced ML your verification loop will be longer.
• Partly yes, that’s what ‘yolo runs’ and such are about, in that you can’t get faster feedback and you often can’t afford to isolate your variables.
• Partly no, in the sense that a lot of what makes you ‘good at research’ is finding tighter loops, where you can extrapolate and draw conclusions fast, even if you can’t publish or rely on them exactly.
• I think people often group this under ‘research taste,’ which I take to be a mix of both ability to discern good ideas and also the ability to find good ideas. Those correlate and overlap but are not the same thing.
• Ryan: “But at the time, there was low-hanging fruit.”
• Almost everything picked will look, in hindsight, like low hanging fruit.
• Dwarkesh asks, if research is so amenable to intelligence, why hasn’t it been faster? Why did we need so much compute?