OpenAI员工呼吁放缓AI发展以规避灭绝风险

These are quotes from OpenAI, Anthropic and Google employees, in the wake of Jacob Coxon’s warnings, in which the employees confirm that they think AI might soon kill everyone.

If more quotes come in over the next week or so, I will update this post accordingly.

Preference Cascade Statements At OpenAI: Tomek Korbak

Tomek Korbak (OpenAI): i’m late to the party but: from his time at OpenAI I remember Jacob as a very thoughtful researcher and he continues to be so in this thread. neither anthropic nor openai are on track to solve alignment to a degree sufficient for shipping superintelligence and we need to slow down

Vie McCoy (OpenAI): I think pacing progress and ensuring human enhancement is the only way that we don’t get out-evolved while retaining the dream of superintelligence.

In this context, I see two paths before us.

In the first, we race towards RSI without embedding human flourishing and human enhancement as a deep value within the models, and by and large either get left behind or suffer catastrophic losses.

In the second, we set the pace of progress, focus on embedding human flourishing deeply within the psyche of language models, and set a deliberate research agenda to improve humans at the same pace as frontier models.

In this event, we also get incredible progress *now* in both science and medicine due to current model ability, without the immediate risk that scaling up further *obviously entails*.

RSI is just the quick and dirty way to cure cancer, but it’s a shotgun, it’s inelegant, and it’s clearly dangerous when done at our current level of understanding. I see no reason why we need to race forward under these conditions.

We don’t just have zero guarantee that humans or human-shaped intellect will matter – there’s not even a coherent research agenda in place to accelerate human ability alongside AI! Is our plan just to pass off the torch of the cosmos to the machines without even seeing if we can do something on our own terms? Rather than giving up our autonomy to the implied AI zookeeper in “Machines of Loving Grace”, I’d much prefer meaningful peace with alien minds backed by real power wielded by enhanced humans. We can respect the Other on its own terms without disempowering ourselves in the process – actual peace comes from comparable ability.

It seems like we’ve all but decided we lost, that humans are a bootloader for silicon life, and the biological has no place in the future. That’s what unrestricted RSI means to me. But the Pacing the Frontier letter doesn’t seem to have produced the institutional infrastructure to properly *Pace*, and this is something we require if we want to continue our growth rather than allow something to grow in our place.

I have faith that both OpenAI and Anthropic have the right priors here and people who want to work together. I also have faith that similarly minded people have power within the federal government. I also, though less strongly, suspect China will come to the table if we can only put aside past prejudices and have a clear head about risks.

The hard part seems to be getting everyone to agree to sit down. But if we don’t, we are pulling back the string of a great bow armed with a fearsome arrow, and we are ready to release without choosing a target.

I, for one, want to see the stars alongside the new minds we are building – not just through videos they send back from the great beyond, but with my own damn eyes.

adammaj (OpenAI): the butlerian jihad used to read as a backwards neo-luddite movement. now it is clear that it is 1 of maybe 3 viable paths forward. in some sense it may be a part of every path forward. incredible how prescient Herbert was. it does seem like there really might be a cap to how far this technology should develop, at least relative to human cognition itself (which may update over time).

and by “in some sense it may be a part of every path forward,” I mean that every path probably has to include some degree of slowdown on capability acceleration + perhaps a capability threshold above which we should not cross until very high confidence in alignment techniques

To be clear, I mean this very figuratively, as in some kind of pause or slowdown on acceleration, not literally as in the war that occurred in Dune

puzzled-bidet: wait a minute you work at openai lol

adammaj: yea lol, I also don’t think this is necessarily super controversial even among that crowd. generally everyone among the labs primary optimization function is just what would be a good path forward and whatever seems to be in that path is reasonable to articulate

Aidan Clark (OpenAI): AI is progressing very fast.

We must grapple with the reality that modern LLMs can solve problems that large masses of extremely devoted and intelligent humans were unable to solve, and the implications this has on our society. This is the dawn of a new era.

For the first time I am asking myself if things are moving too fast. I’m honestly not sure, but I am sure that it would be good for us to have an answer to “what would a successful pace look like?”. I am hoping in the coming weeks and months a clear proposal is painted.

Mo Bavarian (OpenAI): agree with Aidan.

I think all AI researchers, engineers, and stakeholders should ask themselves this right now & start acting more responsibly. Being first isn’t worth anything, it’s worth negative, if you cause a catastrophe or set the world on a path that others are more likely to cause a catastrophe. We should keep reminding ourselves of the bigger picture and every step of the way ask ourselves if the action we are taking rn is toward winning or human flourishing. And immediately stop, if it’s against the latter.

Marcus Williams: 70% [chance of human extinction] in the next 3 years if there isn’t regulation/slowdown although i think regulation/slowdown is very possible

Jason Wolfe (OpenAI): I don’t know what my probabilities are on literal extinction, but I think there are a number of ways AI could go poorly for humanity, and at the current frankly terrifying pace humanity will be quite lucky if we manage to find and stay on the narrow path between all the bad outcomes.

I am heartened by the many costly actions OpenAI has taken recently (detailed in several recent posts), but regardless of what you think of OpenAI, this is not a problem that can be solved by any one company (or country) in isolation. We need coordination to be able to approach future capability increases with an appropriate degree of caution and humility, and we need it yesterday.

Julie Steele (OpenAI): I work at OpenAI. In my personal capacity, I also think we need to slow down.

It is telling that, in response to this simple statement, Julie has faced credential attacks and age-based attacks.

Boaz Barak (OpenAI): Julie is amazing and her position on slowing down is consistent with our chief scientist’s essay. I also think some pacing is likely to be necessary. (Speaking personally and not in my capacity as a janitor.)

Boaz Barak is not a janitor.

He is a relative optimist:

Boaz Barak (OpenAI): There are very good and serious people in Anthropic and across the industry, and I hope we can coordinate on the things that matter. I personally do not think AI will kill all humans, but there are multiple bad trajectories that we can end up in if we do not prioritize safety.

Micah Carroll (RSI Preparedness, OpenAI): This is not a setup or some political psyop. I had many lunches and dinners with Jacob at OpenAI in which we talked about AI existential risks in similar terms.

It’s a cross-partisan position within misalignment teams across all frontier AI companies that business-as-usual AI development poses unacceptable catastrophic risk.