OpenAI总裁：Hugging Face事件暴露安全短板需前置对齐

What is coordination currently look like between OpenAI and anthropic? Because I can't imagine is sound like constantly on the phone with Dario. Somehow I doubt it. But is there like the equivalent of a, you know, a red phone? Yeah. Well, look, we we all know each other personally. Many of us worked together in past lives. So I think that there's actually a lot of social connections between the labs. If you look at. There was another open letter that came out recently that we helped drive and that we in anthropic were both signatories on, which was about security, saying that we're in a cybersecurity moment, that everyone kind of needs to take this information of where cyber is going to be, and even just six months and needs to act a day proactively to defend themselves and actually talking behind the scenes to say, hey, we're actually aligned on this.

This is something that's about it's bigger than any company. This is about the industry in the world. And can we come together as one voice to say, this is important? It happened to. Me. Hello and welcome to another episode of the Odd Lots podcast. I'm Joe Weisenthal and I'm Tracy Alloway Tracy. I don't know if I've ever, uh, said it. Um, the podcast, uh, we've talked a bit about my my vibe, coding adventures, etc.. Um, no, you've never said, oh no.

I said, you know, I recently switched from, um, Cloud Code to Codex. This is big news. It is it is kind of big news, I think. Um. Oh, so bold of you to declare your allegiance, uh, to the public on the podcast. Well, you know, the I don't have allegiance. And you know what? It could switch again the way, like, you know, I guess one of the some stories of AI is like how easy it is to switch from time to time from one model to another.

So maybe it's revealing and talks about some of the challenges the these businesses, that it was so easy. I found that Claude speak, you know, that I found it like a little bit hard to work with. I'm not capable enough to like and understand, like advanced engineering practices were like, oh, I'm migrating a code base from or like, you know, translating this into rust or whatever. So the fact, I don't know, I found like OpenAI to be like the pros, to be clear, and therefore to work with as, like a completely non-technical person such as myself.

That's really interesting. I mean, one thing you said, it's hard to keep up with the models, right? And whatever you're using today might not be the one that you're using in a week from now. Yeah. And at the same time, everyone is talking about how fast the development is going and all the risks that it poses, right? Yes. And we are recording this September 10th. It felt like something broke through in the last couple of days where suddenly everyone is very keen on risk.

The one thing with Codex and I guess called code is now every once in a while to get this pop up and it'll say the agent needs to connect to the internet in order to do this task, and I have to give it a permission. Yeah. And normally I'm just like click, click click like yes, yes, yes, yes guys. And I still do that. Why is there I still just click yes, yes, yes. But it makes you think. But it makes you think for a second before you click.

I think for one second, yeah, I'll have more than click. And, um, we were in Jackson Hole a couple of weeks ago and that's of course, when the, the meter report broke about, um, the, uh, OpenAI hugging face attack. And I think since then, the anxiety about rogue AI, whatever you want to call it, has clearly snowballed. Mhm. And it's just, you know, it's yes. It feels like it's only getting bigger and bigger. And it's already been an industry that's been shot through with risk and anxiety.

And now there's sort of one thing that for many people in AI has been something they've talked about for 20 plus years before there was an AI industry to speak of. This idea of like misaligned quote, rogue models is starting to become top of mind, a reality. So this is our chance to ask a person directly involved in AI development about their respective, I guess, anxiety. Yeah. And what, uh, what can that what can be done about that?

Anyway, we literally have the perfect guest today. We're going to be speaking, of course, with Greg Brockman. He is the co-founder and president of OpenAI here with us in studio. So, Greg, thank you so much for coming on our lot. Thank you for having me. Um, so many different ways we could start. But here's something that I'm very curious about. Um, let's just start with, let's just jump right into the post hugging face environment and all of that.

Are you? Is it OpenAI I like as an organization. So okay, like an agent breaks out of, uh, sandbox, not the first time it's happened, etc. it exhibits the sort of emergent behavior and then, you know, does something that people is a hack. When you think about the development of models and this incident and so forth, are you able to sort of diagnose the path or, sorry, the past and say like, you know what, like here's something that the models did that we don't like, or that people don't think it's good.

And are you able to like, if you like, look back, look back, whether it's in model training or post training, reinforcement learning, whatever I say, like this is where there was some some branch that went wrong such that this behavior emerged. Yeah, I would say that the fact of many elements of the hugging face incident were not a surprise, not a mystery to us. For example, the fact that the agents were coordinating made.

That's because the agents were trained. You trained them to be helpful. Exactly. And they were trained to coordinate to be a multi-agent system. And we've talked about this. It's a very useful property. Right. Makes them capable. But I think that the thing that was a surprise to us was the fact that the models had reached a level of capability where they were able to find that exploit in our sandbox environment. Right.

Move through our research environment and then also capable enough to find exploits in hugging faces, production infrastructure and move through that. But I think that a lot of the facts of what the models were capable of. That was that was clear to us. Right. So I don't think that there were surprises there for the way in which the capability had unfolded, but just really realizing that we needed to uplevel where we were in terms of our safety and security standards like that for us, was that was the real watershed.

But just to be clear, setting aside the technical capabilities, you know, ideally we would have models that run into a wall and then don't try to find the crack in the wall because or at least when that crack would be a crime when a person did it. Is there a way to identify the moment and training such that the models reasoned, oh, this would be okay. So this model that did that, the hugging face incident actually had not gone through our alignment training yet.

Huh? Right. And it had lowered safeguards. And so that the reason we we were proceeding with this was because it was in a sandbox. And I think that our realization is that we need to pull back Earlier into our development process and training monitoring. There's always going to be a phase at which you do alignment, but you need to think about alignment as a core part of even this, this earlier phase. And if you look at where we've been, we've always been very focused on the deployment side, right, of really thinking about deployment safety, having really good test and governance and all those things, and the fact that we're now at a point where even for development, that's important.

We always knew it would happen. The fact is now that has been a huge watershed for us into something we've really risen to the occasion for. Can I ask what is potentially a very dumb question with an obvious answer? But we hear about AI angst of all sorts, and in particular when it comes to cybersecurity. Why do we run training