Kimi K3开源权重模型发布，2.8万亿参数

This is insane. I cannot believe what I'm seeing here. This is a new AI system, Kimi K3, that can code up a mostly working copy of a full Mac OS operating system. Also, a cute Animal Crossing style game, other kinds of games, and so much more. And what are you seeing what I am seeing? Yes, it is very close to the Frontier systems, some of which are kind of getting banned sometimes, but this one won't. Why? Well, here comes the best part.

This is an open weights model. Yes, we can download and own the weights for free forever, and nobody can take this from us. That is absolutely incredible. Now, wait, wait, wait. It is big. Karoly, do you mean that it's big news? No, I mean it is big. It is absolutely stupendously, humongously big. 2.8 trillion [screaming] parameters. Most of us can't afford to be running this at home. Not as is, but with a little luck, you can try it for free on the web, depending on availability, and take it out for a spin.

Or, if you use the API, it is way, way cheaper than current Frontier models. So, even if you don't ever use it, it will be pushing token prices down. Also, don't forget these huge models are often distilled down to smaller, hopefully similarly capable ones on a regular basis. And somehow, it gets even better. They gave us the secret sauce. So, what is the secret sauce? Dear fellow scholars, this is Two Minute Papers with Dr.

Karoly Zsolnai Feher. One, Kimi Delta Attention. Imagine a meeting where every researcher has to reread everything everyone ever did. Oof. That's not a meeting. That's torture, basically. Instead, this says, "Let's have a carefully updated notebook, read and update only that, and it lets all notes gradually fade a bit. This finally lets the institute handle a very long discussion and contribute meaningfully. Now, wait.

What happens when this information passes through dozens of layers? Well, secret sauce number two, attention residuals. Imagine that every document goes through department one, then department two and three and four only gets the latest version of the document. With attention residuals, department four still gets the latest version of the document, but also a version history as well, and see how the document has changed over time.

So, KDA maintains an correct memory. Attention residuals retrieve useful earlier drafts across layers, and wait until you hear what happens when we combine these two ideas. So, what happens? Well, hold on to your papers, fellow scholars, because it results in a two and a half X improvement in scaling efficiency over Kimi K2. Oof. Wow. Now, this does not mean it is two and a half X cheaper or two and a half X faster. No, it means roughly two and a half times more learning progress out of the same amount of training computation.

That is a huge bump over just one version number. So, what does this enable? Well, all of these incredible things here and something more. Don't forget, with every paper like this, we make all the other open models work better. We are building this together, and you see, we get all of this for free forever. Yes, it is not trivial to run, yet, but I think it will be trivial to run a distilled version of this, hopefully soon.

And don't forget this is the golden age of open science. You can have an AI like this running fully free open weights in an operating system that is fully free, open source, and all of this developed by humans working together across the planet. And these AI systems help doctors, scientists, students learn and do their work all across the world for free. Everyone will get access. Isn't that amazing? What a time to be alive.

I use Lambda to reproduce AI research papers often in minutes. It's also great to train your own models or fine-tune an existing one. Run inference or text to image or video, easy-peasy. Running a deep seek chatbot or agent, super fast, super reliable. Lambda gives you powerful Nvidia GPUs to run your own experiments. I test ideas from the papers I cover, and moments later, results. Love it. Seriously, try it out now at lambda.ai/papers.