后训练：如何将你的品味注入模型，构建持久业务

Okay, up next we have Lynn, uh close close friend and collaborator of Brendan's. Um Lynn, uh actually show of hands, who here who here does post training? Okay, good amount of the room. And who here uses Firework? Okay, good amount of the room, too. So, Lynn, you have you have some friendlies in the audience. Um and so for this next talk, what we're going to do is we're going to focus on post training. Um Lynn, si- similar setup to to Brendan's talk.

We're going to do 15 minutes on how to approach the problem of post training, um and then 15 minutes or so for Q&A. Uh and Lynn, really delighted to have you here. Thank you for joining us.

Thanks for having me. Uh hi everyone, good morning. I'm Lynn, I'm CEO and co-founder of Firework. So, as we bring up the slide, today I'm going to do a little bit uh deep dive on post training and why post training could be very relevant to uh you building your own business. So, first of all, a little bit like prior, I think um this year we have seen So, first of all, I'm a Firework weather specializing intelligence platform.

There are tons and tons of application built on top of us, all the way from startups to digital native and enterprises. So, we get uh um the fun part of my job is we get to see a lot of patterns. Um what are the innovation people are uh building on top of us and uh and how they are what are challenges that are uh they're facing and what are trend um people being developers are um building on top of us. So, uh one of the things, especially in the past 1 year, software development and application development has been somewhat disrupted because it goes fine in the past, um you have good idea, you want to implement it, scaling production, it requires a team of tens of very strong product engineers, PMs working together, multiple quarters to deliver that.

And right now, with one person, a few weeks, uh without understanding how to write a single line of code, you can do that. So, that collapsing of resource required both in terms of timeline and deep expertise is shifting how how competitive the application space is, and it's shifting people from thinking about building on top of off-the-shelf off-the-shelf black box API to build a much deeper mode. So, that they can build a much durable business.

As also you have seen lot of discussion, especially in the past 1 week, between open model and the closed model and all the rallying and support across the open model. The depth of that that alliance is because we believe I mean the industry is much deeper because look at the whole entire industry, there are so many companies, right? There are so many company, all of you are building your own company. Every single company exists for a reason because they focus on solving a unique problem in a special way.

And that means they carry their own judgment, taste, and determination, conviction into that product, and that's why company exists. Today, if you build on top of off-the-shelf API, then you really need to think about how you keep that special taste judgment and unique part forward. And we believe one approach for every company to build a build durable business is to actually bake your judgment, taste, and customer deep understanding into the intelligence you build on top of instead of just a off-the-shelf API.

So, that's kind of a little bit context of what's happening in industry, what we are seeing, and why post training could be very very relevant to you. Okay. So, you probably heard a lot about owning intelligence not rent. And what does owning your own intelligence mean? It actually means many things. So, first of all, it start from data. Intelligence is derivative of data. And obviously the the foundation labs, the all the foundation model we're using are building on top of the public data and the label data that is has solved common tasks, but all of you are solving a specific task.

That's why you you're building a business, you're building a company, and be able to curate production data with high quality and even generate synthetic data to enrich your production data is one step of owning your own intelligence. And then after you have data, you will start to kind of use that data turn into a model, build on top of existing model and own the weights. And there are a collection of techniques you can use to get there.

Those techniques are tailored to solve different kind of problem you could possibly have. And those techniques can also interoperate with each other for you to build a reach your final goal. And then after you have a great model belong to yourself solving your specific problem really well, and then you need to work on serving it. You probably first will do a some AB testing to make sure it really move the needle for your product metrics, and then goes back in in this loop.

Obviously, I don't think you should jump into post training right away. And and there are different phases you will go into. First, prompt everyone start from prompt, use the model as is. It's few shots. If quickly you can use that to test your ideas. And then you use rag to ground um, the usage of AI into your own data and you can start to do a lot of contest engineering from that. Those are from minutes of interaction to hours of interaction.

And then you first progress into hey, I have some data. I want to see how my data is going to reflect and make uh, the model work better with my product. So you will start to do supervised fine-tuning that will take you a few hours to um, hey, um, the the model I want to reflect in personalized taste. It is it is very unique uh, choice of my products. So so therefore you want to start to use preferences uh, information where you collect from user interaction and uh, whether thumbs-up, thumbs-down, a lot of those kind of information and uh, help the model learn your product taste.

Um, and and finally, you want to build a model towards understand carrying your expertise in that domain. Whether that expertise is across uh, legal, finance, healthcare, customer support, recruiting, marketing, sales, you name it. Even in one industry vertical, there are so many subdomains. So all of that is unique and special towards the product you're building. So usually um, you probably heard a lot of reinforcement learning and that is to to actually build towards a specialty.

Um, so this progression is very similar to how we human being learn knowledge over time. Uh, for example, uh, we actually run uh, learn a lot of knowledge by reading uh, reading literature, right? In the literature it will say, hey, what is correct, what is it not correct? So this is a very similar to supervised fine-tuning. Um, and uh, as we grow, we develop our own taste and judgment of how we want to conduct the specific way of we want to approach a problem, and that is preference or DPO.

Um and over course of time, we learn to be a doing a really good job at certain area, whether um hey, I want to be accounting accountant, and I would really know how to kind of um build into the financial financial data, or I want to be a specific set of um I want to be a dentist, and you learn the details of how to operate um with dentistry. So, all this is very similar exactly um to how we human being acquire knowledges.

So, um and also very interesting different techniques are there to solve different kind of problems. Um for example, if the model doesn't know um the fact, and the fact actually is very dynamic, the facts of your data is uh in in your product is very dynamic, and then typically you use rag to solve that problem. Um however, if your model output all the behavior uh or um or the structure is off, then you you give you curate data and do supervised fine-tuning to correct that.

Um if your model's answer, the quality is is personal or kind of is specific to the taste of your product, um and then you use preference and tuning. Or the model is quite weak on the special problem you're trying to solve, then