Harvey 如何在预算内构建研究实验室：应用层公司的后训练实战手册

Next up, we are honored to have Gabe with us. Gabe is co-founder and president of Harvey. Um you were a research scientist at DeepMind ages ago, um and then at Meta I think before you showed your college roommates uh what GPT-3 could do. And this duo uh then became Harvey. Um we're really excited to have you here. I think everybody here in the audience is an application company um thinking about how to start doing their own research, post-training their own models, um and uh create their own labs.

And so I think Harvey has really set the example here, and we're really delighted to have you give a talk on how to how you guys built Harvey Labs.

Awesome.

[applause]

So the alternative title to this talk is building a research lab on a budget. And it's an unfair game competing with the frontier labs if you're an application layer company. There are rich teams, there [snorts] are poor teams, and then there's us in the application layer. The frontier labs have more money, more talent, more compute infrastructure, and data. So how do you compete? By using the frontier ecosystem. When we started Harvey 4 years ago, most of these companies either didn't exist or were just getting started.

And so we either had to build everything ourselves or in most cases focus on something different. Like building our GTM org and a great product. But today using the frontier ecosystem I think you can compete with the frontier labs and build frontier intelligence. This talk is going to be our high-level playbook for doing this. And I'm going to talk about how we build benchmarks and training data, how we work with the Neo Labs to do post training and how we serve these models in production.

To start, you want to build a benchmark. If you don't have a good benchmark, you can't train models, and if you can't train models, you don't need to serve them in production. So, this year we released three of these data sets that we built. We started by building Legal Agent Bench, which is a taxonomy of tasks that associates would do at a large law firm. And these cover multiple practice areas, and they're complex tasks like doing drafting complex fund formation documents, doing case law research, things like that.

We followed this up by building a contracting data set. This teaches This allows us to teach agents to do negotiation like you would in an in-house department. And then the one I'm most excited about that we recently released is a large diligence data set. This is, I think, one of the largest RL environments that's been released. The largest data rooms here are 80 million tokens, and it lets us do research on long context, very complex tasks.

And I think the most interesting thing about these data sets is how we built them. One challenge we always had at Harvey for training models is we can't train on our customers' data. We work with the largest law firms and enterprises, and their legal data is incredibly sensitive. It's privileged, and so you can't put it in generic models. We can't even put it in our models. And so, how do you train models given that?

And the thing that started working really well this year is using domain experts to guide synthetic data generation. And uh Brendan from Reka had a good analogy that the same way that engineers now don't write code, they vibe code and guide these coding models. We're starting to do the same thing. And so, my younger brother is actually a lawyer at Harvey, and he's gotten very good at using the coding models. He's trained our other lawyers to do that, and they can generate incredibly realistic data sets that we can use for training and also evaluating our product.

And once you've done that, synthetic data isn't good enough, but it's a way to get started. And so, we work with companies like Mercor and Snorkel, who let you scale up this process and build larger sets particularly for training. Once you've done that, you need to turn these data sets into efficient RL environments. It gets really expensive as these data sets get larger and evaluation gets very expensive. For example, in our diligence data set, we have over 1,000 unit tests that are grading model outputs using LLM as a judge.

If you use the largest models and you want to do RL rollouts and things like this, it gets very expensive. And so, there's a lot of work, and here's some we did with LangChain, of making these very efficient. And then the last thing we did that I think was a little controversial at the time, was open-sourcing some of these data sets. And the motivation for this was it's very hard to know your data set is good unless a lot of people train on it.

When I used to do research at Google Brain and DeepMind, the best data sets were open, like ImageNet, CIFAR, MNIST, and everyone used them, and you were able to find all of the issues, and we get a ton of pull requests, we get suggestions. Um and then I think increasingly we're having the labs when they report new models benchmark on our data set. And then most importantly, we had Elon retweet it. So, once you've built the benchmark, now you have something to train models against.

And the thing that is exciting now is open-source models are getting competitive. In the past, it wasn't worth doing post training because the models were improving so quickly from pre-training that any post training you did quickly got absorbed by the next pre-trained model. But now with models like Kimmy 3, GLM 5.2, NeMo-Megatron, Inkling, and others, it's possible to take these very strong open-source base models and post train them to levels of frontier intelligence.

Maybe not general frontier intelligence, but if you have a specific task like us, they are competitive. And so, the way we recommend getting started is working with the NeMo labs. They have a bunch of expertise and infrastructure already in place to help you make sure that your training data sets are good. They have recipes. And usually, if you work with them and you're not able to get better results, there's probably something you're doing wrong with your data set, and this is a very good way to bootstrap it.

And so, some of the interesting work we did with these different providers, um Fireworks, we got some very interesting results training GLM 5.1 to use Fable or maybe Opus 4.8 as an advisor model. Um Base 10, we did some interesting work on KB compaction. N gram, who I think is here, we're doing interesting work on enterprise search and firm knowledge. Trajectory, um we worked with them to train NeMo-Megatron models. And Applied Compute, we're doing some interesting work on our Vault product.

And I think one question we got is why work with multiple NeMo labs. Why not just pick one? And for us, as we're scaling the research lab, we have more research projects than we have bandwidth to do internally or just with a single Neo lab. And every Neo lab is taking a different bet. They have different ways they think about research. We have different open source models we want to train. And the more we work with, the more we learn.

And it's getting easier than ever to do post training. So one, working with the Neo labs, we're learning a lot in partnership with them. And then we're doing more and more post training ourselves internally with APIs like Tinker and the infrastructure Fireworks and Base 10 have built. It's never been easier to post train these models and then serve them. And there's increasingly more post training talent that we're hiring and is available.

And inspired by Cursor, the goal of these efforts is for us to build our version of Composer one. How do we package all of the work we've done with synthetic data, scaling it with Mercor, the work with the Neo labs, into a model we can serve alongside the closed source models. Now, once you've post trained a model, you need to be able to serve it in production. And this is non-trivial. So I want to start first by talking about our model serving infrastructure because I think so