跳到主内容
@wquguru
精选88Hacker News Best(web_list)技巧与观点

用AI编程Agent在FPGA上实现LLM推理并提速33倍

Vibecoding an inference chip

原文
发到 X
推荐理由

这是一篇极具参考价值的硬核工程复盘,展示了如何用AI Agent高效完成FPGA上的LLM推理部署与性能调优,提供了可复用的优化思路与参数细节。

Ankit Gupta

  • Blog
  • Contact /
  • Github /
  • 博客
  • 联系 /
  • Github /

Vibecoding an inference chip

通过氛围编码(Vibecoding)设计推理芯片

Or rather, I vibe-coded an FPGA into running an LLM. And it… just worked.

或者更准确地说,我通过氛围编码让 FPGA 运行大语言模型。而且……它居然真的跑起来了。

Friday. November 20, 2026

2026年11月20日,星期五

One of the more entertaining things about programming recently is that coding agents let me work in entirely new domains by giving me access to new programming languages and paradigms. Lately, that has meant hacking on hardware, now that writing low-level software is much easier. So, in this blog post, I’m going to try to recount the experience of vibe coding an FPGA, going through a series of “oh shit” moments several times, and giving myself a new appreciation for what’s possible if you have a large token budget (thanks YC) and interesting problems to point the models at.

近期编程中一件颇为有趣的事情是,编码代理(coding agents)让我能够通过接触新的编程语言和范式,涉足全新的领域。最近这意味着我在进行硬件方面的黑客式开发,因为现在编写底层软件变得容易多了。因此,在这篇博客文章中,我将尝试回顾通过氛围编码(vibe coding)设计 FPGA 的经历,多次经历“卧槽”时刻,并让我对拥有大量 token 预算(感谢 YC)以及将模型指向有趣问题时所能实现的可能性有了全新的认识。

I overheard some founders recently talk about how FPGAs work. I’ll admit I was relatively naive about them. I’d never really experimented with them in my startup or in college, so I knew little about them before. The basic idea I was familiar with: you can program a circuit into a reprogrammable chip. So rather than having a general purpose computing architecture (like a CPU) you can have a reprogrammable one, i.e. the configuration loaded onto the chip determines how its logic and routing are connected, and that (in theory) enables performance advantages. I also know that it took a mountain of C++ to write and involved manually analyzing waveforms for simulation and verification. I had never done this before though and it seemed cumbersome.

我最近偶然听到一些创始人谈论 FPGA 的工作原理。我得承认我对它们相对天真。无论是在我的创业公司还是大学期间,我都从未真正实验过它们,所以在此之前我对它们知之甚少。我所熟悉的基本概念是:你可以将电路编程到一个可重编程的芯片中。因此,你不必拥有通用计算架构(如 CPU),而是可以拥有一个可重编程的架构,即加载到芯片上的配置决定了其逻辑和路由的连接方式,而这(理论上)带来了性能优势。我也知道,编写这类代码需要大量的 C++,并且涉及手动分析波形以进行仿真和验证。但我以前从未做过这些,而且看起来相当繁琐。

That’s about what I knew going into this.

这就是我当时所了解的全部。

Naturally, my ML brain thought “could we use these for AI inference”. I’m not the first to think this — there are a few repos attempting to port ML models to FPGAs on Github. But I hadn’t actually seen a writeup on whether coding agents are good at writing the low level C++ and Verilog code to program them and verifying the outputs so that they actually run accurately. (Spoiler alert, they are incredible at programming them.)

自然而然地,我的机器学习思维想到了“我们能否将这些用于 AI 推理”。我不是第一个想到这一点的人——GitHub 上已经有几个仓库试图将 ML 模型移植到 FPGA 上。但我实际上还没有看到过关于编码代理是否擅长编写用于编程它们的底层 C++ 和 Verilog 代码,并验证输出以确保其实际准确运行的文章。(剧透:它们在编程方面表现得极其出色。)

So I decided I needed to get my hands on one. AWS has F2 instances with FPGA hardware, and that seemed easier than going and buying physical hardware, so I asked Codex to spin up a reservation and got cooking. It had a relatively limited memory of 16 GB of FPGA HBM. Around the same time, I was working with the DeepGrove founders who were discussing their new ternary weight architecture that is well suited for memory limited cases. It’s SOTA on Mac hardware with actual agentic tool calling, so I figured why not, let’s see if I can get this working on an FPGA and fast enough to be useful.

于是我决定我要搞一台。AWS 提供带有 FPGA 硬件的 F2 实例,这看起来比购买实体硬件更容易,所以我让 Codex 启动一个预留实例,然后开始干活。它只有相对有限的 16 GB FPGA HBM 内存。大约在同一时间,我正在与 DeepGrove 的创始人合作,他们正在讨论一种非常适合内存受限场景的新型三元权重架构。它在 Mac 硬件上实现了 SOTA(当前最佳)水平的实际智能体工具调用能力,所以我想,为什么不试试呢?让我们看看能否在 FPGA 上让它跑起来,并且速度快到足以实用。

What followed was a cloud agent running in Conductor for several days with me mostly just encouraging it periodically while it both (1) got a model fully running and (2) increased its speed by 33× from the first full-model bit-exact baseline to get it to a real-time relevant speed. It was one of the most mind blowing autoresearch loops I’ve run, ultimately writing roughly 3M lines across the experiment history and parsing 30M lines of output files, burning only $10,000 in Claude and OpenAI tokens to run hundreds of experiments. Most of the code is C++ and some Verilog. It’s been probably a decade since the last time I wrote any serious amount of C++ and I’ve never written Verilog. I honestly don’t know the first thing about Verilog. Most of my prompts were not particularly clever: they consisted mostly of me saying things like “you’re a superintelligent model, time to be more creative” and “let’s go for 2x faster, you got this”. Sometimes I brought some ML knowhow like “hey it’s time to consider speculative decoding” or “why is that running on CPU, you should port it”. But for the most part, it was just vibes and I’m pretty sure that with enough time the model would have figured out those things. Also, it seemed that both Astra and Opus (new Fable wasn’t out yet) were pretty capable at this and I could use them interchangeably. I’m sure someone has a good benchmark for which is smarter but both passed my smart-enough threshold.

接下来是一个在 Conductor 中运行了数天的云智能体,我大部分时间只是定期鼓励它,而它同时完成了两件事:(1) 让模型完全运行起来;(2) 将速度从第一个全模型位精确基线提高了 33 倍,使其达到实时相关的速度。这是我运行过的最令人惊叹的自动研究循环之一,最终在实验历史中编写了约 300 万行代码,并解析了 3000 万行的输出文件,仅花费了 1 万美元的 Claude 和 OpenAI token 来运行数百次实验。大部分代码是 C++,还有一些 Verilog。距离我上次编写大量 C++ 代码可能已经十年了,而且我从未写过 Verilog。说实话,我对 Verilog 一无所知。我的大多数提示并不特别巧妙:主要由我说一些类似“你是一个超级智能模型,是时候发挥创造力了”和“我们要追求快两倍的速度,你能做到”这样的话组成。有时我会带入一些机器学习知识,比如“嘿,是时候考虑推测性解码了”或“为什么那个还在 CPU 上运行?你应该把它移植过去”。但总的来说,主要靠的是感觉(vibes),我很确定只要给足够的时间,模型自己就能搞定那些事情。此外,Astra 和 Opus(新的 Fable 当时还没发布)似乎都相当能干,我可以互换使用它们。我相信有人有衡量哪个更聪明的基准测试,但它们都达到了我认为够聪明的门槛。

Summary of my most impactful prompts. Most of that first third is just getting AWS set up and approving quota for f2 instances.

我最具影响力的提示总结。其中前三分之一的大部分工作只是在设置 AWS 并为 f2 实例批准配额。

From “it works” to 213 tokens per second

从“能运行”到每秒 213 个 token

Getting faster, one experiment at a time

逐步提速,一次实验一步

Measured benchmark tokens/s · select a step to see what changed

测得的基准测试 tokens/s · 选择某个步骤以查看变更内容

Baseline 6.43 Host overhead 8.07 Burst attention 14.02 Parallel router 20.53 Glue engine 30.24 New pipeline 49.21 Prompt speculation 51.25 Router relay 58.43 Head shadow 94.44 Draft model 105.63 Memory ports 108.81 Attention overlap 134.99 Weight packing 171.3 Re-speculation 184.19 Parallel head loader 213.13

基线 6.43 主机开销 8.07 突发注意力 14.02 并行路由器 20.53 胶水引擎 30.24 新流水线 49.21 提示推测 51.25 路由器中继 58.43 头部阴影 94.44 草稿模型 105.63 内存端口 108.81 注意力重叠 134.99 权重打包 171.3 重新推测 184.19 并行头部加载器 213.13

Step 14 · 33.13× baseline

第 14 步 · 33.13× 基线

A head loader that reads both of its memory banks at once (+17.3%, measured on and off on one image), a runtime expert-prefetch policy, and the router-job timing fix; run one clock step lower (225/425 MHz) after a soak test found random faults at 231.25.

一个同时读取其两个存储库的头部加载器(+17.3%,在一张图像上测量开启和关闭状态)、运行时专家预取策略以及路由器作业时序修复;经过浸泡测试发现 231.25 MHz 下存在随机故障后,运行时钟步长降低一级(225/425 MHz)。

The chart follows the measured optimization sequence: 6 → 213 tokens/s, or 33× the baseline.

该图表遵循测量的优化顺序:6 → 213 tokens/s,即基线的 33 倍。

How much did the human advice help?

人类的建议有多大帮助?

At one point, around when it got to 10 TPS, I messaged the DeepGrove founders about this, and they replied surprised that it was working so easily, saying they had tried this a few months back and the models were not quite good enough to be particularly effective. They sent me a prompt that summarizes some of their lessons and I copy-pasted it into my session to see if it’d matter. It was essentially the most significant piece of actual guidance I added to this session. The prompt is reproduced below.

有一次,当它达到约 10 TPS 时,我联系了 DeepGrove 的创始人并告知了此事,他们惊讶于它能如此轻松地工作,并表示几个月前他们尝试过类似方法,但模型效果还不够好,无法产生特别显著的效果。他们发给我一个总结了一些经验教训的提示词,我将其复制粘贴到我的会话中,看看是否会有所影响。这基本上是我在这个会话中添加的最重要的实际指导。提示词如下所示。

Read the full FPGA design prompt

阅读完整的 FPGA 设计提示词

here is what someone else did for a dense bs=1 decode optimized fpga implementation for a dense qwen style model with ternary weights. you should take this with a grain of salt and pick things that you believe are acutally useful to add to your plan. Please also dont do things if you dont think is useful that take a lot of times like runnning really long test of a cycle accurate simulation etc or like physical design since those can take like hours. here are some ideas please pick and choose use this as context and not somehting that you shoudl be strictly guided by somethings that could help. prioritiize speed of like getting this up and running though..

以下是其他人针对具有三元权重的密集 Qwen 风格模型,为密集 bs=1 解码优化的 FPGA 实现所做的工作。请对此持保留态度,并挑选你认为对添加到计划中真正有用的内容。另外,如果你认为某些耗时很长的操作(如运行非常长的周期精确仿真测试等)或物理设计没有用处,请不要执行,因为这些可能耗时数小时。以下是一些想法,请酌情选用,将此作为上下文参考,而非严格遵循的指导。不过要优先考虑速度,尽快让系统运行起来……

MATMUL (TERNARY ENGINE)

矩阵乘法(三元引擎)

  • No multipliers. A 2-bit weight drives a 3:1 mux that picks x, ~x, or 0. The unused 4th code raises an error flag, so corrupt weights get caught.
  • Negation is ~x + 1. Lanes flag negatives, and the row reducer adds the +1s once per row instead of once per lane.
  • Activations are grouped integers: INT12, or INT16 where we needed more range (MLP down projection). Reduce as integers, apply FP32 group scales, accumulate in FP32.
  • The pre-projection RMSNorm is deferred: quantize the activation first, apply the normalization scale after accumulation. It’s cheaper and numerically equivalent.
  • 8 cores in two clusters. Projections that are independent (gate/up) run at the same time on separate core groups.
  • Result: about half the DSPs of an INT8/INT16 design at the same rate, traded for more LUTs.
  • 无乘法器。2 位权重驱动一个 3:1 多路复用器,选择 x、~x 或 0。未使用的第 4 个代码会触发错误标志,从而捕获损坏的权重。
  • 取反运算是 ~x + 1。通道标记负数,行归约器每行只加一次 +1,而不是每个通道加一次。
  • 激活值被分组为整数:INT12,或在需要更大范围时使用 INT16(MLP 下投影)。以整数形式进行归约,应用 FP32 组缩放因子,并在 FP32 中累加。
  • 预投影 RMSNorm 被推迟:先对激活值进行量化,在累加后再应用归一化缩放。这种方式成本更低且数值等价。
  • 两个集群共 8 个核心。相互独立的投影(gate/up)在不同的核心组上并行运行。
  • 结果:在相同速率下,INT8/INT16 设计的 DSP 使用量减半,代价是占用更多的 LUT。

MEMORY

内存

  • Weights are packed at 2 bits, 8x more per transfer than 16-bit. At batch 1, weight fetch costs far more energy and time than the math.
  • Weights stream from HBM straight into the compute lanes and skip the on-chip buffers (they’re used once per token). Activations are cached next to each core and reused across rows.
  • HBM routing is one chain per memory channel instead of a big crossbar. Lay out data so all 32 channels stay busy.
  • On-chip buffers are typed slots with an explicit lifecycle (open -> write -> commit -> read -> release). Each slot has a side memory for scales/exponents. The buffer checks format, range and producer, which catches scheduling bugs in hardware.
  • The KV cache is MXINT8: 8-bit values with one shared exponent per 32-channel block. That’s small enough to stream, with accuracy held.
  • 权重以 2 位打包存储,每次传输的数据量是 16 位的 8 倍。在 batch size 为 1 时,权重读取所消耗的能量和时间远超计算本身。
  • 权重从 HBM 直接流入计算通道,跳过片上缓冲区(每个 token 仅使用一次)。激活值缓存在各核心附近,并在行之间复用。
  • HBM 路由采用每条内存通道一个链式结构,而非大型交叉开关。通过合理布局数据,确保所有 32 个通道保持忙碌。
  • 片上缓冲区是带有明确生命周期(打开 -> 写入 -> 提交 -> 读取 -> 释放)的类型化槽位。每个槽位都配有侧边内存用于存储缩放因子和指数。缓冲区会检查格式、范围和生产者,从而在硬件层面捕获调度错误。
  • KV 缓存采用 MXINT8 格式:8 位值,每 32 通道块共享一个指数。这种格式足够小巧以便流式传输,同时保持精度。

更进一步:量化金融体系

看懂新闻只是起点——沿量化金融路径,把它变成能交付的工程能力

进入量化体系 →

相似阅读

关联信息,但可能不是同一事件