Hot Chips 2026：OpenAI 自研芯片 Jalapeño

By far the biggest announcement at the 37th Hot Chips conference was OpenAI’s stunning progress on their own chip, less than a year after the Broadcom announcement… and that it isn’t an ASIC; but a full on Blackwell-beating alternative.

The key metric now is shifting to performance per watt, and Jalapeno delivers:

The full Hot Chips presentation is not yet out but various takes are below. For a fuller breakdown, watch along with the rest of OpenAI:

AI News for 8/24/2026-8/25/2026. We checked 12 subreddits, 544 Twitters and no further Discords. AINews’ website lets you search all past issues. As a reminder, AINews is now a section of Latent Space. You can opt in/out of email frequencies!

AI Twitter Recap

OpenAI’s Jalapeño Inference Chip and the Shift in the Inference Stack

• Jalapeño’s published numbers are the day’s biggest technical story: OpenAI released first benchmark details for its custom inference chip Jalapeño, claiming materially better efficiency and latency than NVIDIA GB200/GB300 systems on real model workloads. In OpenAI’s tests, Jalapeño delivered 1.5–1.9× more work per watt at peak throughput and 1.7–3.6× lower end-to-end latency, with 2.1–4.1× higher performance for highly interactive workloads; the chip is rated at 700W but reportedly stayed at or below 550W on the tested runs. OpenAI says deployment into its own infrastructure begins by year-end, with Gen 2 already deep in development and Gen 3 underway (OpenAI announcement, deployment roadmap, Sam Altman).
• Why engineers care: the claim is not just raw perf, but a more balanced inference architecture that reduces the usual throughput/latency tradeoff. Multiple technical reactions highlighted that some comparison points are especially notable because Jalapeño reportedly performed well even without tricks like aggressive prefill/decode disaggregation or speculative decoding in some setups, while beating systems that did use them (gdb, kimmonismus summary, eliebakouch analysis, You Jiacheng). SemiAnalysis framed it as unusually strong for a first-generation ASIC and compared it directly against Blackwell and Rubin-class systems (SemiAnalysis, dylan522p).
• A second-order story is model-assisted systems optimization: OpenAI’s post also said GPT-Astra + Codex helped write and optimize low-level kernels, bringing three previously unplanned open-weight models to high performance on Jalapeño in about two months; for selected attention and MoE blocks, these implementations reportedly ran 1.5–1.8× faster than existing human-expert-written code (kimmonismus, eliebakouch). That is a meaningful signal that compiler/kernel work is increasingly being folded into the model improvement loop, not just application-layer coding.
• Broader infra implication: several posts tie Jalapeño to a larger industry transition in which frontier labs may no longer be strictly downstream of NVIDIA for inference economics, even if packaging and foundry capacity remain a hard bottleneck (Liam Fedus, teortaxesTex reaction, LearnOpenCV caveat on TSMC/CoWoS capacity).

Agent Harnesses, Memory Systems, and Eval Engineering Becoming First-Class

• Harness quality is increasingly as important as model choice: several papers and launches converged on the same theme: agent performance depends heavily on the surrounding system. A new Microsoft-led paper on AutoSaddler treats the harness as code and patches prompts, tool configs, and control logic offline using failure traces, reporting gains of +9.0 on GAIA2, +9.6 on SWE-Bench Pro, and +10.0 on Terminal-Bench 2.0 over base harnesses (paper summary). In parallel, another paper quantified harness variance directly, finding that swapping harnesses could move scores far more than swapping models, with model-pair rankings flipping across scaffolds; the proposed fix is a structured Harness Card disclosure standard (analysis, “There Is No Neutral Harness”).
• Long-horizon software engineering remains very unsolved: SWE Refactor Bench measures whole-repository migration tasks like C→Rust, Maven→Gradle, and POSIX→WebAssembly across real projects including SQLite, zlib, and libsodium. Across 520 runs, only 28 survived all three stages, for a 5.4% survival rate, and 13/20 tasks were solved by nobody (EinsiaAI). This is a useful corrective to strong bug-fix numbers on more local coding benchmarks.
• Memory systems are being redesigned as programmable state, not compressed chat history: one Alibaba paper summarized by DAIR backs agent sessions with an append-only event log plus a persistent Python kernel, binding tool outputs and derived state to typed variables instead of continually serializing them into prompts. Reported results include 94.8% on LongMemEval_S, 73.1% on BEAM_10M (+5.1 over the previous best published memory system), and 86.7% on LOCA_256K with Qwen3.8-Max (summary). Related work on Knowledge Triage showed that naive context compaction destroys exact-rule retention; after five rounds of compaction, one setup preserved only 10% of safety rules, while type-aware retention policies preserved 2–4× more (summary).
• Practical eval-engineering is moving from ad hoc to productized workflows: LangChain/partners shared a concrete loop for turning traces and human feedback into task specs, synthetic environments, and evals that can be used to measure and post-train agents over time (Vtrivedy10, hwchase17). LangSmith Engine also shipped >2× better performance on key internal benchmarks with better issue detection/clustering, SaaS and self-hosted support, Slack/Linear integrations, and cost-tiered analysis modes (LangChain).

Local-First Agents, On-Device Inference, and the New Personal Compute Stack

• Perplexity’s Portable Computer is the clearest local-agent product launch of the day: Perplexity launched Portable Computer on NVIDIA DGX Spark, positioning it as a fully local version of Perplexity Computer where the orchestrator LLM, subagent LLM, and agent harness all run on local hardware with no cloud dependency (Perplexity launch, model details, NVIDIA, Arav Srinivas). The initial local stack uses a post-trained PPLX 27B with Qwen 3.8 27B also available; Nemotron 3.5 Lightning support is coming.
• The deeper trend is persistent, always-on local agents: Srinivas explicitly sketched a future of background processes that continuously ingest context from connectors, perform multi-hop reasoning in a perpetual loop, and run on your own hardware (Arav Srinivas). Community reactions were split between excitement about privacy/control and skepticism that “local-first” should mean a $5k DGX Spark rather than commodity consumer devices (theo critique, theo follow-up).
• Apple/macOS local AI tooling is also maturing: exo said Apple featured it on new M5 Ultra Mac Studio and M6/M5 Pro Mac Mini pages, emphasizing low-latency RDMA over Thunderbolt 5 to cluster Macs and run models like Kimi K3 and GLM-5.3 at API-like speeds, with 4× M5 Ultra scaling to about 4.8 TB/s aggregate memory bandwidth (exo). Related posts pointed to Apple’s faster PCIe storage and ANE-based vision pipelines as making small local clusters and mixed CPU/ANE/GPU inference more practical (anemll, onirenaud).
• Tooling continues to fill in around local runtimes: Ollama v0.33 added one-toggle integration to let Claude Desktop use Ollama as a third-party gateway for cloud and local models (Ollama); OpenCode v2 was shown running inside a Cloudflare Durable Object, illustrating how small agent runtimes are becoming embeddable in edge environments (fayazara).

Models, Retrieval, and Search Infrastructure