Supabase 开源 Evals：真实任务评测 AI 编码智能体

Supabase has open sourced Supabase Evals, its benchmark and framework for testing how well AI agents build using Supabase. It runs coding agents including Claude Code, Codex, and OpenCode against real tasks, such as building a schema, debugging a failed Edge Function, or fixing a broken RLS policy, then scores the result. It powers the public leaderboard at supabase.com/evals and an internal regression suite monitored daily. Is it deployable? Yes, today. supabase/evals is public under Apache-2.0 and runs locally via pnpm. Industries: Developer tooling, cloud infrastructure, data platforms, and regulated backends in fintech or healthcare, where an agent writing a wrong RLS policy is a security incident. Applications: Regression-testing docs and skill edits, gating SDK releases, and comparing agent harnesses head to head. Constraints: Local-stack runs need a Docker daemon, provider API keys, and ports 54321–54329 free. How the harness works Supabase defined three dimensions: products (database, auth, storage, edge-functions, realtime, cron, queues, vectors, data-api), topics (RLS, security, migrations, SQL, SDK, observability, self-hosting, tests, declarative-schema), and stages (build, deploy, investigate, resolve). It then picked the smallest scenario set touching each dimension once, grounded in support tickets, bug reports, and GitHub issues. Scenarios split into two suites. Benchmark scenarios cover breadth and are published. Regression scenarios cover known failure modes, refresh daily, and do not move published scores. Every scenario runs against a real environment. The framework boots a hosted-like stack and a local CLI project in containers, so agents call the actual MCP server and CLI. A platform-lite runtime exposes a Management API-compatible surface backed by @supabase/lite. Scoring combines deterministic checks with LLM-as-a-judge. Agents get one retry before grading. Each eval directory holds PROMPT.md (task plus frontmatter), EVAL.ts (the scorer), and optional remote/ and local/ starting states. Shipping a local/ workspace, or declaring interface: cli, boots a Docker sandbox with the real CLI installed. Run the pipeline</button> </div> <!-- ANATOMY --> <div class="pane" id="p2"> <div class="hint">Every eval lives at <b>evals/&lt;id&gt;/</b>. Click a file to see what it holds.</div> <div class="tree" id="tree"></div> <div class="detail" id="det2"></div> </div> <!-- RUNTIMES --> <div class="pane" id="p3"> <div class="hint">The harness picks a runtime <b>automatically</b>, per eval. Toggle to compare.</div> <div class="tog"> <button class="tg on" data-r="0">Tools evals</button> <button class="tg" data-r="1">Local-stack evals</button> </div> <div class="lanes" id="lanes"></div> <div class="note" id="rnote"></div> </div> <!-- FINDINGS --> <div class="pane" id="p4"> <div class="hint">Published <b>Build stage</b> pass rates. Toggle the Supabase agent skill on and off.</div> <div class="tog"> <button class="tg" data-s="0">No skill loaded</button> <button class="tg on" data-s="1">Skill loaded</button> </div> <div class="rows" id="rows"></div> <div class="note"> <b>Also measured:</b> Codex / GPT-5.6 reads about 8 docs pages per scenario, versus roughly 2 for Claude Code, which checks the docs in under 40% of scenarios even with skills loaded. Rewriting the Postgres best-practices skill description lifted its activation from about 1 in 10 sessions to 60%.<br><br> Figures are a snapshot from Supabase's launch post (31 Jul 2026). Results move as models change — check the live page. </div> </div> <div class="ft"> <span>Source: <a href="https://supabase.com/blog/introducing-supabase-evals" target="_blank" rel="noopener">Supabase blog</a> · <a href="https://github.com/supabase/evals" target="_blank" rel="noopener">supabase/evals</a> · Apache-2.0</span> <span><b>Marktechpost</b></span> </div> </div> <script> var STAGES=[ {i:"<img src="https://s.w.org/images/core/emoji/17.0.2/72x72/1f4dd.png" alt="📝" class="wp-smiley" style="height: 1em; max-height: 1em;" />",l:"Scenario",h:"1 · A real scenario",t:"Each eval is grounded in a real problem — a support ticket, bug report, or GitHub issue. <code>PROMPT.md</code> carries the task the agent sees plus frontmatter tagging its stage, product, and topic."}, {i:"<img src="https://s.w.org/images/core/emoji/17.0.2/72x72/1f433.png" alt="🐳" class="wp-smiley" style="height: 1em; max-height: 1em;" />",l:"Environments",h:"2 · Two real environments",t:"The framework boots a hosted-like Supabase stack and a local CLI project in containers. <code>platform-lite</code> serves a Management API-compatible surface backed by <code>@supabase/lite</code>."}, {i:"<img src="https://s.w.org/images/core/emoji/17.0.2/72x72/1f916.png" alt="🤖" class="wp-smiley" style="height: 1em; max-height: 1em;" />",l:"Agent runs",h:"3 · The agent works",t:"Claude Code, Codex, OpenCode, or an AI SDK agent invokes the real Supabase MCP server and CLI — not mocks. Skills load lazily: only name and description sit in the system prompt."}, {i:"<img src="https://s.w.org/images/core/emoji/17.0.2/72x72/1f501.png" alt="🔁" class="wp-smiley" style="height: 1em; max-height: 1em;" />",l:"One retry",h:"4 · One retry allowed",t:"To cut false negatives while keeping runs sustainable, agents may retry once after a failure before they are graded."}, {i:"<img src="https://s.w.org/images/core/emoji/17.0.2/72x72/2696.png" alt="⚖" class="wp-smiley" style="height: 1em; max-height: 1em;" />",l:"Scoring",h:"5 · Deterministic + judge",t:"<code>EVAL.ts</code> exports the scorer. Deterministic checks confirm things like whether a user can reach certain data or an Edge Function returns the expected result; an LLM judge handles semantic calls."}, {i:"<img src="https://s.w.org/images/core/emoji/17.0.2/72x72/1f4ca.png" alt="📊" class="wp-smiley" style="height: 1em; max-height: 1em;" />",l:"Results",h:"6 · Benchmark or regression",t:"Benchmark scenarios go to the public site and run when assessing new changes or harnesses. Regression scenarios track known failure modes and refresh daily, without moving published scores."} ]; var FILES=[ {n:"PROMPT.md",d:"Task + frontmatter",h:"PROMPT.md",t:"Frontmatter plus the task description the agent sees. Keys drive discovery and the site filters: <code>stage</code>, <code>suite</code>, <code>product</code>, <code>topic</code>, <code>motivation</code>. <code>suite</code> is required on every eval."}, {n:"EVAL.ts",d:"The scorer",h:"EVAL.ts",t:"A default-exported scorer. Scorers check what the agent produced, never what the harness provisioned — with <code>projectRunning: true</code>, only the agent's deltas are scored."}, {n:"remote/",d:"Hosted project state",h:"remote/ — optional",t:"The hosted project's starting s

原文超出正文长度上限，此处截断——上游还有内容，完整版见上方「原文 ↗」。