AI编辑器重构：用JSX替代JSON Schema提升UI生成质量

Earlier this year, we shipped an AI editor for our remotely configurable paywalls product. Users can work with our agent to come up with a new paywall or edit an existing one.

On paper we fulfilled user requests, and adoption was growing rapidly. But there was a deeper issue that none of our eval or metrics could capture.

An inflection point: “it’s not even fair”

A pattern of issues started trickling in via user reports and internal feedback. On paper, the agent would accomplish the task the user asked, like creating the paywall, adding or overhauling a section of it. But the result it produced lacked the level of taste and quality that you’d expect a frontier agent to be capable of.

We worked to address this issue in several ways:

• We tried different models from various providers, with higher effort and reasoning levels
• We tweaked the system prompt and skills
• We injected messages with screenshots of the paywall, pushing the agent to check its work and maintain a high bar for quality

There were notable improvements, but the gap between what users wanted, and what the AI delivered, remained massive.

We then got some feedback from Ryan, one of our Product Managers:

(left) our old paywall editor, (right) using Fable in Claude Code(FYI: Astra is our internal term for the paywall agent)

This was a brutal inflection point. The feedback was originally about our agent getting the app’s name wrong. But the bigger issue was blatantly obvious.

How does our agent not understand that the paywall it produced is atrocious? Spacing, alignment, card within a card… there were too many issues to name. How did it think this was acceptable?

And how can Claude Code just one shot something that looks great while our agent flounders? This thought haunted me. No one would use our paywall editor if you could get that much better results with a simple Claude Code prompt.

So where did we go from here?

The old agent technically worked

To understand why our old agent was struggling, you first have to see how it worked.

Beginning with the data shape. Some remote paywall products use HTML, CSS and JS to render on all platforms. This provides the most flexibility, at the cost of feeling native, appearance, and performance.

We opted for native (well, a bit of both now). Our paywalls are stored as JSON trees: each node is a component with its own attributes. When your paywall is rendered, the component tree is mapped into the native components.

Here’s what a purchase button component can look like in our structure:

// ... one of 20+ components on this screen

Our agent had a very custom harness (set of tools, prompts and loop around the model). It understood the shape through an annotated JSON schema, and would operate on the paywall JSON directly, manipulating it with a set of tools.

This was a reasonable design for the first iteration.

• The targeted JSON operations prevented it from making simple formatting mistakes we’d had in earlier prototypes
• The schema showed it what was valid
• Skills gave it reusable patterns and guidance

But we were asking a lot of the model: it needed to learn this new schema on the fly for representing UI, along with the new tools. It needed to translate the user request into visual intent, then translate that visual intent into a series of tool operations on the JSON. And lastly, it needed to self-evaluate using a screenshot.

It was a bespoke workflow and syntax — a far cry from the Jetpack Compose, JSX and SwiftUI use cases it had been extensively trained on.

But the feature shipped.

The format is cooked

Still, I couldn’t shake the feeling that we were not just “one more prompt” away from closing this gap. Something was fundamentally wrong.

I read through Claude Code’s system prompt; there’s very little in there telling Claude how to make tasteful design, and certainly nothing specific to paywalls. Yet it achieved far better results even with models the harness wasn’t built for, like GPT.

Anthropic has a frontend design skill but, in my experience, the newer models still create great UI without it.

This led me to suspect the taste was being lost in translation. In a minimal harness, the models could express taste into code. But not into JSON patching tool calls…

What if our paywall could just be code? The model knows how to build a great UI in JSX. And JSX components are a tree very similar to ours: each node has JSON attributes and an optional set of children.

So we tried it. Here’s that same purchase button from earlier, as JSX:

It turned out to be fairly simple. A transpile step turns the JSON into this intermediate JSX, which the agent can edit as a file. Once it’s done editing, a strict compiler runs, flagging any issues that could prevent the paywall from displaying or functioning correctly. When there are no blocking issues, the compiler turns the JSX back into the same JSON structure our backend & SDKs expect.

Two days after rock bottom, the new prototype set a record on our blank paywalls eval dataset. That’s when we knew we were onto something.

Yet another coding agent

If the theory is that leaning into coding agent RL will outperform learning a new harness on the fly, why not lean in fully. Let’s give the agent a full workspace, simple mechanisms to read, and additional context as needed.

So we did just that. We gave it a folder of snippets to see examples of how to do certain component structures, which it can copy-paste as needed. It doesn’t need the full documentation for every component on every request, so I split it into multiple d.ts files that can be lazily loaded, instead of cramming 175KB of JSON schema at the start of every session. This meant paywall translations just became entries in a strings/ folder. The agent can keep context about the app, like what the app does and who it is for, in a context.md.

And most importantly, this gave us room to expand. New capabilities can be represented with files/folders, and since things are lazily loaded in, there is minimal additional cost/overhead to do so.

├── snippets/ (read-only)

│ └── ... (28 snippets)

│ ├── es_ES.json (14 strings)

│ └── fr_FR.json (14 strings, 2 stale)

└── types/ (read-only)

We didn’t need to reinvent our own tools to work with this format. We used what the model already knows: a free loop with Read, Write, Edit and a compiler. The compiler even feels like tsc: giving line numbers and diagnostics with guidance when there are issues in the paywall code, and handing back a screenshot of the paywall when it succeeds.

With a familiar format and harness, We were also able to remove over half of our system prompt and almost all of our skills. Most of those skills and prompts were compensating for a subpar format. As we continue to iterate, we focus on improving format first, then only prompt our way out if we must.

Works on my machine

The prototype broke eval records in two days. Getting that to production was a full team effort that took several weeks.