AI工程师必知的4种Agent部署策略

4 AI Agent deployment strategies AI engineers should know.

(bookmark this)

An agent can run on a schedule, on a stream, behind a live API, or on the device itself, and the workload decides which.

The visual below explains the 4 main deployment patterns:

> Batch deployment:

It runs the agent on a schedule over data that arrives in bulk. External context lands in batch storage, the agent processes the whole set, and results get written to an inference store that the backend reads from.

Latency is not a concern here, so it fits offline scoring, nightly enrichment, and any job where the answer is needed later, not now.

For instance, if you need a pipeline to score every lead in a CRM overnight or generate weekly summaries for thousands of accounts, it is best handled via batch deployment, since the data is already there and nobody needs the answer this second.

> Stream deployment:

It runs the agent continuously over events that keep arriving, like transactions, clicks, or messages, with no fixed end.

The agent processes each event from streaming storage as it arrives, pulls external context per event, and writes results back to a streaming inference store.

For instance, a system that flags fraudulent transactions as they flow through or ranks posts in a live feed is best handled via stream deployment, since the events arrive on their own and have to be handled as they land.

> Real-time deployment:

It answers one request at a time under a strict latency budget.

Context comes from low-latency storage, the agent talks to backends over gRPC instead of REST to cut serialization overhead, and a load balancer spreads requests across replicas.

This is the synchronous path, used when a user or service is blocked waiting for the response.

For instance, a chatbot answering a user or a coding assistant returning a completion is a real-time deployment, since a caller is stalled on the response and the latency budget is tight.

> Edge deployment:

The agent runs on the device itself, a phone, a watch, or a laptop, with no network round-trip to a backend.

This fits cases where the round trip is too slow, the network is unreliable, or the data cannot leave the device for privacy reasons.

Edge is the hardest of the four to ship, because the model has to fit and run inside the device's own memory and compute.

So this requires shrinking it with quantization and running it through an on-device runtime instead of a server stack.

I wrote a walkthrough on doing exactly that, fine-tuning Qwen3 with Unsloth, quantizing, and deploying it on iOS and Android, fully local with no server.

Read it below.