用7美元ESP32让AI操控iPhone：无需越狱或Mac
onehumanbeing/esp32-iphone-agent: Let an AI model use a real iPhone with a $7 ESP32: AirPlay eyes, USB HID hands, any OpenAI-compatible vis…
给Agent开发者提供了一个极具性价比的iOS自动化新思路，绕过苹果生态限制实现物理层交互，工程细节详实，值得收藏参考。
Let an AI model use a real iPhone with a $7 ESP32: AirPlay eyes, USB HID hands, any OpenAI-compatible vision model. No Mac, no developer account, no jailbreak.
让 AI 模型使用一台真实的 iPhone，搭配一块 7 美元的 ESP32：AirPlay 提供视觉，USB HID 模拟操控，支持任意兼容 OpenAI 的视觉模型。无需 Mac，无需开发者账号，无需越狱。
GitHub repository created 2026-10-11 · 1998 stars in 2.8 days (about 707 a day) · Python
GitHub 仓库创建于 2026-10-11 · 2.8 天内获得 1998 颗星（约每天 707 颗）· Python
ESP32 iPhone Agent
Let an AI model use a real iPhone — with a $7 ESP32. No Mac, no developer account, no jailbreak.
让 AI 模型使用一台真实的 iPhone —— 仅需一块 7 美元的 ESP32。无需 Mac，无需开发者账号，无需越狱。
English · 简体中文
Demo video and updates: X @iamhenryyuan · YouTube demo
演示视频和更新：X @iamhenryyuan · YouTube 演示
Example 1 — draw on a canvas. *"Open Whiteboard and draw a smiley face"*: the agent searches for the app, opens it and draws the face with continuous strokes. Every step's screenshot, with its tap or stroke drawn on top, stays on the step wall.
示例 1 — 在画布上绘图。*“打开白板并画一个笑脸”*：智能体搜索该应用，打开它，并用连续的笔触画出笑脸。每一步的截图及其对应的点击或笔触都会保留在步骤墙上方。
Example 2 — set an alarm. *"Set an alarm for 7:00 AM that repeats every day, then save it"*: the agent opens Clock, types 0700 into the time wheel, ticks the days under Repeat several taps at a time, and saves. Left: the real iPhone with the ESP32 plugged in; right: the web UI.
示例 2 — 设置闹钟。*“设置一个每天早上重复的早上 7:00 的闹钟，然后保存”*：智能体打开时钟应用，在时间轮中输入 0700，多次轻点以勾选重复选项下的天数，然后保存。左侧是插着 ESP32 的真实 iPhone；右侧是 Web UI。
The agent sees the phone through AirPlay screen mirroring and touches it through a tiny ESP32-S3 board that plugs into the phone's USB port as a mouse and keyboard. Any OpenAI-compatible vision model can be the brain — a cloud API, or a model you run yourself.
智能体通过 AirPlay 屏幕镜像看到手机画面，并通过一块微小的 ESP32-S3 板触摸手机，该板插入手机的 USB 端口作为鼠标和键盘。任何兼容 OpenAI 的视觉模型都可以充当大脑——可以是云端 API，也可以是你自己运行的模型。
Why
为什么
Most iPhone automation drives the phone through XCTest / WebDriverAgent. That needs a Mac, Xcode, a signed helper app and an Apple developer account (a free account's signature expires every 7 days). This project uses the same paths a person uses — the screen and a pointer — so the phone needs nothing installed.
大多数 iPhone 自动化工具通过 XCTest / WebDriverAgent 驱动手机。这需要 Mac、Xcode、已签名的辅助应用以及 Apple 开发者账号（免费账号的签名每 7 天过期一次）。本项目使用的是人类使用的相同路径——屏幕和指针——因此手机上无需安装任何东西。
| ESP32 iPhone Agent | XCTest / WebDriverAgent tools | |||
|---|---|---|---|---|
| Apple developer account | Not needed | Needed (to sign the runner) | ||
| Xcode | Not needed | Needed | ||
| Computer | Mac or Linux | Mac | ||
| App installed on the phone | None | Test runner app | ||
| Jailbreak | Not needed | Not needed | ||
| Extra hardware | ESP32-S3 board (~$7) | USB cable | ||
| Works in any app, incl. games | Yes (it only sees pixels) | Mostly (depends on the accessibility tree) |
| ESP32 iPhone Agent | XCTest / WebDriverAgent 工具 | |||
|---|---|---|---|---|
| Apple 开发者账号 | 不需要 | 需要（用于签署运行器） | ||
| Xcode | 不需要 | 需要 | ||
| 计算机 | Mac 或 Linux | Mac | ||
| 手机上安装的应用 | 无 | 测试运行器应用 | ||
| 越狱 | 不需要 | 不需要 | ||
| 额外硬件 | ESP32-S3 开发板（约 7 美元） | USB 线缆 | ||
| 适用于任何应用，包括游戏 | 是（它只识别像素） | 大部分情况（取决于无障碍树） |
How it works
工作原理
- Eyes — AirPlay mirroring. The phone mirrors its screen to UxPlay running on your computer; a GStreamer pipeline turns the stream into JPEG frames for the agent and the web page. - Hands — a USB HID bridge. An ESP32-S3 plugged into the phone shows up as a standard USB mouse and keyboard. With iOS AssistiveTouch on, the mouse moves an on-screen pointer, so a click is a tap and a drag is a swipe. The board connects to the computer over Wi-Fi and executes simple line commands (`move`, `click`, `type`, …); all logic stays on the computer. - Brain — a planner and an actor. A planner model writes a short plan from the goal and the current screen. An actor model then looks at each new screenshot and picks the next action: tap, swipe, draw a path, type, press a key, go home, or finish. Both roles can be the same model. - Closing the loop. The pointer is relative, so the server tracks where it is. Before each tap it wiggles the pointer, finds it on the mirrored screen and corrects any miss, refining the calibration as it goes. After each action it checks whether the screen actually changed and tells the model, so a missed tap is noticed and retried instead of built upon. Repeated actions trigger a re-plan; a dropped link or a bad reply is recovered from.
- 眼睛——AirPlay 镜像。手机将屏幕镜像到运行在电脑上的 UxPlay；GStreamer 管道将流转换为 JPEG 帧，供代理和网页使用。 - 手——USB HID 桥接。插入手机的 ESP32-S3 会显示为标准 USB 鼠标和键盘。 开启 iOS 辅助触控后，鼠标移动屏幕指针，因此点击即为轻触，拖拽即为滑动。 开发板通过 Wi-Fi 连接到电脑并执行简单的行命令（`move`、`click`、`type` 等）； 所有逻辑都保留在电脑上。 - 大脑——规划器和执行者。规划器模型根据目标和当前屏幕生成简短计划。 然后执行者模型查看每张新截图并选择下一个动作：轻触、滑动、绘制路径、输入文本、按下 按键、返回主屏幕或完成。这两个角色可以是同一个模型。 - 闭环控制。指针是相对坐标，因此服务器会跟踪其位置。每次轻触前，它会轻微晃动 指针，在镜像屏幕上找到该指针并纠正任何偏差，从而逐步优化校准。每次操作后，它会检查屏幕是否实际发生变化并告知模型，以便发现未成功的轻触并重试，而不是基于错误继续操作。重复操作会触发重新规划；断开的连接或错误的回复会被恢复处理。
Hardware
硬件
| Part | Notes | ||
|---|---|---|---|
| ESP32-S3 board | Tested with a Seeed Studio XIAO ESP32S3, plain version without pre-soldered headers (not the Sense camera version, not the Plus). About $7 from Seeed, or about ¥40-50 on Taobao in China (search "XIAO ESP32S3"). Any ESP32-S3 board whose USB-C port is wired to the S3's native USB works. | ||
| iPhone | USB-C models plug in directly. Lightning models need a Lightning to USB Camera Adapter. The phone powers the board. | ||
| Computer | macOS or Linux, on the same network as the phone; runs UxPlay and the server. | ||
| Vision model | Any OpenAI-compatible chat API that accepts images. Running a model locally is optional. |
| 部件 | 说明 | ||
|---|---|---|---|
| ESP32-S3 开发板 | 已测试 Seeed Studio XIAO ESP32S3，普通版，无预焊排针（非 Sense 摄像头版，非 Plus 版）。从 Seeed 购买约 7 美元，或在淘宝搜索“XIAO ESP32S3”约 40-50 元人民币。任何 USB-C 端口连接到 S3 原生 USB 的 ESP32-S3 开发板均可使用。 | ||
| iPhone | USB-C 型号可直接插入。Lightning 型号需要 Lightning 转 USB 相机适配器。手机为开发板供电。 | ||
| 电脑 | macOS 或 Linux，与手机在同一网络中；运行 UxPlay 和服务器。 | ||
| 视觉模型 | 任何接受图像的 OpenAI 兼容聊天 API。本地运行模型为可选。 |
Use a USB-C hub with power pass-through if the phone should keep charging while the board is plugged in.
如果希望开发板插入时手机保持充电，请使用支持电源透传的 USB-C 集线器。
Quick start
快速开始
> Set up with an AI coding agent. Paste this into Claude Code, Codex or a similar agent on the computer the > board will connect to: > > ```text > Clone https://github.com/onehumanbeing/esp32-iphone-agent and follow its README to set it up on this machine: > flash the ESP32-S3 plugged in over USB, install UxPlay and the server, write server/.env for my model API, start > the server and give me the web UI address. Ask me for the Wi-Fi name/password and the model API key. > ```
> 使用 AI 编程代理进行设置。将此粘贴到 Claude Code、Codex 或连接到该板子的计算机上的类似代理中： > > ```text > Clone https://github.com/onehumanbeing/esp32-iphone-agent and follow its README to set it up on this machine: > flash the ESP32-S3 plugged in over USB, install UxPlay and the server, write server/.env for my model API, start > the server and give me the web UI address. Ask me for the Wi-Fi name/password and the model API key. > ```
1. Flash the board
1. 烧录主板
Install arduino-cli and the ESP32 core (3.x), then:
安装 arduino-cli 和 ESP32 core (3.x)，然后：
2. Install and start the server
2. 安装并启动服务器
Open `http://:8890` in a browser.
在浏览器中打开 `http://:8890`。
3. Set up the phone (once)
3. 设置手机（一次性）
1. Settings → Accessibility → Touch → AssistiveTouch: turn it on. 2. Plug the board into the phone. A round pointer appears; the web page shows *controller connected*. 3. Control Center → Screen Mirroring → iPhone Agent. The live screen appears in the web page. 4. In the web page, open Manual controls → Calibrate. The server moves the pointer, finds it on the live screen and calibrates itself in a few seconds; if it cannot see the pointer, it asks you to click exactly on it.
1. 设置 → 辅助功能 → 触控 → 辅助触控：开启。 2. 将主板插入手机。会出现一个圆形指针；网页显示 *控制器已连接*。 3. 控制中心 → 屏幕镜像 → iPhone Agent。实时画面出现在网页上。 4. 在网页中，打开 手动控制 → 校准。服务器移动指针，在实时画面上找到它并在几秒内完成自我校准；如果看不到指针，它会要求你精确点击它。
4. Give it a task
4. 给它一个任务
Type a goal (for example *"Open Settings and go to Wi-Fi"*) and press Start, or click one of the examples.
输入目标（例如 *"打开设置并进入 Wi-Fi"*），然后按 Start，或者点击其中一个示例。
Web UI
- Live phone — click to tap, drag to swipe, scroll to scroll; manual Home, Back, typing and keys. - Live log — every plan, thought and action as it happens, with timings. - Step wall — the screenshot the model decided on for every step, with its tap or swipe drawn on top, plus the model's reasoning and how long thinking and acting took. Click a card to zoom in. - Taps are marked with a red ring and a spreading ripple, swipes and strokes with a yellow path; the Motion switch turns the animation off. Taps the model marked `precise` (pointer checked before tapping) are labelled in the log. - Emergency stop (or Esc) stops the agent and releases the pointer. Clear history empties the log. - English and Simplified Chinese interface.
- 实时手机 — 点击以轻触，拖动以滑动，滚动以滚动；手动 Home、Back、打字和按键。 - 实时日志 — 记录发生的每一个计划、思考和动作，以及时间戳。 - 步骤墙 — 模型为每个步骤决定的截图，上面绘制了其轻触或滑动的轨迹，以及模型的推理过程和思考与操作所花费的时间。点击卡片可放大查看。 - 轻触标记为红色圆环和扩散的涟漪，滑动和笔画为黄色路径；Motion 开关可关闭动画。模型标记为 `precise`（轻触前检查过指针）的轻触会在日志中标注。 - 紧急停止（或 Esc）会停止代理并释放指针。清除历史记录会清空日志。 - 英文和简体中文界面。
Configuration
配置
更进一步：量化金融体系
看懂新闻只是起点——沿量化金融路径，把它变成能交付的工程能力