NVIDIA 开源全双工语音模型 VoiceChat 11B
NVIDIA Releases NemotronLabs VoiceChat 11B: An Open Full-Duplex Speech-to-Speech Model with ~450 ms Turn-Taking and Live Tool Calling
NVIDIA has released NemotronLabs VoiceChat 11B, an open 11B end-to-end speech-to-speech model for real-time, full-duplex conversation. Instead of chaining ASR, an LLM, and TTS, it performs streaming speech understanding and speech generation in one unified network. That removes the multi-model orchestration and API handoffs a cascaded stack requires, and cuts end-to-end latency: measured smooth turn-taking latency is 448 ms on Full-Duplex-Bench 1.0. The model listens while it speaks, so a user can barge in mid-turn and the agent yields, with a take-over rate of 1.00 at 480 ms. It is also first open full-duplex model to support tool calling while conversation keeps flowing, using a separate output channel for <TOOLCALL> scripts along with operator-defined “on-hold” lines that fill the gap while an API runs.
NVIDIA 发布了 NemotronLabs VoiceChat 11B，这是一个开放的 11B 端到端语音到语音模型，用于实时全双工对话。它不串联 ASR、LLM 和 TTS，而是在一个统一的网络中执行流式语音理解和语音生成。这消除了级联堆栈所需的多模型编排和 API 交接，并降低了端到端延迟：在 Full-Duplex-Bench 1.0 上测得的平滑轮流对话延迟为 448 毫秒。该模型在说话时也能聆听，因此用户可以在对话中途打断，智能体会让出话语权，在 480 毫秒时接管率为 1.00。它也是首个支持在对话持续进行时调用工具的开源全双工模型，使用单独的输出通道输出 <TOOLCALL> 脚本，并带有操作员定义的“保持”语句，在 API 运行期间填补空白。
Is it deployable?
它可以部署吗？
PARTIAL — deployable today for pilots, not for production. Weights and container are both public, and the license is permissive. But NVIDIA team states the checkpoint is ‘ready for research purposes only,’ and the repo documents real failure modes: a two-minute audio context ceiling, degradation into non-recoverable gibberish after several turns, runaway self-talk after a turn ends, and dropped words in user transcription.
部分可以——目前可用于试点，不适合生产环境。权重和容器均已公开，许可证也较为宽松。但 NVIDIA 团队表示，该检查点“仅用于研究目的”，且仓库中记录了真实的失败模式：两分钟的音频上下文上限、几轮对话后退化为不可恢复的胡言乱语、一轮对话结束后出现失控的自言自语，以及用户转录中的词语丢失。
- Which companies: any team that can allocate one GPU with at least 80 GB of VRAM — A100, H100, RTX 6000 Pro, or B200 on x86_64 Linux. That covers AI-native startups, funded scaleups, enterprise R&D and innovation labs, GPU cloud providers, and university speech groups. There is no hosted API and no inference provider currently serves the model, so teams without GPU access may not evaluate it.
- Industries: contact centers and CX platforms, automotive in-cabin assistants, retail and drive-thru ordering, telecom IVR modernization, games and NPC dialogue, and accessibility tooling.
- Applications: barge-in-capable voice agents, voice front-ends over internal APIs, live-lookup assistants (weather, pricing, order status), and duplex latency benchmarking harnesses.
- 哪些公司：任何能够分配一块至少 80 GB 显存的 GPU（A100、H100、RTX 6000 Pro 或 B200，运行 x86_64 Linux）的团队。这包括 AI 原生初创公司、有资金的成长型公司、企业研发和创新实验室、GPU 云提供商以及大学语音研究组。目前没有托管的 API，也没有推理提供商提供该模型，因此没有 GPU 访问权限的团队可能无法评估它。
- 行业：联络中心和客户体验平台、汽车座舱助手、零售和得来速点餐、电信 IVR 现代化、游戏和 NPC 对话，以及无障碍工具。
- 应用：支持打断的语音代理、内部 API 之上的语音前端、实时查询助手（天气、定价、订单状态），以及双工延迟基准测试工具。
Architecture
架构
The model is a hybrid Mamba/Transformer, assembled from three existing NVIDIA components along with one new output path:
该模型是一个混合 Mamba/Transformer 模型，由三个现有的 NVIDIA 组件和一个新的输出路径组装而成：
- A Fast Conformer speech encoder from Nemotron-Speech-Streaming-En-0.6b, which encodes the incoming 16 kHz stream continuously.
- The NVIDIA Nemotron Nano v2 LLM backbone, which consumes audio tokens and predicts text tokens.
- An NVIDIA TTS decoder and codec that predicts audio codes, rendered as 22.05 kHz agent speech.
- A separate output channel dedicated to tool-calling scripts.
- 来自 Nemotron-Speech-Streaming-En-0.6b 的 Fast Conformer 语音编码器，持续编码输入的 16 kHz 音频流。
- NVIDIA Nemotron Nano v2 LLM 主干，消耗音频令牌并预测文本令牌。
- NVIDIA TTS 解码器和编解码器，预测音频码，渲染为 22.05 kHz 的智能体语音。
- 一个专门用于工具调用脚本的独立输出通道。
Outputs include agent audio, agent text, and a running user transcription. Training used roughly 550k hours of audio across real and synthetic corpora, building on SALM-Duplex and Audio Flamingo 3.
输出包括智能体音频、智能体文本和实时用户转录。训练使用了约55万小时的音频，涵盖真实和合成语料库，基于SALM-Duplex和Audio Flamingo 3构建。
Tool calling without dead air
无静默工具调用
Tool calls are emitted on the side channel as a <TOOLCALL> block; your code returns results in a <TOOL_RESPONSE> block. The notable piece is the on-hold message: per tool, an operator defines a line the agent speaks the moment the model generates the text triggering the call, so the conversation does not fall silent while an API runs.
工具调用在侧通道上以<TOOLCALL>块的形式发出；你的代码在<TOOL_RESPONSE>块中返回结果。值得注意的是保持消息：对于每个工具，操作员定义一行文本，当模型生成触发调用的文本时，智能体立即说出该行，从而在API运行期间对话不会陷入沉默。
Constraints are explicit. NVIDIA recommends a maximum of five tools per session, the model cannot reliably call multiple tools simultaneously, and the user cannot interrupt the agent during tool execution. System prompts and tool responses must be ASCII-only and TTS-friendly.
约束是明确的。NVIDIA建议每个会话最多使用五个工具，模型无法可靠地同时调用多个工具，并且在工具执行期间用户无法打断智能体。系统提示和工具响应必须仅包含ASCII字符且适合TTS。
Performance
性能
On Full-Duplex-Bench 1.0: smooth turn-taking TOR 0.82 at 448 ms, user-interruption TOR 1.00 at 480 ms, and pause-handling TOR of 0.153 (synthetic) and 0.255 (Candor), where lower is better.
在Full-Duplex-Bench 1.0上：平滑轮流对话TOR为0.82（448毫秒），用户打断TOR为1.00（480毫秒），暂停处理TOR为0.153（合成）和0.255（Candor），数值越低越好。
On AU Harness BFCL-v3 spoken tool calling: 58.5% simple, 62.5% multiple, 42.5% parallel, 27.5% parallel-multiple, 89.6% irrelevance, 56.1% average. On Full-Duplex-Bench v3: 82.5% tool selection, 44.2% argument accuracy, 33% pass@1.
在AU Harness BFCL-v3口语工具调用上：简单58.5%，多个62.5%，并行42.5%，并行多个27.5%，不相关89.6%，平均56.1%。在Full-Duplex-Bench v3上：工具选择82.5%，参数准确率44.2%，pass@1为33%。
NVIDIA reports the model ranks #2 among open full-duplex models on VoiceBench and #2 among open models on Full-Duplex-Bench 1.0.
NVIDIA报告该模型在VoiceBench上排名第二，在Full-Duplex-Bench 1.0上排名第二。
Interactive explainer
交互式说明
Key Takeaways
关键要点
- One 11B model replaces the ASR → LLM → TTS chain, at 448 ms measured turn-taking latency.
- First open full-duplex model with tool calling, using a side channel plus operator-defined on-hold messages.
- Weights are OpenMDW-1.1 permissive, but NVIDIA labels the checkpoint research-only.
- Requires one 80 GB GPU; no hosted API exists today.
- 一个11B模型取代了ASR → LLM → TTS链，实测轮流对话延迟为448毫秒。
- 首个支持工具调用的开源全双工模型，使用侧通道和操作员定义的保持消息。
- 权重采用OpenMDW-1.1许可，但NVIDIA将检查点标记为仅限研究。
- 需要一块80 GB GPU；目前没有托管API。
Check out the Hugging Face model card, GitHub (NeMo Speech) and NGC container. Also, feel free to follow us on Twitter and don’t forget to join our 150k+ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well.
查看Hugging Face模型卡、GitHub（NeMo Speech）和NGC容器。另外，欢迎在Twitter上关注我们，别忘了加入我们的150k+ML SubReddit并订阅我们的通讯。等等！你在Telegram上吗？现在你也可以在Telegram上加入我们。
Need to partner with us for promoting your GitHub Repo OR Hugging Face Page OR Product Release OR Webinar etc.? Connect with us
需要与我们合作推广你的GitHub仓库或Hugging Face页面或产品发布或网络研讨会等？请联系我们
The post NVIDIA Releases NemotronLabs VoiceChat 11B: An Open Full-Duplex Speech-to-Speech Model with ~450 ms Turn-Taking and Live Tool Calling appeared first on MarkTechPost.
文章《NVIDIA发布NemotronLabs VoiceChat 11B：一个具有约450毫秒轮流对话和实时工具调用的开源全双工语音到语音模型》首次出现在MarkTechPost上。
更进一步：量化金融体系
看懂新闻只是起点——沿量化金融路径，把它变成能交付的工程能力