跳到主内容
@wquguru
精选88PyTorch 博客(RSS)技巧与观点

PyTorch ExecuTorch优化Arm端流式语音识别实战

Optimizing Streaming Speech Recognition on Arm® Edge Devices with ExecuTorch

原文
发到 X
推荐理由

这是一篇极具价值的端侧 AI 工程实践指南,详细拆解了在 Arm 设备上部署流式 ASR 模型的精度、延迟权衡及性能优化技巧,适合做端侧部署的同学参考。

When you’re speaking to a device, you expect text to appear as you talk. That makes streaming automatic speech recognition (ASR) a different challenge from batch transcription, where the model has the complete recording before it starts. Strong batch accuracy does not necessarily carry over into streaming. One cross-model ASR study illustrates this gap. Qwen3-ASR-1.7B achieved the best batch accuracy among the models tested, but its word error rate (WER) rose by 77% when evaluated with 2.4-second audio chunks.

当你对设备进行语音交互时,你期望文字能随着你的说话实时出现。这使得流式自动语音识别(ASR)与批量转录面临不同的挑战:在批量转录中,模型在开始处理前已拥有完整的录音记录。出色的批量准确率并不一定能直接迁移到流式场景中。一项跨模型的 ASR 研究展示了这一差距:Qwen3-ASR-1.7B 在被测模型中实现了最佳的批量准确率,但当使用 2.4 秒的音频块进行评估时,其词错误率(WER)上升了 77%。

In that study’s cross-model streaming comparison (Table 5), NVIDIA® Nemotron Speech Streaming 0.6B had the lowest average streaming WER when operating over 560 ms audio chunks. The comparison makes a critical point: accuracy must be measured at the chunk cadence used in deployment.

在该研究的跨模型流式对比(表 5)中,NVIDIA® Nemotron Speech Streaming 0.6B 在处理 560 毫秒音频块时具有最低的平均流式 WER。这一对比揭示了一个关键点:必须在部署所使用的音频块节奏下测量准确率。

This case study takes the newer, multilingual Nemotron 3.5 from PyTorch to ExecuTorch and profiles its on-device execution on a Google Pixel 11. We ask how precision and chunk size change streaming quality, how much processing time the audio requires with one inference thread, where the workload spends its time, and which native kernels run. The goal is a practical way to evaluate accuracy and latency together.

本案例研究将 PyTorch 中的较新多语言 Nemotron 3.5 模型移植到 ExecuTorch,并分析其在 Google Pixel 11 上的设备端执行性能。我们探讨了精度和音频块大小如何影响流式质量、单推理线程下音频所需的处理时间、工作负载的时间分布以及哪些原生内核被调用。目标是提供一种实用方法,同时评估准确率和延迟。

In this blog post, you will learn:

在本篇博客文章中,你将了解:

  • How ExecuTorch connects PyTorch models to optimized Arm CPU kernels.
  • Why streaming ASR needs its own quality evaluation, and how longer chunks improve final-transcript accuracy at the cost of higher latency.
  • How Arm Scalable Matrix Extension 2 (SME2) reduces total ASR processing time with FP16 and INT8 quantization using per-channel weights (QC8W).
  • How to combine model operator timings and native kernel samples to understand performance and identify remaining bottlenecks.
  • How to choose chunk size and precision together to meet your accuracy and latency targets.
  • ExecuTorch 如何将 PyTorch 模型连接到优化的 Arm CPU 内核。
  • 为什么流式 ASR 需要独立的质量评估,以及较长的音频块如何在增加延迟的同时提高最终转录的准确率。
  • Arm Scalable Matrix Extension 2 (SME2) 如何利用 FP16 和 INT8 量化(使用每通道权重 QC8W)来减少总 ASR 处理时间。
  • 如何结合模型算子计时数据和原生内核采样数据来理解性能表现并识别剩余瓶颈。
  • 如何协同选择音频块大小和精度,以满足准确性和延迟目标。

From PyTorch model to CPU instructions

从 PyTorch 模型到 CPU 指令

ExecuTorch is PyTorch’s edge inference framework, with export tools and a lightweight runtime. The export process prepares the model for inference and produces a .pte file that the device runtime can load and execute.

ExecuTorch 是 PyTorch 的边缘推理框架,包含导出工具和轻量级运行时。导出过程为模型进行推理做准备,并生成一个 .pte 文件,供设备运行时加载和执行。

Supported operations are delegated to XNNPACK, a CPU inference library. On Android, XNNPACK can select microkernels from Arm® KleidiAI software: small, optimized routines that perform the inner matrix calculations. Operators not delegated to XNNPACK run using ExecuTorch’s portable CPU kernels.

支持的操作会被委托给 XNNPACK,这是一个 CPU 推理库。在 Android 上,XNNPACK 可以从 Arm® KleidiAI 软件中选择微内核:这些是经过优化的小型例程,用于执行内部矩阵计算。未委托给 XNNPACK 的算子则使用 ExecuTorch 的可移植 CPU 内核运行。

Figure 1. From model export to CPU instructions.

图 1. 从模型导出到 CPU 指令。

SME2 extends Arm’s instruction set to accelerate matrix calculations, including general matrix multiplication (GEMM). In the Arm C1 CPU cluster implementation with SME2, the matrix unit is shared by cores in the CPU cluster, and KleidiAI provides microkernels that use it for supported operations. Enabling SME2 in a build makes those paths available. However, native profiling is needed to show which kernels actually ran.

SME2 扩展了 Arm 的指令集以加速矩阵计算,包括通用矩阵乘法(GEMM)。在采用 SME2 的 Arm C1 CPU 集群实现中,矩阵单元由 CPU 集群中的核心共享,KleidiAI 提供了使用它来执行支持操作的微内核。在构建中启用 SME2 可使这些路径可用。然而,需要原生性能分析才能显示实际运行了哪些内核。

What one streaming call contains

一次流式调用包含的内容

Nemotron 3.5 is a multilingual streaming ASR model built around an RNN-T architecture with three parts:

Nemotron 3.5 是一个基于 RNN-T 架构构建的多语言流式自动语音识别(ASR)模型,由三个部分组成:

  • The encoder uses FastConformer attention and convolution layers to represent the audio.
  • The predictor uses an LSTM to represent the history of emitted tokens.
  • The joiner uses a small feedforward layer to combine the encoder’s audio representation with the predictor’s token-history representation and predict the next token. It can also produce a blank, which advances decoding to the next audio position.
  • 编码器使用 FastConformer 注意力和卷积层来表示音频。
  • 预测器使用 LSTM 来表示已发出标记的历史。
  • 连接层使用一个小型前馈层将编码器的音频表示与预测器的标记历史表示相结合,并预测下一个标记。它还可以产生空白,从而将解码推进到下一个音频位置。

The frontend converts audio into log-mel feature vectors, producing one vector for every 10 ms. The encoder reduces the number of time positions by a factor of 8, so its output has one position for every 80 ms of audio. This creates a more compact sequence representing the same audio duration.

前端将音频转换为对数梅尔特征向量,每 10 毫秒生成一个向量。编码器将时间位置的数量减少 8 倍,因此其输出每 80 毫秒音频对应一个位置。这创建了一个更紧凑的序列来表示相同的音频持续时间。

Figure 2. From waveform windows to streaming encoder positions.

图 2. 从波形窗口到流式编码器位置。

Lookahead is the future context available within a chunk; chunk cadence is how often fresh audio is collected for a call. At lookahead 13, a call collects 1,120 ms of audio and produces 14 encoder positions. The predictor and joiner processes all 14 valid positions, accumulating their tokens into the transcript.

前瞻是块内可用的未来上下文;块的节奏是为一次调用收集新音频的频率。在前瞻为 13 时,一次调用收集 1,120 毫秒的音频并生成 14 个编码器位置。预测器和连接层处理所有 14 个有效位置,将其标记累积到转录文本中。

Lookaheads 0, 3, and 6 give 80, 320, and 560 ms chunks. Longer chunks increase the wait for the first text and subsequent updates; preprocessing and inference add time after audio collection.

前瞻为 0、3 和 6 时分别给出 80、320 和 560 毫秒的块。较长的块会增加等待第一个文本及后续更新的时间;预处理和推理会在音频收集后增加时间。

Choose chunk size and precision together

同时选择块大小和精度

We measured transcription accuracy on the host using the ExecuTorch runtime with its XNNPACK CPU backend. We created a local streaming adapter that uses the Hugging Face Open ASR Leaderboard evaluation harness for dataset preparation, transcript normalization, and WER scoring.

我们使用带有 XNNPACK CPU 后端的 ExecuTorch 运行时,在主机上测量了转录准确率。我们创建了一个本地流式适配器,该适配器使用 Hugging Face Open ASR Leaderboard 评估工具进行数据集准备、转录文本归一化和 WER 评分。

We evaluated FP32, FP16, and INT8 at four chunk sizes. The INT8 specification is QC8W which uses dynamic INT8 activations and per-channel INT8 weights for eligible linear operations, with FP32 outputs.

我们在四种块大小下评估了 FP32、FP16 和 INT8。INT8 规范为 QC8W,它对于符合条件的线性运算使用动态 INT8 激活值和逐通道 INT8 权重,并输出 FP32。

Figure 3. Streaming WER versus chunk size across four corpora.

图 3. 四个语料库上不同块大小的流式 WER。

Each point on the above figure covers 14,019 utterances from LibriSpeech Other (LS Other), AMI Cleaned, Earnings22, and VoxPopuli Cleaned AA datasets. WER divides total word errors by total reference words across all four corpora.

上图中的每个数据点涵盖了来自 LibriSpeech Other(LS Other)、AMI Cleaned、Earnings22 和 VoxPopuli Cleaned AA 数据集的 14,019 条话语。WER 计算的是四个语料库中总词错误数与总参考词数的比值。

Longer chunks improve final-transcript accuracy. From 80 to 1,120 ms, FP16 WER falls from 15.2% to 11.5%, and QC8W from 15.5% to 11.7%. FP16 essentially matches FP32, differing by less than 0.03 percentage points at every chunk size, so we exclude FP32 from further performance analysis. QC8W preserves strong accuracy, adding 0.21–0.31 percentage points of WER relative to FP32.

更长的分块能提高最终转录准确率。从 80 到 1,120 毫秒,FP16 的 WER 从 15.2% 降至 11.5%,QC8W 从 15.5% 降至 11.7%。FP16 的表现几乎与 FP32 持平,在每个分块大小下差异小于 0.03 个百分点,因此我们在后续的性能分析中排除 FP32。QC8W 保持了较高的准确率,其 WER 相比 FP32 仅增加 0.21–0.31 个百分点。

These host WER results inform precision and lookahead choices. However, it does not measure how long it takes for text to appear on the device.

这些主机端的 WER 结果指导了精度选择和前瞻窗口大小的设定。然而,它并未测量文本在设备上显示所需的时间。

Test the real-time budget on an edge device

在边缘设备上测试实时预算

We processed a 24.9-second recording on a Google Pixel 11 smartphone device using 1,120 ms chunks and one inference thread pinned to a mid-core CPU. We chose the mid-core CPU as a representative target for a sustained streaming workload on the system. We tested two builds of ExecuTorch 1.3.1, toggling SME2 support on and off. Each build retained XNNPACK and KleidiAI, and used identical model artifacts.

我们使用 1,120 毫秒的分块大小和一个绑定到中间核心 CPU 的推理线程,在 Google Pixel 11 智能手机设备上处理了一段 24.9 秒的录音。我们选择中间核心 CPU 作为系统上持续流式工作负载的代表性目标。我们测试了两个 ExecuTorch 1.3.1 版本,分别开启和关闭 SME2 支持。每个版本都保留了 XNNPACK 和 KleidiAI,并使用相同的模型工件。

Each configuration had two warm-ups and one measured tracing run. Totals cover processing the loaded recording from audio preprocessing through final detokenization.

每种配置包含两次预热和一次测量的追踪运行。总计时间涵盖从音频预处理到最终去标记化(detokenization)处理加载录音的全过程。

Figure 4. Whole-recording processing time with SME2 disabled and enabled.

图 4. 禁用和启用 SME2 时的整段录音处理时间。

SME2 substantially reduces processing time: FP16 falls from 9.79 to 4.61 seconds (2.12x faster), and QC8W from 5.20 to 3.24 seconds (1.61x faster).

SME2 显著减少了处理时间:FP16 从 9.79 秒降至 4.61 秒(快 2.12 倍),QC8W 从 5.20 秒降至 3.24 秒(快 1.61 倍)。

With and without SME2, both process the recording in much less time than its audio duration. Dividing the processing time by the audio duration we calculate real-time factors of 0.19 for FP16 and 0.13 for QC8W (with SME2 on). This provides substantial processing headroom with only a single inference thread. Audio still has to arrive before it can be decoded, so this headroom does not remove the chunk-collection delay. But it leaves capacity for other CPU work between chunks and can help manage thermal load during sustained transcription.

无论是否启用 SME2,两种配置处理录音的时间都远短于其音频时长。将处理时间除以音频时长,我们计算出在开启 SME2 的情况下,FP16 的实时因子为 0.19,QC8W 为 0.13。这提供了 substantial 的处理余量,且仅使用单个推理线程。音频必须在解码前到达,因此这种余量并不能消除分块收集延迟。但它留下了在分块之间执行其他 CPU 工作的容量,并有助于在持续转录期间管理热负荷。

Find the bottleneck with ETDump

使用 ETDump 查找瓶颈

ETDump records ExecuTorch operator events and timings. To understand where the 2.12X FP16 and 1.61X QC8W speedups come from, we grouped machine learning (ML) operators across the encoder, predictor, and joiner into categories:

ETDump 记录 ExecuTorch 算子事件和时间。为了理解 2.12 倍的 FP16 和 1.61 倍的 QC8W 加速来源,我们将编码器、预测器和连接器的机器学习(ML)算子按类别分组:

  • GEMM (blue): Matrix multiplication, mainly the encoder’s dense layers.
  • Data movement (orange): Copies, concatenation, repeats, and embedding lookup.
  • Elementwise (red): Activations, comparisons, and other per-value work.
  • Convolution and normalization (green and purple): Convolution filters and normalization operators; both are relatively small here.
  • Other (violet): Small remaining timed operations, such as tensor creation.
  • Overhead (gray): Full-run time outside the grouped operator events; this can include runner and framework overhead.
  • GEMM(蓝色):矩阵乘法,主要是编码器的密集层。
  • 数据移动(橙色):复制、拼接、重复和嵌入查找。
  • 逐元素操作(红色):激活函数、比较以及其他按值进行的工作。
  • 卷积与归一化(绿色和紫色):卷积滤波器和归一化算子;两者在此处相对较小。
  • 其他(紫罗兰色):剩余的少量计时操作,例如张量创建。
  • 开销(灰色):分组算子事件之外的完整运行时间;这可能包括运行器和框架的开销。

Figure 5. Operator costs and overhead behind the whole-run speedup.

图 5. 整体加速背后的算子成本与开销。

更进一步:量化金融体系

看懂新闻只是起点——沿量化金融路径,把它变成能交付的工程能力

进入量化体系 →

相似阅读

关联信息,但可能不是同一事件