将多速率DSP原理应用于LLM的层级语义声码器架构
Applying multirate DSP principles to LLMs: A hierarchical "Semantic Vocoder" architecture [P]
将DSP多速率理论引入LLM结构设计的探索性研究，提供了完整的架构细节、实验数据及工程复盘，对关注模型效率与分层建模的研究者极具参考价值。
Hey everyone,
大家好，
I’ve been experimenting with bridging Digital Signal Processing (DSP) multirate theory and discrete text generation. I wanted to share a PyTorch reference architecture I've been working on, mostly to get feedback from folks who work on hierarchical modeling or continuous-to-discrete latent spaces.
我一直在尝试将数字信号处理（DSP）的多速率理论与离散文本生成相结合。我想分享一个我正在开发的 PyTorch 参考架构，主要是为了获得从事分层建模或连续到离散潜在空间工作的人员的反馈。
The premise: Standard dense LLMs treat text generation as a flat sequence. Predicting the "e" in "the" costs the same attention compute as calculating the crux of a logical argument. In audio (TTS), it is standard to decouple this: a model generates a slow-rate continuous signal (e.g., a mel-spectrogram), and a high-rate vocoder (e.g., WaveNet) synthesizes the discrete audio samples.
基本前提：标准的密集大语言模型（LLM）将文本生成视为扁平序列。预测“the”中的“e”所消耗的注意力计算量与计算逻辑论证核心是一样的。在音频（TTS）领域，通常会将这两者解耦：一个模型生成低速率的连续信号（例如梅尔频谱图），而高速率的声码器（例如 WaveNet）合成离散的音频样本。
I built a dual-rate architecture to do exactly this for BPE tokens.
我构建了一种双速率架构，专门用于 BPE 令牌执行此操作。
How it works:
工作原理：
- The Planner (Slow Rate): A sentence-level autoregressive transformer. It operates on highly compressed semantic embeddings (via a frozen SentenceTransformer) and predicts the next sentence's continuous embedding.
- The Vocoder (Fast Rate): An autoregressive GPT that handles high-frequency local grammar. It uses a banded sliding-window causal mask to restrict attention strictly to the local context.
- The Bridge: The slow-rate semantic timeline is step-repeated (upsampled) to exactly match the discrete BPE token boundaries. A late-stage adapter uses cross-attention to calculate a "delta" probability distribution, adjusting the base GPT's logits right before the softmax: Logits_final = Logits_base + softplus(alpha) * Logits_delta.
- 规划器（慢速率）：一个句子级别的自回归 Transformer。它在高度压缩的语义嵌入上运行（通过冻结的 SentenceTransformer），并预测下一个句子的连续嵌入。
- 声码器（快速率）：一个处理高频局部语法的自回归 GPT。它使用带状滑动窗口因果掩码，将注意力严格限制在局部上下文中。
- 桥接层：慢速率的语义时间线被逐步重复（上采样）以精确匹配离散的 BPE 令牌边界。后期适配器使用交叉注意力来计算“delta”概率分布，在 softmax 之前调整基础 GPT 的 logits：Logits_final = Logits_base + softplus(alpha) * Logits_delta。
The Results : Testing on TinyStories, the decoupled architecture converges much faster and deeper than a standard unconditioned baseline GPT of equivalent size (validation loss hit 0.61 vs the baseline's 2.37 at similar steps).
结果：在 TinyStories 上进行测试时，解耦架构比同等大小的标准无条件基线 GPT 收敛得更快、更深（在相似步骤下，验证损失降至 0.61，而基线为 2.37）。
However, I hit some interesting architectural bottlenecks that I've documented in the repo:
然而，我遇到了一些有趣的架构瓶颈，已在仓库中记录：
- Conditioning Over-Reliance: The adapter transmits the semantic signal too efficiently. The base GPT gets lazy and starts using the 384D semantic vector as a hash-key for the sentence rather than learning robust local grammar. Even with 15% Semantic Dropout (Classifier-Free Guidance), the Top-1 accuracy sits artificially high (~85%), which can lead to exposure bias and repetitive loops during greedy decoding.
- Hardware vs. Logical Complexity: Logically, the attention splits to O((N/C)^2) for the planner and O(N*W) for the vocoder. However, since my reference implementation uses standard PyTorch boolean masking for the sliding window, it still allocates the full NxN matrix under the hood. True VRAM savings will require swapping the base blocks for FlashAttention-2 block-sparse masks.
- 条件过度依赖：适配器传输语义信号过于高效。基础 GPT 变得懒惰，开始将 384D 语义向量用作句子的哈希键，而不是学习稳健的局部语法。即使有 15% 的语义丢弃（无分类器引导），Top-1 准确率也人为地高（约 85%），这可能导致在贪婪解码期间出现暴露偏差和重复循环。
- 硬件与逻辑复杂度：从逻辑上看，注意力机制对规划器拆分为 O((N/C)^2)，对声码器为 O(N*W)。然而，由于我的参考实现使用标准的 PyTorch 布尔掩码来处理滑动窗口，底层仍会分配完整的 NxN 矩阵。真正的显存节省需要将基础模块替换为 FlashAttention-2 的块稀疏掩码。
This is an exploratory proof-of-concept, not a SOTA claim against monolithic multi-billion parameter models. But I think the residual logit delta and the continuous-to-discrete phase alignment provide an interesting alternative to standard prefix-tuning or deep cross-attention.
这是一个探索性的概念验证，并非针对单体多亿参数模型的 SOTA 声明。但我认为残差 logit 增量和连续到离散的相位对齐，为标准的前缀微调或深度交叉注意力提供了一种有趣的替代方案。
Code & Architecture Diagrams: https://github.com/eladwf/topdown-semantic-vocoder
代码与架构图：https://github.com/eladwf/topdown-semantic-vocoder
Would love to hear if anyone has successfully stabilized similar hierarchical text models, or has suggestions for applying more aggressive continuous noise injection to fix the vocoder's exposure bias!
非常希望听到是否有人成功稳定了类似的层次化文本模型，或者有关于应用更激进的连续噪声注入以修正声码器暴露偏差的建议！
更进一步：量化金融体系
看懂新闻只是起点——沿量化金融路径，把它变成能交付的工程能力