精选8501:57Hacker News Best（web_list）技巧与观点

125M 端侧模型实时续写钢琴曲，关键在 MIDI 表示与数据清洗

Show HN：125M 模型在端侧自动续写钢琴曲

推荐理由

做端侧生成模型或音乐 AI 的同学必看，作者把 MIDI 表示、数据清洗、训练与后处理的完整取舍都讲透了，可直接照搬这套流程到自己的序列生成任务上。

TL;DR: I trained a 125M-parameter transformer to autocomplete piano performances in real time (~108 notes/sec on an iPhone 15). The biggest improvements came from finding the right MIDI representation, cleaning the training data aggressively, and adding DPO post-training.

TL;DR：我训练了一个 1.25 亿参数的 transformer，用于实时自动续写钢琴演奏（在 iPhone 15 上约每秒 108 个音符）。最大的改进来自于找到合适的 MIDI 表示、积极清理训练数据，以及加入 DPO 后训练。

Almost a year ago, I started tinkering with an idea: connect my MIDI piano to my phone, play something, and have AI autocomplete the song for me. Think GitHub Copilot, but for piano.

大约一年前，我开始琢磨一个想法：把我的 MIDI 钢琴连接到手机上，弹奏一些内容，然后让 AI 为我自动续写歌曲。想象一下 GitHub Copilot，但用于钢琴。

It turned out to be a deeper rabbit hole than I expected. Fourteen experiments later, it is finally at a point where I am happy enough with it to write about.

结果这比我预想的要深得多。经过 14 次实验，它终于达到了一个我足够满意、愿意写下来的程度。

Your browser does not support the video tag.

您的浏览器不支持视频标签。

Potato-quality video because the good phone was busy running the MIDI model.

视频画质较差，因为好手机正忙着运行 MIDI 模型。

The app, RollTab, is available for free here if you have a MIDI keyboard and an iPhone/iPad. 1

这款应用 RollTab 可在此免费获取，如果您有 MIDI 键盘和 iPhone/iPad。1

A few sound samples

一些音频样本

Each audio starts with a short prompt, followed by the model's continuation.

每段音频以简短提示开始，随后是模型的续写。

Pokémon, Pallet Town (8-note prompt)

宝可梦，真新镇（8 音符提示）

Your browser does not support the audio tag.

您的浏览器不支持音频标签。

Final Fantasy VI, Terra's Theme (16-note prompt)

最终幻想 VI，蒂娜的主题（16 音符提示）

Your browser does not support the audio tag.

您的浏览器不支持音频标签。

Für Elise (16-note prompt)

致爱丽丝（16 音符提示）

Your browser does not support the audio tag.

您的浏览器不支持音频标签。

What’s in a MIDI File?

MIDI 文件中有什么？

A MIDI file is quite different from an MP3 or other audio formats. Rather than storing recorded sound, it stores music as a sequence of events: a key is pressed at a certain pitch and velocity, a key is released, the sustain pedal changes state, and so on. Other events include switching instruments or changing volume.

MIDI 文件与 MP3 或其他音频格式有很大不同。它不是存储录制的声音，而是将音乐存储为一系列事件：按键以特定音高和力度按下，按键释放，延音踏板改变状态，等等。其他事件包括切换乐器或改变音量。

These events are often organised into multiple tracks. A pop or game MIDI might have melody, chords, bass, drums, strings, and several synth parts. This project is focused on piano continuation, so I mostly kept piano-like material and removed or reduced the rest.

这些事件通常组织成多个音轨。流行或游戏 MIDI 可能有旋律、和弦、贝斯、鼓、弦乐和几个合成器部分。这个项目专注于钢琴续写，所以我主要保留类似钢琴的材料，并移除或减少其余部分。

How Do You Tokenize Music?

如何对音乐进行分词？

To train a transformer on these performances, I first needed to turn the MIDI events into a discrete sequence the model could read and predict. The most obvious mapping is to make a token for every MIDI event:

为了在这些演奏上训练 transformer，我首先需要将 MIDI 事件转换为模型可以读取和预测的离散序列。最直接的映射是为每个 MIDI 事件创建一个 token：

代码 · 3 行

NOTE_ON_60_80 # {pitch}_{velocity}
NOTE_OFF_60   # {pitch}
TIME_SHIFT_12 # {time step}

代码 · 3 行

NOTE_ON_60_80 # {pitch}_{velocity}
NOTE_OFF_60   # {pitch}
TIME_SHIFT_12 # {time step}

If you include pitch and velocity directly in a NOTE_ON token, the vocabulary can grow quickly. There are 128 MIDI pitches and 128 velocity values, so the naive combined note-on vocabulary has up to:

如果你直接在 NOTE_ON token 中包含音高和力度，词汇量会迅速增长。有 128 个 MIDI 音高和 128 个力度值，所以简单的组合音符开启词汇量最多有：

代码 · 1 行

128 * 128 + 128 = 16,512

代码 · 1 行

128 * 128 + 128 = 16,512

tokens just for note-on and note-off. In practice you would probably bucket velocity, but the basic issue remains: many combinations are rare, and the model has to learn a lot of structure from sparse tokens.

仅音符开启和音符关闭的 token。实际上你可能会对力度进行分桶，但基本问题仍然存在：许多组合很少见，模型必须从稀疏的 token 中学习大量结构。

A common improvement is to factor the representation with a grammar:

一个常见的改进是使用语法对表示进行分解：

代码 · 1 行

[NOTE_ON, PITCH, VELOCITY] | [NOTE_OFF, PITCH] | [TIME_SHIFT, DURATION]

代码 · 1 行

[NOTE_ON, PITCH, VELOCITY] | [NOTE_OFF, PITCH] | [TIME_SHIFT, DURATION]

Now the output spaces are smaller:

现在输出空间更小了：

代码 · 4 行

NOTE_ON / NOTE_OFF / TIME_SHIFT
PITCH: 128 values
VELOCITY: ~16
DURATION: ~100

代码 · 4 行

NOTE_ON / NOTE_OFF / TIME_SHIFT
PITCH: 128 values
VELOCITY: ~16
DURATION: ~100

You can enforce the grammar during generation by masking invalid next tokens. After NOTE_ON, only pitch tokens are valid. After pitch, only velocity tokens are valid. This guarantees syntactically valid output.

你可以在生成过程中通过屏蔽无效的下一个标记来强制语法。在NOTE_ON之后，只有音高标记是有效的。在音高之后，只有力度标记是有效的。这保证了语法上有效的输出。

I tried note-on/note-off style representations, but my models tended to drift. They would forget to emit note-off, leave hanging notes, or lose track of active state. That was especially bad for my target: a small model running close to real time on a laptop or phone.

我尝试过音符开/关（note-on/note-off）风格的表示，但我的模型容易漂移。它们会忘记发出音符关（note-off），留下悬空的音符，或者失去对活动状态的跟踪。这对我目标特别不利：一个在笔记本电脑或手机上接近实时运行的小模型。

Another representation I tried was closer to:

我尝试的另一种表示更接近：

代码 · 1 行

[NOTE, PITCH, VELOCITY, DURATION] | [TIME_SHIFT, DURATION]

代码 · 1 行

[NOTE, PITCH, VELOCITY, DURATION] | [TIME_SHIFT, DURATION]

This avoids note-off drift because note duration is explicit. The time shift token advances the playhead when no note is played.

这避免了音符关（note-off）漂移，因为音符持续时间是显式的。时间移位标记在没有音符播放时推进播放头。

This worked better musically, but it was slow. One musical note took roughly four autoregressive transformer steps. It also burns through the context window quickly.

这在音乐上效果更好，但速度慢。一个音符大约需要四个自回归变换器步骤。它也很快消耗上下文窗口。

The final representation

最终表示

The representation I eventually settled on was:

我最终确定的表示是：

代码 · 1 行

NOTE(pitch, delta_onset, duration, velocity)

代码 · 1 行

NOTE(pitch, delta_onset, duration, velocity)

There is no separate TIME_SHIFT event in the final version. Silence is represented by delta_onset on the next note: the time since the previous note onset.

最终版本中没有单独的TIME_SHIFT事件。静音由下一个音符的delta_onset表示：自上一个音符开始以来的时间。

For example:

例如：

代码 · 2 行

NOTE(C4, delta=0,  duration=12, velocity=80)
NOTE(D4, delta=24, duration=12, velocity=80)

代码 · 2 行

NOTE(C4, delta=0,  duration=12, velocity=80)
NOTE(D4, delta=24, duration=12, velocity=80)

means: play C4, wait 24 time steps before the next note onset, then play D4.

意思是：播放C4，等待24个时间步长后再开始下一个音符，然后播放D4。

Chords are represented as multiple notes with delta_onset = 0, sorted by pitch2:

和弦表示为多个音符，delta_onset = 0，按pitch2排序：

代码 · 3 行

NOTE(C4, delta=24, duration=24, velocity=80)
NOTE(E4, delta=0,  duration=24, velocity=78)
NOTE(G4, delta=0,  duration=24, velocity=82)

代码 · 3 行

NOTE(C4, delta=24, duration=24, velocity=80)
NOTE(E4, delta=0,  duration=24, velocity=78)
NOTE(G4, delta=0,  duration=24, velocity=82)

It's also not a flat token stream like:

它也不是像这样的扁平标记流：

代码 · 1 行

NOTE, PITCH, DELTA, DURATION, VELOCITY

代码 · 1 行

NOTE, PITCH, DELTA, DURATION, VELOCITY

Instead of spending four transformer passes generating the attributes of a note, the transformer advances the music by one complete note at a time. In practice, this gets the large model to about 108 notes/second on an iPhone, well above anything a human would need for live playing.

变换器不是花费四次变换器传递来生成音符的属性，而是每次将音乐推进一个完整的音符。在实践中，这使大模型在iPhone上达到约108个音符/秒，远高于人类现场演奏所需的任何速度。

Internally each note has five categorical fields, each with its own vocabulary3, with timing quantized to fixed steps.4

内部每个音符有五个分类字段，每个字段有自己的词汇表3，时间量化到固定步骤4。

代码 · 1 行

[event_type, pitch_id, delta_id, duration_id, velocity_id]

代码 · 1 行

[event_type, pitch_id, delta_id, duration_id, velocity_id]

Each field gets its own embedding. The note token is the sum of all the embeddings:

每个字段有自己的嵌入。音符标记是所有嵌入的总和：

代码 · 6 行

note =
    event_type_embedding[NOTE]
  + pitch_embedding[C4]
  + delta_embedding[12]
  + duration_embedding[24]
  + velocity_embedding[80]

代码 · 6 行

note =
    event_type_embedding[NOTE]
  + pitch_embedding[C4]
  + delta_embedding[12]
  + duration_embedding[24]
  + velocity_embedding[80]

The model then has separate output heads: pitch, delta, duration, and so on.

然后模型有单独的输出头：音高、增量、持续时间等。

There is a small nested decoder between the fields, so later fields can condition on earlier predicted fields. But the expensive transformer backbone runs only once per note, not once per field.

字段之间有一个小的嵌套解码器，因此后面的字段可以依赖于先前预测的字段。但昂贵的变换器骨干网络每个音符只运行一次，而不是每个字段一次。

Sustain Pedal

延音踏板

As you might know, pressing down the sustain pedal on a piano makes notes play even after you release them. I didn't want to muddy the implementation with adding sustain pedal events. Instead, sustain is baked into note duration during preprocessing.

你可能知道，按下钢琴上的延音踏板会使音符在你释放后仍然播放。我不想通过添加延音踏板事件来使实现变得混乱。相反，延音在预处理期间被烘焙到音符持续时间中。

If the key is released while the sustain pedal is down, the note is extended to the pedal-up time. If the same pitch is played again first, the earlier note is cut off at the retrigger. The result is a note duration that approximates the actual sounding duration.

如果在延音踏板踩下时松开琴键，音符会延长至踏板抬起时。如果先再次弹奏相同音高，则较早的音符会在重新触发时被切断。结果是音符时长接近实际发声时长。

This loses the explicit pedal gesture, but it makes the modeling problem much simpler: the model only has to predict pitch, onset, duration, and velocity.

这失去了明确的踏板动作，但使建模问题简单得多：模型只需预测音高、起始、时长和力度。

Dataset

数据集

I searched through a lot of publicly available datasets and collections, focusing mostly on older classical music in the public domain. The quality varied wildly, so I ended up writing quite a few cleaning scripts.

我搜索了大量公开可用的数据集和收藏，主要关注公有领域的较老古典音乐。质量差异很大，所以我最终编写了不少清理脚本。

The final dataset contained a few hundred thousand MIDI files, representing roughly 300 million note events.

最终数据集包含数十万个 MIDI 文件，代表约 3 亿个音符事件。

The final pipeline:

最终流程：

selected piano-focused material
removed or reduced pathological multi-track mixtures
filtered by density and pitch/time coverage
deduplicated by fingerprints that ignore global transposition and uniform tempo changes
grouped alternate versions of the same composition into the same split

选择以钢琴为重点的材料
移除或减少病态的多轨混合
按密度和音高/时间覆盖范围过滤
通过忽略全局移调和统一速度变化的指纹进行去重
将同一作品的不同版本分组到同一分割中

I tried scaling the dataset to roughly 5x the size, hoping it would improve performance, but the resulting models were worse. Cleaning and selecting the data mattered more than simply adding more of it.

我尝试将数据集扩展到约 5 倍大小，希望提高性能，但结果模型更差。清理和选择数据比简单增加数据量更重要。

Training

训练

Initially, training is just cross-entropy over the five output heads, summed together:

最初，训练只是五个输出头的交叉熵之和：

代码 · 5 行

type_loss
+ pitch_loss
+ delta_loss
+ duration_loss
+ velocity_loss

代码 · 5 行

type_loss
+ pitch_loss
+ delta_loss
+ duration_loss
+ velocity_loss

This makes it easy to track pitch, duration, and velocity accuracy separately, rather than relying on a single aggregate next-token loss.

这使得分别跟踪音高、时长和力度准确性变得容易，而不是依赖单一的聚合下一个标记损失。

Still, the training objective has an important limitation: music continuation does not have a single correct answer. A held-out song only gives the model one "correct" next note, even though there are often many continuations that would work musically. Cross-entropy is useful for learning the mechanics of music, but not a great proxy for how good a full continuation sounds.

尽管如此，训练目标有一个重要限制：音乐续写没有单一正确答案。一个保留的歌曲只给模型一个“正确”的下一个音符，尽管通常有许多在音乐上可行的续写。交叉熵对学习音乐机制有用，但不是衡量完整续写听起来好坏的好代理。

Augmentation

增强

Augmentation was important because the live input is not a pristine MIDI file. It is me playing piano, badly enough that notes might be slightly early, late, too hard, etc.

增强很重要，因为现场输入不是原始的 MIDI 文件。它是我弹钢琴，弹得不够好，音符可能稍早、稍晚、过重等。

更进一步：量化金融体系

看懂新闻只是起点——沿量化金融路径，把它变成能交付的工程能力

进入量化体系 →

话题#端侧模型 #MIDI #Transformer #DPO #工程实践

125M 端侧模型实时续写钢琴曲，关键在 MIDI 表示与数据清洗

A few sound samples

一些音频样本

What’s in a MIDI File?

MIDI 文件中有什么？

How Do You Tokenize Music?

如何对音乐进行分词？

The final representation

最终表示

Sustain Pedal

延音踏板

Dataset

数据集

Training

训练

Augmentation

增强

更进一步：量化金融体系

相似阅读