GitHub工程复盘：用ModernBERT替代LLM实现低成本密钥检测

OUR PIPELINE VS. THE PREVIOUS LLM PIPELINE

Smarter secret scanning, at a fraction of the cost

higher acceptance

Owner-weighted alert precision

lower median scan execution time

Post-push scanning

lower serving cost

Compact model vs. remote LLM calls

Secret scanning works best when developers are confident in the findings. Catch too little, and real credentials can remain in a repository. Alert too often, and developers may spend time investigating strings that aren’t secrets.

Generic secrets make that balance especially difficult. The same string could be a real credential or a documentation placeholder, so detecting it requires understanding the surrounding code and text.

To address this challenge, we trained a secret classifier specifically for GitHub’s product needs: distinguishing real credentials from lookalikes while meeting the speed and efficiency requirements of scanning at scale.

In our companion post, Secret protection must scale with software, we explained why secret protection must keep pace with AI-accelerated development. Here, we go behind the scenes of the context-aware classifier powering that effort: how we designed, trained, and evaluated it to find more real secrets with less noise.

What we enabled

GitHub secret scanning operates at two key moments:

• Push protection helps prevent secrets from entering a repository by scanning changes before a push is accepted, giving developers an opportunity to address the finding.
• Post-push scanning works after code has been committed, continuously scanning repository content and creating alerts when potential secrets are detected so developers and security teams can investigate and remediate them.

We developed a fast classifier that reduced median post-push scan execution time by 86% compared with the previous LLM-based pipeline. That efficiency made it practical to extend generic secret detection to push protection, returning results before developers push changes, not only after code is committed.

Secret scanning blocks a commit with push protection after the model detects a secret on line 5 of the file.

Why secrets are hard to detect

Structured credentials often include a distinctive prefix, length, alphabet, or checksum. Generic secrets may not. Their syntax overlaps with build identifiers, checksums, asset hashes, sample values, and other non-sensitive strings. Context is the deciding signal.

These cases are hard negatives: strings that match a credential pattern but are not usable credentials. Adding more rules can suppress some of them, but each exception risks hiding a real secret. A learned classifier can instead use signals such as file path, secret type, variable names, and nearby code or text.

That is why we shipped a new scanner using a two-stage pipeline:

• Candidate generation: Regex filters scan content for potential secrets with high recall. This stage is optimized to catch broad families of credential-like patterns quickly and deterministically.
• Context-aware classification: A fine-tuned ModernBERT classifier evaluates each candidate and its surrounding context. It is trained to recognize the signals that separate real secrets from false positives, including placeholder values, examples, documentation snippets, generated content, and non-sensitive identifiers. Because the model is small and efficient to run on our inference infrastructure, it costs much less to serve and is simpler to scale than an LLM-based pipeline.

Impact:

• 11.5% user acceptance improvement, reducing false positives.
• 86% faster post-push secret detection execution time.
• More than 98% cost savings compared with the LLM-based pipeline.

How our pipeline works

As content is scanned, regex patterns cast a wide net and flag anything that looks like a credential, so real secrets aren’t missed at the start. Each candidate then goes to a fine-tuned ModernBERT classifier, which reads the value together with its file path, variable names, and surrounding code. The classifier estimates how likely it is to be a real secret. High-confidence findings become alerts, and lookalikes such as placeholders, test fixtures, and checksums are filtered out before they reach developers.

HOW OUR PIPELINE WORKS

Broad first, careful second

STEP 1

Scan

Repository content, commits, pull requests, and other scanned surfaces.

STEP 2

Generate candidates

Regex patterns identify possible tokens, keys, and credential-like strings.

STEP 3

Classify

Fine-tuned ModernBERT evaluates candidate + context and assigns likelihood of a true secret.

STEP 4

Alert or suppress

High-confidence findings move forward; likely false positives are filtered.

The important design choice is that the model does not replace deterministic scanning. Instead, it complements it. Regexes remain the right tool for finding candidate strings quickly. The classifier is used where machine learning provides the most leverage: distinguishing “looks like a secret” from “is likely a real secret.”

The classifier model makes a best effort to distinguish “looks like a secret” from “is likely a real secret.”

A hard-negative problem: secrets that almost look real

Most false positives are not random. They are hard negatives: strings that match a credential pattern but are not usable credentials. Examples include documentation placeholders, toy examples in tests, redacted values, non-sensitive IDs, and generated fixtures. These cases are easy for a regex to match and hard for a purely rule-based system to suppress without accidentally losing real secrets.

ModernBERT helps because it can read the candidate together with nearby code and text providing contextual understanding comparable to an LLM without the latency and computational overhead of generating natural-language output. For example, a candidate surrounded by words like “example,” “dummy,” “fake,” or “replace-me” may be lower risk than a candidate assigned to an environment variable in a deployment file. The classifier learns these contextual differences from labeled examples rather than requiring engineers to encode every exception as another rule.

We chose ModernBERT because it provides a strong fit for this workload: it is compact, efficient, and well suited for classification over code-adjacent text. For secret scanning, the model needs to make a calibrated decision over short candidate windows with high throughput and low latency.

That makes a fine-tuned encoder classifier a better serving shape than a general-purpose LLM for this stage of the pipeline. The model can be trained directly on our decision boundary, and evaluated using the same recall and precision metrics that matter to secret scanning quality.

How we trained

The challenge was finding representative data for cases in which syntax alone is misleading. The training and evaluation strategy combines three sources:

• Product-derived examples from GitHub repositories, using feedback from the secret detection experience and additional human validation.
• High-recall regex candidates from GitHub repository data, including human-annotated examples and weak labels used to expand coverage.
• Synthetic examples targeted at sparse and difficult cases, such as documentation secrets, test fixtures, public keys, checksums, structured lookalikes, and near-miss pairs.