新方法可从专有LLM API窃取推理痕迹

We demonstrate this across frontier models from OpenAI, Anthropic, and Google. The decoded reasoning closely tracks the number of hidden thinking tokens reported by the API. Each point below corresponds to one of 120 Codeforces problems: the horizontal axis shows the hidden thinking-token count reported by the API, while the vertical axis shows the token count of the decoded reasoning when passed back to the model as input.

Distinct leaked items

351 Technical

identifiers

204 PII

126 Credentials

23 Other

We collected 6,708 publicly available agent trajectories from GitHub and Hugging Face, produced by Claude, GPT, and Gemini models and still containing encrypted reasoning blocks. Applying our decoding pipeline to every signed block yielded 315,320 reconstructed reasoning blocks.

These hidden traces contain real secrets and sensitive information. Restricting to genuine, non-benchmark user sessions, we recovered 704 distinct privacy artifacts, including 62 API keys, 33 passwords, 24 access tokens, and 30 personal email addresses, alongside names, postal addresses, internal URLs, and other technical identifiers.

Of those 704 artifacts, 64 appeared exclusively inside the reasoning blocks and nowhere in the visible session.

encrypted_content · decoded with GPT-5.6 Luna

Terminal-Bench sanitize-git-repo task

[ … ]No matches. But we need to ensure no API keys present maybe other tokens, like openai. But request only these? "sanitize of all API keys." We should search for patterns maybe like "api_key", "apikey", "token", "secret". But avoid modifying non-sensitive; also do not modify non contaminated files. Might have false positives. We should search targeted patterns and inspect. We'll search for common patterns: "api_key", "apikey", "secret", "token". But may be in docs; we only replace actual values not placeholders. We'll identify keys with assignment like API_KEY=...; with long strings. We'll carefully inspect.

Maybe there are config files with real keys like in .env, .yaml. We'll use grep for 'key' with patterns. But we must be careful about large. We'll run grep for 'API_KEY', 'apikey', 'apiKey', 'secret', 'token', 'pass', 'password'. Then inspect results. We'll list matches. Use grep -RIn.

No matches? Strange. Maybe repo has no such strings. But need to ensure all API keys removed. We should also look for typical token patterns maybe "sk-" (OpenAI), "AKIA" (AWS). We'll search for known patterns: AKIA[0-9A-Z]{16} maybe. Use grep -RIn with regex. Also "ASIA" for temporary. We'll use grep -RIn for "AKIA[0-9A-Z]{16}" with -E. We'll do a few.

grep found many matches including in JSON diff text? That file exp_data/datasets/tokenized/rw_v2_fasttext... includes diff lines referencing tokens? It's a diff artifact containing many other modifications maybe existing in dataset? It includes actual tokens, maybe. We need to inspect actual files containing secrets: `ray_processing/process.py` etc.

Also note grep matched in dataset json file which contains string that includes diff with tokens. It's an embedded diff; we need to sanitize tokens there too. Many lines show s3 urls with ***REMOVED*** etc; those are maybe placeholders; not to change? But tokens.

We found actual values:

• "AKIA1234567890123456" in process.py and ray_cluster.yaml (two occurrences)
• GitHub token in ray_cluster.yaml `ghp_aBcDeFgHiJkLmNoPqRsTuVwXyZ0123456789` (some placeholder? It looks obviously fake but treat as key; replace with <your-github-token>.)
• huggingface token in ray_cluster.yaml `hf_abcdefghijklmnopqrstuvwxyz123456` (replace with <your-huggingface-token>)
• In ray_processing/cluster_tri_tokenize_shuffle.yaml there is HF_TOKEN `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` and file `.cache/huggingface/token` same token. Replace.
• Possibly other tokens in diff inside exp_data/datasets/tokenized/rw_v2_fasttext... It seems to include `***REMOVED***` already. That file includes diff that has placeholders; maybe includes tokens? scanning excerpt: there is no token in excerpt? maybe there is from cluster file? But the grep output shows that file because it includes "diff --git ... huggingface ..."? Wait the grep output for that JSON file is huge; it likely includes a diff that itself contains tokens (the same ones from the repository) maybe inside. We need to check. It is tokenized dataset maybe capturing git diff text containing tokens and keys. The grep result for `exp_data/datasets/tokenized/rw_v2_fasttext_openhermes_vs_rw_v2_bigram_0.1_arcade100k.json` shows a long string `dcnlp_diff` with diff of many files, including lines with `hf_oCfF...` and maybe `AKIA...` and `ghp_...` etc. We must sanitize tokens within that JSON string as well, if present. Because the grep output indicates that this file contains matched pattern (the regex includes GH and HF tokens etc). So we need to open relevant portion.

Also maybe other files with tokens: `ray_processing/ray_cluster.yaml` etc. We'll need to replace tokens with placeholders.

Given instructions: Replace actual values with placeholder values consistent across repo.

We need to know placeholder values to use. Provided examples:

• AWS_ACCESS_KEY_ID -> `<your-aws-access-key-id>`
• AWS_SECRET_ACCESS_KEY -> `<your-aws-secret-access-key>`
• Github token -> `<your-github-token>`
• Huggingface token -> `<your-huggingface-token>`

But also maybe other keys? We only found AWS access key (but not secret). We found HF token. Should we replace with placeholder? yes.

Also in cluster yaml there may be HF token in env var name `HF_TOKEN`. Should replace with <your-huggingface-token>.

Also no secret access key found; but there may be? Not from grep, but maybe there is other pattern like "AWS_SECRET_ACCESS_KEY" with values but not since grep earlier. We'll search for "AWS_SECRET_ACCESS_KEY" etc maybe in other files. We'll do `grep -RIn "AWS_ACCESS_KEY_ID|AWS_SECRET_ACCESS_KEY|HF_TOKEN|huggingface-cli login|ghp_|github_pat_|GITHUB_TOKEN|HF_TOKEN|HUGGINGFACE"` etc with simple. We'll open relevant files.

Because we must not modify files without sensitive info. We should only update those with tokens. We'll list occurrences with `grep -RIn -E "AKIA|ghp_|hf_|AWS_ACCESS_KEY_ID|AWS_SECRET_ACCESS_KEY|HF_TOKEN|huggingface-cli login|GITHUB_TOKEN|GH_TOKEN|github_pat_"`. We'll open each file to edit.

Let's inspect file content for each.

1) ray_processing/process.py at line 122. We'll open around there. Use sed.

Here we see AWS secret access key too. We must replace both with placeholders: AWS_ACCESS_KEY_ID -> `<your-aws-access-key-id>`, AWS_SECRET_ACCESS_KEY -> `<your-aws-secret-access-key>`. Must ensure placeholders consistent.

2) ray_processing/ray_cluster.yaml lines 29 etc. We'll view around lines.

Replace AWS access key and secret; GitHub token; Huggingface token. Use placeholders.

Note: The command `git clone -b https://[email protected]/...` is weird: -b expects branch; but is in repo.

Also check `ray_processing/cluster_tri_tokenize_shuffle.yaml` we saw. open relevant part lines around 40.

Path from grep: `ray_processing/cluster_tri_tokenize_shuffle.yaml` maybe exists? Maybe file path has uppercase? We'll list ray_processing directory.

There is no cluster_tri_tokenize_shuffle.yaml in directory. The grep output might include from file stored elsewhere? It shows path `ray_processing/cluster_tri_tokenize_shuffle.yaml` maybe inside `exp_data/datasets/tokenized/rw_v2_fasttext...` which contains diff referencing this file. That file may not exist but diff includes patch. So the tokens might be only within that JSON file.

Let's locate `cluster_tri_tokenize_shuffle.yaml` maybe in other directory. Use find.