LLM实验避坑：利用已有结果文件重评分析，避免无效GPU消耗

The instinct when you spot a confound is to queue another run. Three times this week the answer was already in files the author had published, and the rerun would have cost hours of GPU for a number they could have had in an afternoon.

u/Electronic_Put4530 writes facts into the n-gram side table of Qwen3.8-Flash-Next, no weights touched.

https://www.reddit.com/r/LLMDevs/comments/1wm7lbf/

Those rows are keyed by exact tokens, so how a name is spelled matters more than how the question is phrased, and a name at the start of a sentence is a different token from the same name mid-sentence. That means different rows. He hadn't run a name-perturbation test, but his probes record where the name falls, so the split was already in the published results: 0.881 over 219 sentences with the name at the start, 0.826 over 622 with it inside. He then recomputed it inside each question family, across the published overlay, a newer one, and a rerun of the newer recipe with a different seed. The published overlay favours the sentence-initial name in all four families where the split exists, by 0.05 to 0.13. The newer one reverses in three of those four, and the second seed repeats the pattern with the same exception. The headline average barely moves while the map underneath it changes. So the gap isn't a property of the method. It's a readout of which rows the training budget bought, and it costs nothing to print on every run.

u/Micha0827 moved a local Qwen from 89% to 100% by putting the question before the context.

https://www.reddit.com/r/LocalLLaMA/comments/1wmd3xx/

On 38 situations that is four cases flipping. A per-case diff of runs he already had showed all four were items where a distractor comes first and the decisive detail comes last, which is where the mechanism predicts it should help, and it also turned up a German case that the change breaks. The claim got smaller and much harder to argue with, without a single new inference.

u/Hungry-Horror-7577 reported 68% faithfulness on a gold set and then found his judge was only ever shown the first 600 characters of each retrieved note, against a median note length near 1,500.

https://www.reddit.com/r/Rag/comments/1wfzp1a/

He re-judged the same fixed answers at a 2,000-character cap, no regeneration, and bucketed by note length. Where no note exceeded 600 characters the score moved 9 points and ten of eleven questions didn't move at all. Where five or more notes were long it moved 33. Monotone in the direction truncation predicts, which is what turns a suspicion into a cause.

The move underneath all three is the same. Before you queue a rerun, ask what each row of your results file already records about its input, whether the effect you're chasing is a split of something you already stored, and whether you can re-score fixed outputs instead of regenerating them. That last one matters more than it looks at small n, because re-scoring holds generation noise fixed, and at 38 or 83 cases that noise is usually the same size as the effect.

The limit is that you can only split on what you stored, so the value gets decided at logging time rather than analysis time. Which is an argument for recording the boring fields before you have a hypothesis: position, length, question shape, source document, which prompt template, which seed.

What's the field you wish past-you had logged? Mine is the template id, every time.