Qwen与Mistral复现模型原生数值记忆提取机制

Two different 7B transformers. Two different internal coordinates. The same memory mechanism.

I started AKBASCORE MAM on Qwen2.5-7B-Instruct. I have now independently localized and replicated the mechanism on Mistral-7B-Instruct-v0.3.

Final Mistral result: 127/128 correct memories at Top-1 — 99.22%.

Counterfactual retrieval: 125/128 — 97.66%.

Shifted-pointer control: 0/128.

The model weights were not changed. No fine-tuning. No LoRA. No optimizer. No learned router. No gold B-memory ID is supplied to the retriever. At query time, none of the 128 candidate B memories is forwarded through Mistral.

But I don't think 99.22% is the most interesting part of this experiment.

The more important question is: what exactly is the model reading?

Because this system is not searching through 128 text documents in the conventional sense. (Fig. 1)

• The text enters the model once

When a memory is formed, its source text is processed through the frozen transformer.

The transformer already produces K and V states for its own attention computation. I do not treat selected parts of those states merely as temporary computational residue. I extract them as model-native numerical memory.

So instead of only having a sentence such as:

“Instrument Zyrhyn carries seal PCI.”

we now have a numerical memory structure derived directly from the model's own internal computation.

• That numerical structure can exist outside the model

The memory does not have to remain inside the model weights.

The numerical structure can be held in RAM, serialized, written to a file, and therefore moved to persistent storage.

This distinction matters.

When people say “persistent LLM memory,” the usual architecture often means storing text, documents, embeddings, database records, or summaries and later retrieving them so the model can read them again.

The path I am investigating is different:

instead of storing human-readable information so the model can reread it, store a machine-native numerical state derived from the model itself.

The model weights remain frozen.

• Then how does the model know which numerical memory to retrieve?

This is the part I find most interesting.

A new question arrives:

“What seal does instrument Zyrhyn carry?”

The active numerical A memory is installed into the transformer cache.

Mistral processes the new question.

A particular channel of the frozen model's own attention mechanism then produces a question-conditioned distribution over the A-memory positions.

For Mistral, that channel is:

L28H00.

I call this endogenous attention pointer ÇAĞRIİZ.

It is not an external neural router.

It is not a trained classifier.

It is not a gold memory ID.

It comes from the transformer's own native:

attention computation.

• The pointer is then applied to V space

This is where K and V take different operational roles.

K helps answer:

Where should I look?

V helps answer:

What numerical address should I construct from what I found?

For Mistral, the independently localized address space is:

L00-V, uncentered.

8 KV heads × 128 dimensions:

1024 dimensions.

Applying the ÇAĞRIİZ distribution to those V states produces a question-conditioned 1024-dimensional address.

At this point there is no filename telling the retriever which B memory to choose.

There is no B-memory ID supplied by the experiment.

There is a 1024-dimensional retrieval address derived from the frozen model's own internal computation. (Fig. 3)

• Then all 128 B memories compete

Every B memory has already been independently converted into its own numerical structure.

Each retains token-level V rows in the localized address space.

The live 1024D address is compared against every B memory.

For each B candidate, the system computes the maximum cosine similarity between the live address and that candidate's numerical rows.

Then:

128 candidates → 128 scores → argmax → one associated memory. (Fig. 2)

In the real Item 001 demo run:

Expected:

Returned:

Rank:

Top-1 cosine:

Second candidate:

Margin:

And the number of Mistral model forwards required for that retrieval was:

The model processed:

27 question tokens over a preinstalled 32-slot A cache.

Model forwards through the 128 candidate B memories:

That distinction is important.

After receiving the question, the system did not send 128 source texts back through Mistral to find the answer.

• Then I changed the memory

I changed the association for the same Zyrhyn record:

The model remained the same.

The weights remained the same.

The question remained the same.

The B-memory bank remained the same.

The changed A memory was re-forged and the same frozen retrieval mechanism was run again.

This time the system selected:

Rank:

So changing the numerical A association redirected the same frozen mechanism to a different B memory. (Fig. 4)

• This was not evaluated on only one showcase item

For the final Mistral experiment, TEST528, the mechanism was frozen before the final evaluation panel.

The result was:

Primary retrieval: 127/128 — 99.22%

Counterfactual retrieval: 125/128 — 97.66%

Controls:

Shifted pointer: 0/128

NO-A: 1/128

The single NO-A “success” has a simple implementation-level explanation: a zero address produces equal zero scores for every candidate, so the deterministic tie-break selects B#001. I therefore do not interpret it as meaningful memoryless retrieval. (Fig. 5)

The failed cases are preserved in the experimental record.

• Qwen did not use the same coordinates

This is probably the most important result of the cross-model experiment.

Qwen2.5-7B-Instruct:

Mistral-7B-Instruct-v0.3:

I did not copy Qwen's layer/head coordinates into Mistral.

Mistral's pointer and address regions were independently localized and then frozen before the final evaluation.

So the strongest statement I think the current evidence supports is:

The mechanism transferred across model families; the internal coordinates did not.

Two models do not establish universality.

But this is also no longer a result tied to one accidental coordinate in one transformer.

So where is the “persistent machine memory” part?

I want to be precise about this because it is easy to overstate what the current prototype demonstrates.

This experiment does not demonstrate 150 million memories.

It does not yet demonstrate a production-scale persistent memory database.

It does not yet demonstrate the model discovering the first A memory automatically from an enormous inactive bank.

What it does establish is a smaller but, in my view, more fundamental chain:

Model-derived numerical memory can be extracted.

Numerical memory can exist outside the model weights.

Active numerical memory can be connected back to transformer computation.

A natural-language question can produce a pointer through the model's own attention.

That pointer can become a model-native V-space address.

That address can select an associated numerical memory from an external bank.

Once these numerical structures are serialized, whether the storage medium is RAM, SSD, or another persistent storage layer becomes primarily a systems-engineering question.

The harder problem is not writing an array of numbers to a hard disk.

The harder problem is:

How does the frozen model later determine which machine-native numerical memory it wants back?

That is the connection I am testing here.

And this is where I prefer to stop the speculation and let the architecture speak for itself.

If model-native memory states can exist outside model weights, persist in storage, and later be addressed by signals generated endogenously from natural language inside the transformer, then is long-term machine memory necessarily limited to:

find old human-readable information → put it back into context → make the model read it again?