MICA v0.3：引入64词整数记忆模块，首次实现长上下文利用

A few days ago I posted an update on MICA and said that the biggest problem was context.

Flame-W could generate sentences and its next-word scores were improving, but when I actually measured how much earlier context changed its predictions, the answer was basically: almost none.

So v0.3 is mostly about attacking that problem.

Quick recap for anyone new: MICA is my experimental language model built around a cellular automaton. At inference there is no Transformer, GRU, floating point network, or external LLM. A ring of integer cells is updated by learned integer rewrite rules and an integer readout produces the next-word scores.

The main change in v0.3 is memory.

The new official Flame-W is B-740, a 34 MB automaton, plus a 674 KB integer memory module.

The memory looks back over the last 64 words. It combines direct recall of previous words with a small 16-channel association representation of related words. The automaton itself is frozen while the memory readout is fitted.

And for the first time, I can actually measure MICA making meaningful use of older context.

Context-use at an 8-word distance went from essentially zero — about 0.0006 / 0.0004 on chat/everyday text — to 0.0209 / 0.0407 with memory.

That finally passes the context-use threshold I had set before running the experiment.

There was also a small prediction improvement: 5.309 → 5.268 bits/word on the no-TinyStories validation set.

The result I didn’t expect was on Tiny Theory-of-Mind.

On the 2,000-example test, Flame-W went from 28.15% to 30.95%, a +2.8 percentage-point paired improvement. On that leaderboard it puts this tiny experimental model above 6 of the 36 listed models, including Syn-2.6M at 30.30%.

That’s still nowhere near claiming MICA can “reason”. I’m treating it as evidence that giving this architecture persistent information can affect tasks that require tracking who knows what.

There is an important negative result too:

The sentences didn’t noticeably get better.

In a blind 50-prompt comparison, B-740 + memory was not significantly better than B-740 without it.

That is probably the most interesting part of v0.3 for me.

The memory can influence the final prediction, but the cellular automaton’s internal rules still mostly operate on a very short local state. In other words, MICA can now remember further back, but that memory doesn’t yet substantially change the computation happening inside the automaton.

I’ve also been experimenting with a persistent “topic register” that feeds longer-term information into rule selection. The first version beat its zero-code control but not the original model. A second version also failed to improve sentence quality.

So the next question is no longer just:

“How do I give MICA more context?”

It is:

“How do I make long-term state change which rules fire?”

That’s what I’m working on now.

For transparency, I’m continuing the same workflow I mentioned before: I direct the architecture, experiments and QA, while Claude, Codex and other AI coding agents do a large part of the implementation and experiment work. Every promoted result has to identify the exact model hash, evaluation set and integer-engine protocol, and I keep the failed experiments in the repo too.

The current code, models, experiment reports and negative results are public in Vovala14/Mica-Ai. The browser playground has also been updated to run the v0.3 model.

I’d especially like input from people who have worked on unusual recurrent/state-space architectures:

If you had a discrete rule system where long-term memory reaches the output but not the internal dynamics, how would you inject that memory into rule selection without destroying the local structure that already works?

I’m considering things like persistent routing state, memory-conditioned hashes, separate slow/fast state, or letting only some rule phases see memory.

Curious what you would test first.