H公司发布NeoMME：去视觉塔的单塔多模态编码器

Most visual document retrievers in production today are hand-me-downs. ColPali and the models that followed it take a generative vision-language model and repurpose it as an encoder. The result still carries a separately pretrained vision tower and a causal decoder that never generates a token. That is parameter and compute overhead for a task that only needs representations.

H Company has released NeoMME, a family of 260M and 800M bidirectional encoders that drops both components. One Transformer processes multilingual text tokens and raw 32×32 RGB image patches through the same layers, trained from random initialization. The retrieval fine-tune, NeoMME-Retriever, reaches 0.523 nDCG@10 on ViDoRe v3 at 260M parameters.

Is it deployable? Yes. Every checkpoint ships under Apache 2.0 with day-zero support in Hugging Face Transformers. The 260M model indexes 51.3 pages per second on a single NVIDIA L40S and encodes a query in 78.3 ms on a CPU-only host.

https://arxiv.org/pdf/2609.01657

One tower, two modalities

Text enters through an ALBERT-style factorized embedding: a 256-dimensional lookup projected to model width. Images are split into non-overlapping 32×32 patches and projected by a 2-layer MLP trained from scratch. No patch-merging module, no SigLIP2 tower.

Both models support a 16,384-token context, enough for two standard 3,840×2,160 4K UHD images after patching. Most layers use symmetric sliding-window attention; every sixth layer and the final layer attend globally. The stack uses grouped-query attention, query-key normalization, gated attention, 2D rotary position embeddings, and squared-ReLU MLPs. Exact parameter counts are 262,937,906 and 793,715,032.

The tokenizer is a whitespace-unconstrained BPE with a 131,072-entry vocabulary, trained from scratch. Across 14 target languages in FLORES-200 devtest, it emits 44.4% fewer tokens than ModernBERT.

Trained as a masked diffusion denoiser

Pretraining is discrete masked diffusion over text, optionally conditioned on visible image patches. Text-only segments draw a corruption rate uniformly from 0 to 1. Multimodal segments draw from 0.30 to 1, which removes the language-only shortcut and forces the model to read the page.

A cross-modal ablation probe confirms this works. At 90% masking, visible page patches raise masked-token accuracy by 38.4 points for the 260M model and 40.5 points for the 800M model. Each run processes about 524 billion packed input tokens, roughly 290 billion of them text-only, on 16 and 32 H100 accelerators respectively.

Retrieval results

NeoMME-Retriever adds two jointly trained heads on the shared backbone: a mean-pooled dense head with Matryoshka widths, and a late-interaction head projecting every token and patch to 128 dimensions. One forward pass returns both.

On ViDoRe v3, the 260M model scores 0.523 nDCG@10 and the 800M model 0.556. The 260M result sits within 0.002 of ColQwen2.5-v0.2 at 3.75B parameters, and 26.1 points above the best other sub-300M model. The 800M model lands 0.9 points behind the similarly sized Vultron Retriever Flash. On ViDoRe v1 and v2 the models reach 0.860/0.522 and 0.874/0.559 nDCG@5.

Text retrieval is weaker. On BEIR-15, late interaction reaches 0.4881 and 0.5126, against 0.5722 for LateOn at 149M parameters. The authors attribute this partly to supervision scale: NeoMME saw roughly 430K text query examples, against roughly 660M contrastive examples for mLateOn.

Storage and throughput

Late-interaction indexes are expensive. A 2048×2048 page yields 4,162 vectors, about 1.5 MB per ViDoRe v3 document in float32. Two methods bring that down. Hierarchical token pooling at factor 10 with int8 queries and documents gives 39.0 kB per page, a 39.4× reduction retaining 99.16% of baseline nDCG@10. Pool factor 8 with int8 queries and binary documents gives 6.0 kB, a 255.5× reduction retaining 95.19%.

Indexing is fast for the vector count. At a matched 2048×2048 input on one L40S, NeoMME-260M encodes 51.3 pages per second against ColModernVBERT’s 26.0, a 1.97× gap.

Interactive explainer

Key Takeaways

• One bidirectional Transformer handles text and raw image patches, with no vision tower and no decoder.
• NeoMME-Retriever-260M scores 0.523 nDCG@10 on ViDoRe v3, beating every evaluated model below 800M.
• It matches 3.75B-parameter ColQwen2.5 on ViDoRe v3 while being 14.4× smaller.
• Token pooling plus asymmetric quantization cut the index from roughly 1.5 MB to 6 kB per page.
• Text-only retrieval and frozen natural-image transfer remain clear weak spots.

Check out the Paper, Model Collection and Demo. Also, feel free to follow us on Twitter and don’t forget to join our 150k+ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well.

Need to partner with us for promoting your GitHub Repo OR Hugging Face Page OR Product Release OR Webinar etc.? Connect with us

The post H Company Releases NeoMME: A Family of 260M and 800M Single-Tower Multimodal Encoders That Drop the Vision Tower and Causal Decoder appeared first on MarkTechPost.