Cactus Compute 发布 45M 参数工具调用模型 Needle 2

Cactus Compute has released Needle 2, an open 45M-parameter model for tool calling, device use, and structured extraction. The entire model ships as a single 14MB binary that runs a full session in about 28MB of RAM. Weights are trained and deployed at CQ2-bit using Cactus Quants, and the model is sealed inside the company’s own C++ engine, so there is no runtime to install and no download at inference time. Reported decode throughput is 500 tokens/sec on a Raspberry Pi 5, 400–1,500 tokens/sec on Meta Quest 3S and Apple Vision Pro, and 300–700 tokens/sec on sub-$200 phones. The design premise is narrow and stated plainly by the team: mapping a messy sentence onto a typed function signature needs no world knowledge and no open-ended prose. That framing is why 45M parameters are enough here, and why the model targets hardware with no GPU and no NPU.

Is it deployable?

Yes, Needle 2 ships as prebuilt binaries and a static library for macOS, Linux (x86-64, ARM64, ARMv7, RISC-V, MIPS32el), Windows, Android, iOS/watchOS/tvOS, and WebAssembly. Cactus says Pebble already runs Needle locally in the Index 01 app for offline voice actions.

• Which companies: Any team shipping firmware or apps on constrained hardware. Seed-stage wearable and IoT startups, mid-market consumer-electronics OEMs, robotics teams, and large device makers needing an offline fallback. Cloud-first SaaS teams gain less.
• Industries: smart home, wearables, low-end mobile, automotive in-cabin control, service robotics, retail kiosks and POS, routers and IP cameras, and regulated settings where audio cannot leave the device.
• Applications: voice-to-action on screenless devices, offline appliance control, receipt and invoice field extraction, enum tagging, and local routing that escalates to the cloud only on low confidence.

Architecture: Simple Attention Network

Needle 2 uses what Cactus team calls a Simple Attention Network. The recipe replaces the FFN with a Hadamard MLP, keeps GQA attention, adds engram key-value memory from hashed n-gram tables, and uses multi-lane hyper-connections. The network is 27 layers and 512 wide. The underlying study is on arXiv as A Controlled Study of Attention-Only Transformers.

Pretraining used a proprietary 115B-token corpus, with 38B tokens of post-training. The research team notes LFM2.5-230M was pretrained on 19 trillion tokens.

Needle 2 spends 70 MFLOPs per token, with 35M of 45M parameters matmul-active. LFM2.5 230M spends 460, FunctionGemma 270M spends 540, and Apple FM sits near 6,000.

Engine, grammar, retrieval, and confidence

Weights never decompress into RAM. The 2-bit codes expand inside vector registers and fuse into integer dot products, so the arithmetic path stays int8. One binary probes the CPU at startup and selects a kernel tier: SDOT, NEON, AVX2, RISC-V vectors, wasm SIMD, or scalar.

A byte-level grammar compiled from your JSON schemas constrains every emitted token. Because the matcher knows which tokens are legal before logits exist, the engine skips up to 98% of the vocabulary projection on structural tokens.

Attention uses a 256-token sliding window, and the system turn plus tool declarations are pinned as KV sinks. Memory stays near 28MB regardless of conversation length.

Declare five or fewer tools and they render directly. Above five, a contrastive retrieval head embeds each schema once, scores the query per turn, and admits only the top five. Unselected tools are unreachable, not merely unlikely.

Every response carries a confidence value, the minimum of a calibrated post-hoc head and the decoding probability of the call tokens. Off-topic requests return the empty call []. The contract is a threshold: act above it, re-ask or escalate below it.

Evaluation

Cactus team evaluates on five public function-calling benchmarks using ordered strict exact match, where names, call order, and every argument must match. Needle 2 runs end-to-end through the shipped engine at CQ2-bit with retrieval on; baselines run f16 under vLLM.

Needle 2 leads both Seal-Tools splits and posts 98.3 function-name accuracy on Mobile Actions. It trails on BFCL v4, which Cactus attributes to distribution: its corpus is consumer device actions, not general or enterprise APIs. Well-formed output rate across the 3,641 BFCL rows is 93.4. The team states two asymmetries upfront: f16 baselines favor them, and task specialization favors Needle.