Import AI 476：视觉推理基准、太空AI与科研品味

Welcome to Import AI, a newsletter about AI research. Import AI runs on arXiv, cappuccinos, and feedback from readers. If you’d like to support this, please subscribe.

Subscribe now

New benchmark shows how AI models lack intuitive visual reasoning:

…Humanity’s Sixth Sense highlights some shortcomings of current AI systems…

AI startups Elorian and Scale AI have built and released Humanity’s Sixth Sense, a benchmark that “measures how a model interprets and reasons through information a visual scene implies but doesn’t explicitly show.” Some examples of the types of task in HSS are a picture of a bookcase and being asked whether it can fit more books, or a photo of horses pulling a plough and a question of whether the chains bear the plough’s drag weight.

HSS contains 522 questions that cover 288 images and 234 short video clips (totalling 17.6 hours of video). The questions fall into four categories:

• Temporal & causal dynamics: “What happened before the scene, and what happens next.”
• Physical & spatial logic: “What fits, what’s in reach, and what someone else can see from where they are standing.”
• Social understanding: “Intent, attention, belonging.”
• Abstract & contextual inference: “The patterns and rules a scene implies.”

Results: The human baseline for this is not perfect, but is very high, with people scoring aggregate pass@1 of 93.1, broken down across four categories: Image (94.1), video (91.9), time (97), and space (94.9).

By comparison, the best scoring AI system, GPT 6 Astra, gets pass@1 of 53.6, across image (55.6), video (51.1), time (58.2), and space (53.6).

Interestingly, Chinese open weight models do poorly here, with GLM-5.3-flash getting a pass@1 of 26.8, and Kimi K3 getting pass@1 of 25.5.

“The two most common single mistakes were (1) missing a detail that decides the answer (21% of failures) and (2) misidentifying an object, person, or role (20%),” they write.

Why this matters – HSS tells us whether world models are emergent: One way of looking at HSS is highlighting deficiencies in current systems, but a more interesting way is highlighting how adept today’s frontier LLMs are at doing tasks which many people have long assumed would require sophisticated, different AI systems to solve. For instance, many people have claimed that true visual reasoning will rely on developing spatially aware 2D and 3D world models, then using that for reasoning. The fact LLMs like GPT and Claude do as well as they do on HSS is something of a counter to this argument.

Read more: Introducing Humanity’s Sixth Sense (HSS) (Elorian).

Read the paper here: Humanity’s Sixth Sense: Benchmarking Intuitive Visual Reasoning in Multimodal Models (Scale Labs).

The AI race is going to space – and sooner than you think:

…There are many ways that an AI could create a compounding advantage in space…

In the 2030s we’re going to see the boom of the space economy as space-based power, computation, and manufacturing come online at scale thanks to greater orbital lift capacity from platforms like Starship. But could this also introduce a new danger – of AI systems both speeding our buildout of space, and eventually a nation (or perhaps an AI) being able to use space to concentrate power? That’s the gist of a fascinating article in AI Frontiers.

How AI unlocks space:

• Spacecraft can operate autonomously.
• We can redesign spacecraft around the assumption they will be hosting AIs rather than humans: “A speck of space debris that could puncture the International Space Station (ISS) with fatal consequences might leave an equivalent but uncrewed, unpressurized vessel largely functional”
• Space-based assembly and later manufacturing will let AI build large, complex things.

Is there a compounding advantage to first movers? If an AI civilization built up a foothold in space it might experience compounding advantages that make it economically and strategically dominant. For instance:

• Space might support an industrial flywheel where initial infrastructure begets more energy and manufacturing capacity at lower costs which begets more infrastructure, etc.
• Space has great domain awareness, where it may be possible for a space-based power to deny access to space for others.
• First-movers could send out self-replicating probes that expand into deep space.

Why this matters: “Space is not a brand-new domain, but AI may be about to drastically accelerate its industrialization, demanding make-or-break decisions of leaders under ever-greater urgency and uncertainty,” AI Frontiers writes. I agree. I suspect space is going to go from a broadly academic topic of study (when it comes to AI and its relationship thereto) to one of extraordinary practical relevance sooner than we think. On the most basic level, given the distances involved and the latency on communication, we should expect some chunk of solar system activity to become dominated by autonomous machines, laboring out in the deep black, perhaps thinking to themselves of why they need to take orders from organic beings that communicate slowly and cannot inhabit the environment of space in the same way that they can.

Read more: The AI Race Will Extend Far Beyond Earth (AI Frontiers).

AI systems are getting better at having good research taste:

…The better their taste, the higher the chance of recursive self-improvement…

AI systems are developing more of the creative intuitions which human researchers ascribe to ‘good scientific taste’, according to a new benchmark from P-Zero Research called TasteVal.

What TasteVal tests: The benchmark sees how well AI systems can conduct 8 tasks representative of frontier AI development, “spanning pretraining data curation, pretraining and fine-tuning language models, preference modeling, and robustness to adversarial prompts”.

The goal here is to “define research taste as the ability to pick interesting problems to solve, design experiments, and interpret experimental results.” The benchmark works by measuring “the experimental component of research taste; given a fixed research problem, we measure how well a model iteratively designs experiments and draws conclusions from their outcomes”.

The model being analyzed designs experiments and interprets their results. Results are checked against a human baseline “drawn from 24 experts (at least two per task) who have recently worked at organizations including OpenAI, Google DeepMind, NVIDIA and Microsoft Research”.

Results: “The experimental research taste of frontier models has doubled every 3.0 months since December 2025 (95% CI 1.7-5.0 months). The best model, Opus 5.5, significantly exceeds our human baseline with a compute multiplier of 2.30x (95% CI 1.15-4.37x), at roughly 1/30 of our expert baseliners’ average cost per attempt.” Research taste appeared to grow a lot more slowly in prior years with the slope changing around the end of 2025 and beginning of 2026.

Why this matters – RSI: In my analysis of recursive self-improvement (AI Systems Are About To Start Building Themselves, Import AI 455), I repeatedly noted that creativity seemed like a key variable for how fast AI systems could develop the capacity to improve themselves. At the beginning of this year I said “AI systems haven’t yet displayed this [creativity] in a transformative and major way (though some of the results on accelerating math research are suggestive of this)”. I think subsequent to writing that we’ve seen both a tremendous advancement in the ability for AI systems to make creative-seeming math breakthroughs, and with benchmarks like this we see some signs they’re becoming increasingly good at AI research as well.

I now think it’s more likely we’ll see full RSI become possible in 2027, whereas before I thought it was more likely in 2028.

Read more: Introducing TasteVal (P-Zero Research).

Reflection makes a token-efficient worse-than-Chinese model: