01Spatial reasoning空间推理

How can a model ground language in 3D space, and from whose viewpoint?

Most 3D vision-language benchmarks name the target object in the query, so a model can succeed by matching names instead of reasoning about space. My work studies language-guided spatial reasoning in 3D scenes: relative position, absolute distance, narrative and parametric perspective, and the viewpoint a query assumes.

SURPRISE3D defines 3D spatial reasoning segmentation and provides 200k+ queries over 900+ ScanNet++ scenes with no object names. MLLM-For3D transfers the reasoning of frozen 2D multimodal LLMs to 3D segmentation without any 3D annotation. EgoAudit tests whether 3D vision-language models infer a speaker's viewpoint or only follow one they are given.

Papers

SURPRISE3D: A Dataset for Spatial Understanding and Reasoning in Complex 3D Scenes

Jiaxin Huang^*, Ziwen Li^*, Hanlue Zhang^*, Runnan Chen, Zhengqing Gao, Xiao He, Yandong Guo, Wenping Wang, Tongliang Liu, Mingming Gong

NeurIPS 2025, Datasets & Benchmarks Track · first author, project leader

arXiv Proceedings Project page Code Dataset Poster (PDF)

A benchmark for 3D spatial reasoning segmentation: 200k+ vision-language pairs over 900+ ScanNet++ scenes, including 89k+ human-written spatial queries that name no object. Spatial queries cover relative position, absolute distance, and narrative and parametric perspective; knowledge queries cover common sense and human intention.

Example SURPRISE3D queries over a 3D room, grouped into spatial reasoning (relative position, absolute distance, narrative perspective, parameter perspective) and knowledge reasoning (common sense, human intention). — TeaserSURPRISE3D asks for objects without naming them. Spatial queries cover relative position, absolute distance, and narrative or parameter perspectives; knowledge queries cover common sense and human intention.

SURPRISE3D poster, NeurIPS 2025 — PosterNeurIPS 2025 poster. Opens the PDF.

MLLM-For3D: Adapting Multimodal Large Language Model for 3D Reasoning Segmentation

Jiaxin Huang, Runnan Chen, Ziwen Li, Zhengqing Gao, Xiao He, Yandong Guo, Mingming Gong, Tongliang Liu

NeurIPS 2025 · first author

arXiv Proceedings Project page Code Poster (PDF)

The first label-free framework for 3D reasoning segmentation. A frozen 2D multimodal LLM segments several posed views from one implicit query; token-for-query alignment and a spatial consistency strategy turn the multi-view masks into reliable pseudo-labels that train a 3D encoder without any 3D annotation.

MLLM-For3D framework: a frozen multimodal LLM segments randomly selected posed views from an implicit query; token attention keeps reliable views as pseudo-labels that pre-train a 3D encoder with spatial and mask losses. — FrameworkA frozen 2D multimodal LLM segments several posed views from one implicit query. Token attention keeps the reliable views, and their masks become pseudo-labels that train a 3D encoder without any 3D labels.

MLLM-For3D poster, NeurIPS 2025 — PosterNeurIPS 2025 poster. Opens the PDF.

Demo

2D to 3DMasks from a frozen 2D reasoning-segmentation model (LISA), predicted on every fifth frame and lifted into the reconstructed point cloud. Colours mark semantic and instance queries for the chairs, tables and screen.

Scenes from videoFour ScanNet rooms reconstructed with VGGT, which predicts geometry from video frames in a single feed-forward pass. Point clouds like these are the 3D scenes that reasoning segmentation runs on.

S14

EgoAudit: Auditing Egocentric Spatial Reasoning in 3D Vision-Language Models

Jiaxin Huang, Chengcheng Lin, Xiyang Huang, Yankai Chen, Min Peng, Qianqian Xie

Under review · first author

An audit of egocentric spatial reasoning in 3D vision-language models: whether a model infers the speaker's viewpoint or only follows a viewpoint it is given.

WIP

3D & 4D Spatial Reasoning: A Comprehensive Review

In preparation · first author, project leader

子非鱼，安知鱼之乐Zhuangzi · Aristotle

East · 东

“You are not a fish. How do you know what fish enjoy?” “You are not me. How do you know I don't know what fish enjoy?”

Zhuangzi 庄子, “Autumn Floods” 《秋水》

子，非，鱼，安，知，鱼，之，乐，子，非，我，安，知，我，不，知，鱼，之，乐。

West · 西

The soul never thinks without a mental image.

Aristotle, De Anima III.7, 431a