MoMa-SG: Articulated 3D Scene Graphs

Introduction

Semantic scene understanding has enabled robots to reason and perform affordance-driven object interaction. Despite this, real-world manipulation remains fundamentally limited by the inability to predict how objects move and are articulated. Long-horizon mobile manipulation requires closing the gap between semantics, geometry, and kinematics. We present MoMa-SG, a novel framework for building semantic-kinematic 3D scene graphs of articulated scenes containing a myriad of interactable objects. Given RGB-D sequences containing multiple object articulations, we temporally segment object interactions and infer object motion using occlusion-robust point tracking. We lift point trajectories to 3D and estimate articulation models using a novel unified twist estimation formulation that robustly estimates revolute and prismatic joint parameters in a single optimization pass. Next, we associate objects with articulations and detect contained objects by reasoning over parent-child relations at identified opening states. We also introduce the Arti4D-Semantic dataset, which provides parent-child relation annotations between container and contained objects together with object axis annotations across 62 in-the-wild RGB-D sequences containing 600 object interactions and three distinct observation paradigms. We extensively evaluate the performance of MoMa-SG on two datasets and ablate key design choices of our approach. In addition, real-world experiments on both a quadruped and a mobile manipulator demonstrate robust manipulation of articulated objects in everyday home environments.

Method

We first discover interaction segments via probabilistic fusion of an interaction prior (YOLOv11) and depth disparity. For each segment, CoTracker3 tracks keypoints; a novel regularized twist formulation with geometric cosine prior estimates articulation models. SemanticSAM with CLIP features builds an open-vocabulary 3D map; objects are matched to articulations via a binary integer program; and child objects within articulated containers are discovered at maximum-opening states.

🔍 Interaction Discovery

Probabilistic fusion of hand/agent visibility (YOLOv9) and depth warp disparity. Robustly finds interaction segments even under occlusions or low-dynamics scenarios such as opening large doors.

🔧 Regularized Twist Estimation

Novel geometric cosine prior on point trajectory pairs enforces orthogonality for revolute joints and suppresses rotation for prismatic joints — eliminating need for separate type classification rules.

🗺️ Scene Graph Construction

Boundary-aware incremental 3D mapping with SemanticSAM + CLIP, IoU-constrained articulation–object assignment via BIP, and hierarchical parent-child containment discovery.

Contributions

A unified framework for constructing articulated 3D scene graphs from ego-centric, exo-centric, or robot-centric observations, enabling one-shot open-world mobile manipulation.
The novel Arti4D-Semantic benchmark — the first dataset bridging semantic scene understanding and articulated object estimation in in-the-wild settings (62 sequences, 600 interactions, 3 observation paradigms).
A novel twist optimization objective with a scaled dot-product cosine prior for robust estimation of revolute and prismatic articulations directly from real-world point trajectories with drift and occlusions.
Real-world demonstrations on Toyota HSR and Boston Dynamics Spot, achieving over 80% task success across articulation types and environments, plus language-guided long-horizon manipulation.
Full dataset, evaluation routines, and code are publicly available at momasg.cs.uni-freiburg.de.

Arti4D-Semantic Dataset

RGB-D Sequences

600

Object Interactions

Distinct Scenes

Observation Paradigms

Parent Object Categories

Child Object Categories

Arti4D-Semantic extends the original Arti4D dataset with rich semantic labels: instance segmentation of articulated parents and contained children, independently verified semantic categories, articulation axes, ground-truth interaction segments, and parent-child containment type (STATIC vs. ARTICULATED). It is the first benchmark to jointly address articulated object estimation and semantic 3D mapping in unconstrained in-the-wild settings.

Observation Paradigms

👁️ Ego-Centric (EGO)

Human demonstrator with a head-mounted RGB-D camera. Four labeled scenes (RH078, RR080, DR080, RH201) with ground-truth camera poses, plus an unlabeled scene (MHZH). Covers drawers, cabinets, doors, fridges, and sliding furniture.

📷 Exo-Centric (EXO)

External fixed observer viewpoint (MHZH-EXO). Introduces additional challenges from bystander motion and perspective changes. No ground-truth camera poses — pseudo-GT constructed via DROID-SLAM.

🤖 Robot-Centric (ROBOT)

Teleoperated Toyota HSR with on-board camera (HSR split, 8 sequences). Characterized by higher depth noise and robot motion artifacts. Tests generalization to real manipulation scenarios.

Download Links

Please download the following splits of the dataset including the metadata and unzip each at the same location, e.g., path/to/artti4d-sem/raw/, to maintain the correct folder structure.

Metadata (Arti4D-Semantic) RH201 (ego-centric) RH078 (ego-centric) RR080 (ego-centric) DIN080 (ego-centric) MHZH (ego-centric) MHZH-EXO (exo-centric) HSR (robot-centric)

Experimental Results

Interaction Segmentation on Arti4D-Semantic

Method	1D-IoU ↑	Precision ↑	Recall ↑	Segment-IoU ↑	I_on [s] ↓	I_off [s] ↓
Pandora	0.359	0.690	0.400	0.689	0.744	0.725
HMM	0.471	0.871	0.532	0.683	0.770	0.762
ArtiPoint	0.575	0.678	0.714	0.698	1.050	0.558
MoMa-SG (Ours)	0.649	0.786	0.800	0.718	0.816	0.742

MoMa-SG achieves the highest 1D-IoU, recall, and segment IoU. The depth warp prior provides a significant boost over hand-tracking alone.

Articulation Estimation on Arti4D

Method	Pris. θ_err ↓	Rev. θ_err ↓	Rev. d_L2 ↓	Type Acc. ↑	Recall Pris./Rev. ↑	Mode Acc. ↑
Ours w/ CoTracker3	13.190°	22.982°	0.091 m	88.4%	91.7 / 86.6%	98.3%
Ours w/o ξ-regul. + w/o RANSAC	34.089°	23.175°	0.103 m	90.8%	87.7 / 95.8%
Ours w/o ξ-regul. + RANSAC	21.427°	17.119°	0.103 m	90.8%	87.7 / 95.8%
Ours w/o RANSAC	11.205°	20.382°	0.113 m	90.8%	87.7 / 95.8%
ArtGS	52.070°	62.638°	0.301 m	61.9%	100.0 / 0.0%	—
Pandora	46.814°	50.620°	0.195 m	55.9%	77.2 / 44.4%	—
ArtiPoint w/ Sturm et al.	17.754°	28.467°	0.313 m	87.2%	85.4 / 90.5%	—
ArtiPoint w/o prior	22.953°	27.057°	0.269 m	77.6%	71.3 / 89.9%	—
ArtiPoint w/ prior	23.272°	26.358°	0.248 m	77.6%	70.9 / 90.6%	—
MoMa-SG (Ours)	10.374°	14.800°	0.087 m	90.9%	87.7 / 95.8%	98.3%

MoMa-SG outperforms all baselines on axis and position errors. Ablation rows (yellow) show the impact of twist (ξ) regularization and RANSAC. Bold = best, underlined = second-best; "—" marks settings not evaluated for a given method.

3D Part Mapping & Contained Object Discovery

Method	3D Part Mapping					Contained Object Discovery
	Object	IoU	Recall			IoU	Recall [%]	Relation Acc. [%]
	Set	IoU	@0.25	@0.50	@0.75	IoU	@IoU=0.25	STATIC / ARTIC.
OpenMask3D	O	0.082	0.087	0.019	0.000	—
ConceptGraphs		0.438	0.332	0.127	0.018	—
Ours w/o boundary-aware merging		0.481	0.807	0.534	0.086	—
MoMa-SG (Ours)		0.533	0.824	0.646	0.163	—
Pandora	O^A	0.065	0.012	0.000	0.000	00.02	00.35	19.77
Ours w/ CoTracker3		0.221	0.370	0.218	0.027	09.14	16.67	59.23
Ours w/ greedy matching		0.423	0.703	0.453	0.110	09.50	37.43	67.19
MoMa-SG (Ours)		0.434	0.720	0.471	0.111	11.14	42.60	90.56

MoMa-SG significantly outperforms baselines on 3D part mapping and enables accurate contained object discovery on matched objects (O^A). Ablation rows (yellow) show the impact of boundary-aware merging, CoTracker3, and greedy matching. Bold = best, underlined = second-best; "—" marks settings not evaluated for the free object set (O).

Real-World Mobile Manipulation

85.5%

Toyota HSR overall success
(opening & closing)

80.8%

Boston Dynamics Spot
overall opening success

1.7 cm

Mean prismatic state error
across 54 tested configurations

6.5°

Mean revolute state error
across 54 tested configurations

MoMa-SG produces an embodiment-agnostic semantic-kinematic map. The same scene graph drives both the Toyota HSR and Boston Dynamics Spot without modification. Experiments span two kitchen environments, including autonomous opening, closing, online articulation state estimation with retrial behavior, and full language-grounded long-horizon manipulation (e.g., "get the milk from the fridge") using a GPT-5-mini planner.

🔄 State Estimation & Retrial

Online articulation state estimation across diverse opening configurations. Enables automatic retrial after gripping failures by re-estimating current state via point-cloud overlap under the estimated twist.

🗣️ Language-Grounded Tasks

Natural language commands mapped to sequences of open(), inspect(), retrieve(), and close() actions using the open-vocabulary scene graph as context for a receding-horizon GPT-5-mini planner.

🤖 Embodiment-Agnostic

Identical scene representation used across Toyota HSR (whole-body) and Boston Dynamics Spot (quadruped arm). Online grasp generation via SAM3 handle detection and predominant-axis estimation.

Publication

Martin Büchner, Adrian Röfer, Tim Engelbracht, Tim Welschehold, Zuria Bauer, Hermann Blum, Marc Pollefeys, and Abhinav Valada.
Articulated 3D Scene Graphs for Open-World Mobile Manipulation
Conference on Robot Learning (CoRL), 2026.

@article{buechner2026momasg,
  title={Articulated 3D Scene Graphs for Open-World Mobile Manipulation},
  author={Buechner, Martin and Roefer, Adrian and Engelbracht, Tim
          and Welschehold, Tim and Bauer, Zuria and Blum, Hermann
          and Pollefeys, Marc and Valada, Abhinav},
  journal={Conference on Robot Learning (CoRL)},
  year={2026}
}