NeurIPS 2026
A Stagewise Framework for Robust Contact-Rich Manipulation
1 The Chinese University of Hong Kong, Shenzhen
2 DexForce Co., Ltd. 3 Shenzhen Loop Area Institute.
† Corresponding author
Contact-rich manipulation requires precise control under complex contact dynamics while remaining robust to diverse visual conditions. However, real-world reinforcement learning often overfits to the visual appearance of training scenes. We propose Semantics-to-Contact (S2C), a stagewise framework for visually robust contact-rich manipulation. The insight is to decompose the problem into two stages: semantic focusing and contact refinement. In the first stage, a vision-language model localizes task-relevant regions across visually diverse scenes, providing coarse spatial guidance and reducing the burden of exploration. In the second stage, a real-world human-in-the-loop residual reinforcement learning policy learns fine-grained contact behaviors within the localized region using force feedback. To further improve visual robustness, we introduce an object-centric visual augmentation strategy that randomizes background appearance while preserving the robot and task-relevant objects. Experiments on six contact-rich real-world manipulation tasks exhibit that S2C consistently improves success rates over strong baselines and maintains reliable performance under significant visual and positional variations. These results demonstrate that stagewise semantic focusing and contact refinement provide a practical path toward visually robust real-world contact-rich manipulation.
The highlights
Success in the training setting
113 / 120 trials · six tasksSuccess under visual + positional shifts
378 / 480 trials · four perturbation settingsSemantic focusing → contact refinement
Learned through real physical interactionHow it works
A vision-language model brings the robot to a task-relevant region. A force-aware residual policy learns the fine adjustments that turn an approach into a successful interaction.
Semantic focusing
MolmoPoint localizes the target in a third-person image. Depth and hand–eye calibration turn the predicted point into a robot-frame goal; motion planning brings the arm close to the contact region.
Contact refinement
A residual reinforcement learning policy improves an ACT base policy using wrist-camera observations, robot state and force feedback. Human corrections support learning directly on the physical robot.
Object-centric visual augmentation
YOLOE segments the robot and task-relevant objects. Background colors and textures are randomized during training, while foreground color and brightness are also perturbed. The physical interaction remains real.
Real-world experiments
Explore all 18 individual recordings.
Select a clip to watch the full trial.
Changed tablecloths, strong illumination and colored-light perturbations.
No clips match this selection. Try another task or distribution.
ID denotes the ordinary black-cloth setup. OOD includes changed surfaces and deliberately altered illumination, even when the tabletop remains black. Removal clips demonstrate additional tasks and are separate from the six-task insertion benchmark.
Quantitative evaluation
Across six insertion tasks, S2C retains substantially higher success under combined visual and positional shifts. Each method is evaluated for 20 trials per task in each setting.
| Method | Charger | USB Easy | USB Hard | Tube Easy | Tube Medium | Tube Hard | Average |
|---|
Results are taken from the supplied manuscript. A trial succeeds when the object is fully inserted. Averages are calculated from the per-task trial counts. HIL baselines use their original initialization procedures; they do not use S2C’s VLM-guided pre-contact initialization. Reported training times in the manuscript measure learner-side optimization, not total wall-clock robot interaction.
Beyond insertion
Additional removal tasks use a diffusion-policy base: test-tube removal succeeds in 9/10 trials and hard-drive removal in 10/10 trials.
Watch removal trials ↗Understanding the boundary
S2C tolerates many background, lighting and small positional changes. Changes inside the task-critical contact region remain difficult: replacing the charger with a black charger yields 0/10 on both charger and USB-easy tasks in the reported stress test.
@inproceedings{qiao2026semanticstocontact,
title = {Semantics-to-Contact: A Stagewise Framework
for Robust Contact-Rich Manipulation},
author = {Qiao, Guanren and Ouyang, Ruixiang and
Xu, Sheng and Deng, Yueci and Tai, Yunxin and
Jia, Kui and Liu, Guiliang},
booktitle = {Advances in Neural Information Processing Systems},
year = {2026}
}