Motivation
Human dexterity in demonstrations does not automatically translate into deployable robot dexterity.
How can robots learn deployable dexterity from human demonstrations?
Our answer is a physical interface that aligns collection with deployment, while making dexterous demonstrations intuitive and scalable through a wearable design.
In other words, RealDexUMI is designed to collect not just dexterous demonstrations, but deployable dexterity.
System overview
RealDexUMI integrates a lightweight dexterous hand, fingertip tactile sensing, an in-hand camera, a 6-DoF tracker, and a palm-side isomorphic glove. The glove maps press-and-release finger inputs to linear commands in the robot hand's action space, while the measured hand state records what contact allows, forming action-state correspondence between operator intent and physical interaction.
During collection, the operator wears the module and directly commands the robot hand. During deployment, the same end-effector module is mounted on the robot arm.
Experiments
We evaluate RealDexUMI on eight real-robot tasks spanning fine-grained, contact-rich, long-horizon, and bimanual manipulation. For each evaluated task, policies are trained on 200 RealDexUMI demonstrations and tested over 20 real-robot trials.
The resulting policies achieve an average full-task success rate of 88.75%.
Action representation
RealDexUMI predicts hand-frame relative end-effector motion instead of actions tied to a fixed robot base frame. This reduces dependence on the robot's initial pose.
We test the same cube pick-and-place checkpoint from four unseen initial robot poses. The policy succeeds from left, right, center, and raised-center starts, with five trials per pose.
Cross-embodiment deployment
RealDexUMI transfers across robot bodies by keeping the dexterous end-effector module and policy interface unchanged. The policy predicts hand-frame motion and hand commands, while each robot uses its own IK and low-level controller to realize the motion.
We deploy the same task checkpoints on Franka FR3, RealMan RM65, and one arm of PND Adam-U without retraining. Remaining differences in rollout quality mainly arise from robot-side kinematics, reachable workspace, IK behavior, and low-level control.
Action-state correspondence
RealDexUMI records executable glove commands together with measured hand states. In contact-rich manipulation, the measured state reflects what contact allows, while the glove command preserves what the operator intends. When a finger is blocked, the state may stop changing, but the command can still indicate continued closing.
This action-state correspondence helps the policy learn persistent contact, recovery, and corrective hand actions.
Limitation
RealDexUMI preserves a matched in-hand observation interface between collection and deployment, but this local view limits global scene awareness.
Failures can occur when target objects or task-progress cues fall outside the in-hand camera view. Future work should add broader visual context without reintroducing a collection-to-deployment visual gap.
Citation
@article{xu2026realdexumi,
title={RealDexUMI: A Wearable Universal Manipulation Interface for Dexterous Robot Learning},
author={Xu, Chaoyi and Jiang, Yixuan and Huan, Jiahui and Fu, Yuhui and Zhou, Haoyu and Yuan, Weitian and Yu, Jiayi and Zhang, Wanpeng and Yuan, Haoqi and Lu, Zongqing},
journal={arXiv preprint arXiv:2606.06033},
year={2026}
}