Human-Centric Transferable Tactile Pre-Training for Dexterous Robotic Manipulation

TTP introduces human-centric tactile pre-training for fine-grained robotic manipulation. It presents H-Tac, a 160-hour visuo-tactile-action dataset spanning 300+ tasks and 135k+ episodes, and trains a VLA policy to jointly predict future action chunks and future tactile signals. By keeping tactile and action spaces unified across human pre-training and robot post-training, TTP transfers contact-aware priors to downstream robotic tasks.

Overview

Fine-grained manipulation often depends on contact signals. Standard VLA policies can imitate visually plausible motions, but without tactile grounding they are not explicitly supervised to model how contact will evolve.

TTP addresses this through scalable human-centric tactile pre-training. We introduce H-Tac and keep tactile and action spaces unified across human pre-training and robot post-training, while training the model to predict both future action chunks and future tactile signals.

The Dataset: H-Tac

H-Tac is the dataset behind TTP. It combines three sources with different strengths and projects them into a unified tactile-action representation.

HOI-Tac derives contact-based tactile supervision from public hand-object, hand-face, and hand-scene interaction datasets. For each frame, contact labels are generated on the MANO hand mesh and projected to the 351-taxel UniTacHand tactile space.

DeskTask-Tac collects desktop human manipulation with three RealSense cameras, tactile gloves, and hand reconstruction pipelines. It aligns first-person video, tactile readings, MANO hand states, task labels, and action targets.

InternData-Tac augments robot interaction data with contact forces and patch-level details, then projects active contacts onto the shared MANO surface across Genie1, Lift2, and Split ALOHA configurations.

The resulting dataset contains 160 hours of visuo-tactile-action data, 300+ tasks, and 135k+ episodes. Dataset statistics show contact activations concentrated on fingertips, while task instructions are dominated by manipulation verbs, matching the contact-heavy setting targeted by tactile pre-training.

The Method: Transferable Tactile Pre-Training

TTP extends a foundation VLA model with tactile observation tokens and tactile prediction. Instead of being trained to touch only as a downstream sensor, the model learns actions and future tactile signals together from human-centric data.

The key is consistency: human pre-training and robot post-training share unified action and tactile spaces, so the contact priors learned from human demonstrations can be preserved when transferred to robot hands and tactile grippers.

Architecturally, TTP adds a tactile prediction expert alongside the action expert. Both experts are trained with flow matching, while Tactile-Action Manifold-Preserving Gating helps stabilize the shared action-tactile representation.

After tactile-based pre-training, TTP can generate hand motion and tactile heatmaps in both original validation scenes and inpainted out-of-distribution scenes.

Simulation Experiments

Simulation benchmarks do not natively provide tactile observations, so TTP uses a tactile proxy during post-training. Even with the additional tactile prediction objective, it remains competitive on LIBERO, LIBERO-plus, and RoboCasa.

These results suggest that adding tactile prediction preserves strong general manipulation performance, even on benchmarks without native tactile observations.

Real-World Experiments

The real-robot suite evaluates TTP across multiple platforms and embodiments: Franka arms, Realman arms, Inspire tactile hands, DM-Tac tactile grippers, and DexBotic hands.

The task suite covers three regimes:

• Fine-grained: peeling and wiping tasks where sustained contact matters.
• Contact-rich and fragile: chip picking and paper folding, where pressure must stay in a narrow useful range.
• Visually ambiguous or occluded: softness sorting and plug insertion, where tactile feedback helps when visual cues are incomplete.

Across In-distribution(ID) and Out-of-distribution(OOD) tests, TTP outperforms tactile-free baselines and the same architecture without tactile pre-training. Because the tasks use different metrics, including success rate, peeling length, and folding progress, results are reported as normalized average task progress rates.

The gains are largest where the task is high on contact. In peeling, TTP maintains contact long enough to remove longer strips. In chip picking, it applies moderate force instead of crushing or slipping.

OOD tests include new objects, new object locations, altered visual appearances, and material changes. These settings probe whether the model learned a transferable tactile-action prior rather than only memorizing visual layouts.

Citation

If you find this project useful, please consider citing:

@article{zhang2026human,
  title={Human-Centric Transferable Tactile Pre-Training for Dexterous Robotic Manipulation},
  author={Zhang, Chi and Cai, Penglin and Xi, Ziheng and Yuan, Haoqi and Luo, Hao and Zhang, Wanpeng and Zheng, Sipeng and Xu, Chaoyi and Lu, Zongqing},
  journal={arXiv preprint arXiv:2607.01067},
  year={2026}
}