Being-0: A Humanoid Robotic Agent with Vision-Language Models and Modular Skills

Achieving human-level performance in real-world embodied tasks requires combining high-level cognition with low-level control. We propose Being-0, a hierarchical agent system that enables a full-sized humanoid robot to solve complex, long-horizon tasks with active vision and dexterous hands.

Key Concept

Directly connecting Foundation Models (FMs) to robotic skills often suffers from high latency, weak embodied reasoning, and compounding errors. Being-0 addresses this by introducing a hierarchical structure with a novel Connector module.

The Connector, powered by a lightweight Vision-Language Model (VLM) trained on egocentric planning data, acts as a bridge. It translates the FM's high-level language plans into actionable skill commands and dynamically coordinates locomotion and manipulation, ensuring robust execution on onboard devices.

Real-World Demos

Being-0 demonstrates robust performance in long-horizon household tasks.

It is equipped with a skill library of diverse dexterous manipulation tasks.

Pipeline

The Being-0 framework consists of three layers:

1. Foundation Model (FM): Handles instruction understanding, high-level reasoning, and task planning (deployed on the cloud).
2. Connector: A lightweight VLM that refine plans and trigger executable skills (deployed onboard).
3. Modular Skill Library: Provides atomic locomotion and manipulation skills (deployed onboard).

The figure below illustrates the workflow for "making a cup of coffee".

Experiments

The Connector Module

We compared Being-0 with a baseline lacking the Connector ("w/o Connector"). Being-0 w/o Connector frequently queries the FM, which often fails to provide correct plans due to its limited embodied scene understanding. In contrast, Being-0 with the Connector completes the task, requiring only a few queries to the FM.

Coordination of Navigation & Manipulation

Precise positioning is crucial for manipulation. Comparing Being-0 with a version lacking pose adjustment ("w/o Adjustment"), results show that dynamic adjustment is essential for transitioning from navigation to manipulation successfully.

Active Vision

We analyzed the necessity of the active camera system. Fixed camera configurations limit the agent's field of view, causing failures in both navigation and manipulation tasks, whereas the active camera ensures robust perception.

Citation

@article{yuan2025being,
  title={Being-0: A Humanoid Robotic Agent with Vision-Language Models and Modular Skills},
  author={Yuan, Haoqi and Bai, Yu and Fu, Yuhui and Zhou, Bohan and Feng, Yicheng and Xu, Xinrun and Zhan, Yi and Karlsson, B{"o}rje F and Lu, Zongqing},
  journal={arXiv preprint arXiv:2503.12533},
  year={2025}
}