Skip to main content

GBC_MUJICA

Explore the diffusion guided motion planner for robots with GBC standard! (Will be submitted to ICRA 2026, code release by September 2025)

GBC-MUJICA: Motion Diffusion and Trajectory Planning Model for Cross-Morphology Humanoid Architectures Based On GBC Framework

Yifei Yao1

1 All authors are with Machine Vision and Autonomous System Laboratory, Shanghai Jiao Tong University, Shanghai, China.

⚠️ Important Notice​

🔥 Note: GBC-MUJICA is an extension work based on the GBC retargeting framework. If you want to reproduce or conduct research based on this work, please first configure the basic runtime environment according to the GBC repository installation instructions.

✨ GBC-MUJICA is an extension work based on the GBC framework, designed to generate complex motion trajectories for cross-morphology humanoid robots. This framework utilizes diffusion models to generate a series of executable robot actions to accomplish specific tasks, and allows adjustment of action sequences through continuous text guidance.

Abstract​

Universal humanoid robot control remains a major challenge, as existing methods often lack adaptability to different robot morphologies and tasks. Although our previous GBC framework achieved imitation of human actions, it also highlighted the industry's urgent need for high-quality, physically feasible motion data. To address this issue, we introduce the GBC-MUJICA framework. This is a text-driven, robot morphology-adaptive diffusion model framework that can directly generate control commands for specific robots. Our model integrates an enhanced motion retargeting network to ensure the physical feasibility of actions and can synthesize motion under the guidance of text prompts and optional kinematic constraints (such as end-effector positions). The core contribution of this research is our proposed "zero-code" pipeline, which can automate the complete workflow from data conversion, model training to simulation for any new humanoid robot that only needs to be defined through simple configuration files. Validation in the Isaac Sim simulation platform shows that GBC-MUJICA significantly outperforms the "generate-then-retarget" baseline method, demonstrating its efficient, direct control capabilities and greatly reducing the entry barrier for universal humanoid robot research.

🤖 Main Components​

The structure of this model mainly consists of the following parts:

  • GBC_MUJICA/data_preparation/: 📊 Data preparation and model definition module, used to generate training datasets and training parameters according to corresponding robot configuration files.
  • GBC_MUJICA/diffusion_planner/: 🧠 Diffusion Model core, the framework inherits from Human motion diffusion model, and has been reconstructed for humanoid robot structures with data and automated adaptation.
  • GBC_MUJICA/utils/ (code under review): 🔧 Tool interface module for robot control and simulation.
  • GBC_MUJICA/controller/ (code under review): 🎮 Robot controller module, which performs control and simulation by calling the runtime and simulation interfaces of the GBC framework.

📦 Installation Guide​

This project is tested under Ubuntu 22.04.5 LTS and requires the following dependencies:

  • If you want to configure an environment that can reproduce all GBC functions:

    • 🐍 Python 3.10+ (mainly for compatibility with the Isaac Lab part in the GBC framework. If simulation and reinforcement learning are not needed, you can configure retargeting according to the GBC documentation, which can greatly reduce requirements for Python version, GPU memory, and CUDA version)
    • 🔥 PyTorch 2.0.1+
    • 🐧 miniconda3
    • 🎮 NVIDIA GPU supporting CUDA 11.8+ (at least 8GB memory, 12GB+ recommended)

  • If you want to configure an environment that only reproduces GBC-MUJICA conversion, training, and diffusion generation:

    • 🐍 Python 3.8+
    • 🔥 PyTorch 1.13.1+
    • 🐧 miniconda3
    • 🎮 Any CUDA-supported GPU (no minimum memory requirement). Can also run on CPU devices if training is not needed.

1. Prepare GBC Basic Configuration​

Please refer to the GBC Installation Guide for basic GBC configuration (Isaac Lab optional), and configure the corresponding version according to the GBC functions you want. Even if you don't need to use GBC's training environment, you still need to configure all libraries required for its retargeting part (Isaac Lab can be excluded).

Due to library dependencies, please ensure that GBC-MUJICA runs in the same Conda environment as GBC.

2. 📥 Clone and Install This Repository​

git clone https://github.com/sjtu-mvasl-robotics/GBC_MUJICA.git
conda activate <your_gbc_env>
cd GBC_MUJICA
pip install -e .

3. 📚 Download All Required Datasets​

  1. 📊 Prepare AMASS Dataset

Please ensure you have registered an account and password at https://amass.is.tue.mpg.de/, then run the following program:

python GBC_MUJICA/data_preparation/download_amass.py --username <your_username> --password <your_password> --output_dir <your_dataset_dir> --smplh_only --cleanup --force
  1. 📝 Prepare GloVe Pre-trained Word Vectors
python glove/prepare_data.py

4. 💾 (Optional) Download GBC-MUJICA Pre-trained Models for Your Robot Type​

We currently only provide pre-trained models for unitree_h1_2, unitree_g1, fourier_gr1, and turin_v3 configurations. You can visit our Hugging Face address to select and download. If you need to experiment with this framework on other configurations, please refer to the instructions in the Running Guide.

🚀 Running Guide​

The operation of this framework includes data preparation, training, validation, and deployment parts (not yet released).

1. 📊 Data Preparation​

(⚠️ Important): You must carefully complete the robot data configuration work in this part, otherwise you will not be able to generate valid learnable data!

  1. Follow the GBC guide to train the SMPLH regression model and motion retargeting network suitable for your robot configuration, and record their file paths. For training instructions, please refer to GBC's retargeting training guide. We now allow you to call our GBC compatible scripts directly to generate the fitting result and ik model, with properly configured yaml files. Detailed instructions can be found in the GBC Retargeting Guidebook. We recommend you to run step 2 first. Then run the following code:
export HUMANOID_NAME=<your_robot_name>
python GBC_MUJICA/data_preparation/gbc_retargeting/smplh_fit.py # wait patiently for 1-5 minutes, depending on your configuration

Then, run

export HUMANOID_NAME=<your_robot_name>
python GBC_MUJICA/data_preparation/gbc_retargeting/poseformer_trainer.py

This step may take 1-2 days, depending on your hardware. After this is done, you can find the generated smplh_fits and poseformer directories in your configuration file (they will be automatically replaced).

  1. In the GBC_MUJICA/data_preparation/configs/ directory,

    • Copy the unitree_h1_2.yaml file and rename it to <your_robot_name>.yaml.
    • Modify urdf_path, ik_model_path (retargeting network path), smpl_fits_dir (regression model path) in the file to the paths you recorded in the first step.
    • According to the comments in the file, modify other parameters to adapt to your robot configuration.
    • In the data sub-option, configure your downloaded dataset address (AMASS dataset address), your training dataset export address, whether to use temporary working directory (for storing intermediate variables), working directory (if using temporary working directory, you must set a valid working directory), and whether to enable full angular velocity loading (default false, only loads angular velocity z-axis component; if fully enabled, may bring risks of physically infeasible action generation).
    • We recommend you (⚠️ Important): This step must be done! You have to add a copy of your implemented robot_flip_left_right to the GBC_MUJICA/data_preparation/humanoid_cfg/ directory, then you can add the import path in your configuration file.

  2. Add your robot name to environment variables and run the dataset conversion file (takes about 1-2 hours).

export HUMANOID_NAME=<your_robot_name>
python GBC_MUJICA/data_preparation/prepare_data_amass.py
  1. Prepare modified Mean and Std files for training the feature extractor.
python ds_config/prepare_mean_std.py

2. 🎯 Training (Optional)​

  1. 🧠 Train text_motion_matcher Feature Extractor

Before formal model training begins, you need to first train the feature extractor for model evaluation. We allow you to monitor your training logs through WandB. If you haven't configured a WandB account, please add the flag --disable_wandb to the startup command.

python -m GBC_MUJICA.diffusion_planner.train.train_text_mot_match \
    --name <your_experiment_name> \
    --batch_size 32 \
    --lr 1e-4 \
    --dataset_name hn_t2m \

Of course, you can also directly load the configuration from the file in ds_config/humanoidml_opt.txt:

python -m GBC_MUJICA.diffusion_planner.train.train_text_mot_match \
    --config ds_config/humanoidml_opt.txt \

This training usually completes within 1-2 hours.

  1. 🌟 Train Diffusion Model

After the feature extractor training is complete, you can start training the diffusion model. This stage will occupy different amounts of memory depending on your batch size, but a 6GB graphics card is sufficient to support the entire training. The current model main architecture still follows the basic design of Human motion diffusion model, namely the corresponding transformer encoder structure and transformer decoder structure. The text feature encoders we support include clip, bert (corresponding to distilled bert) and vlm (corresponding to llava token output) (VLM-related parts are not yet published and will be synchronized to this repository in subsequent paper updates). You can enable training in different modes through the following code. For more runtime parameters, please refer to GBC_MUJICA/diffusion_planner/utils/parser_util.py.

  • 🎯 Training for Encoder Architecture
    python -m GBC_DP.diffusion_planner.train.train_hnmdm \
        --save_dir <your_save_dir> \
        --dataset hn_t2m \
        --cuda True \
        --device 0 \
        --arch trans_enc \
        --batch_size 32 \
        --text_encoder_type clip \
        --labmda_target_loc 0.0 \
        --overwrite \
        --eval_during_training \
        --train_platform_type WandBPlatform \
  • 🎨 Training for Decoder Architecture

Since there are relatively many configuration parameters here, it is recommended that you directly run train_dec.sh and configure your parameters in it. Please note that for available parameters for target links, please check the definitions in the paper and code (they usually automatically detect and constrain the positions of your robot's end effectors):

    bash train_dec.sh

The entire training process usually takes 1-2 days, depending on your GPU performance and batch size.

3. ✨ Generation and Modification​

We also implement MDM-based action generation and modification functions. To verify the generation function, you can modify the given cases in .vscode/launch.json and replace the model address with your trained checkpoint or downloaded pre-trained checkpoint. Here are some examples:

  • 🎭 Basic Generation
{
    "name": "Generate Humanoid Motion",
    "type": "debugpy",
    "request": "launch",
    "module": "GBC_MUJICA.diffusion_planner.sample.hn_generate",
    "console": "integratedTerminal",
    "env": {
        "HUMANOID_NAME": "unitree_h1_2"
    },
    "args": [
        "--model_path", "/home/yifei/codespace/GBC_MUJICA/save/hnmdm_trans_enc_512/model000250000.pt",
        "--text_prompt", "a person is skipping rope.",
        "--lambda_target_loc", "0.0",
        "--use_pyrender" // If you want to generate actions through pyrender during generation, please add this flag, otherwise it will be plotted through matplotlib.
    ],
    "justMyCode": false,
    "purpose": ["debug-in-terminal"]
},
  • 🎯 Target-Constrained Generation (dec):
{
    "name": "Generate Humanoid Motion DiP (with target)",
    "type": "debugpy",
    "request": "launch",
    "module": "GBC_MUJICA.diffusion_planner.sample.hn_generate",
    "console": "integratedTerminal",
    "env": {
        "HUMANOID_NAME": "unitree_h1_2"
    },
    "args": [
        "--model_path", "/home/yifei/codespace/GBC_MUJICA/save/hnmdm_trans_dec_512/model000305000.pt",
        "--text_prompt", "a man kicks with his left leg.",
        // "--lambda_target_loc", "0.0",
        "--diffusion_steps", "1000",
        "--context_len", "20",
        "--pred_len", "40",
        "--num_repetitions", "3",
        "--sampling_mode", "goal",
        "--target_link_names", "DIMP_FINAL",
        "--target_link_source", "data",
        // "--input_text", "dynamic_text_prompts.txt",
        // "--dynamic_text_path", "dynamic_text_prompts.txt",
        "--autoregressive",
        "--use_pyrender"
    ],
    "justMyCode": false,
    "purpose": ["debug-in-terminal"]
},
  • 🔄 Continuous Online Generation Based on Prompts (Autoregressive)
{
    "name": "Generate Humanoid Motion DiP On the fly (with target)",
    "type": "debugpy",
    "request": "launch",
    "module": "GBC_MUJICA.diffusion_planner.sample.hn_generate",
    "console": "integratedTerminal",
    "env": {
        "HUMANOID_NAME": "unitree_h1_2"
    },
    "args": [
        "--model_path", "/home/yifei/codespace/GBC_MUJICA/save/hnmdm_trans_dec_512/model000345000.pt",
        // "--text_prompt", "A person is slowly raising his left arm.",
        // "--lambda_target_loc", "0.0",
        "--diffusion_steps", "10",
        "--context_len", "40",
        "--pred_len", "20",
        "--motion_length", "8",
        "--num_repetitions", "1",
        "--sampling_mode", "goal",
        "--target_link_names", "DIMP_FINAL",
        "--target_link_source", "data",
        // "--input_text", "dynamic_text_prompts.txt",
        "--dynamic_text_path", "dynamic_text_prompts.txt",
        "--autoregressive",
        "--use_pyrender"
    ],
    "justMyCode": false,
    "purpose": ["debug-in-terminal"]
},
  • ✏️ Unconditional Full-Body Motion Modification
python -m GBC_MUJICA.diffusion_planner.sample.hn_edit \
    --model_path <your_model_path> \
    --edit_mode in_between

Add --text_condition "text" to enable text-guided modification. Change edit mode to upper_body to achieve pure upper body modification.

4. 🎮 Simulation Verification​

Note: Please ensure you have correctly configured the Isaac Sim environment and generated the USD file corresponding to your robot. To realize this, you have to register a configuration file under GBC/gyms/isaaclab_45/lab_assets with correct actuator configuration and usd path. Please refer to the original GBC repository documents and Isaac Lab documents for more details. We will not support the deprecated isaacgym environment, but you can try to implement it by yourself. Simulating with Dynamics requires you to train a GBC controller first, you have to prepare full GBC library, adjust training parameters & reward functions to build your own controller.

  1. Run the following file to verify your generated actions:
python -m GBC_MUJICA.controller.isaacsim_viewer \
    --pkl_file <your_generated_motion.pkl> \
    --usd_file <your_robot_usd_file> \

If you want to run this verification script in a server environment, please add --headless and --record_videos parameters, or configure remote VNC display service according to Isaac Lab tutorials.

  1. 🎮 Run Isaac Sim Dynamics Simulation

Please ensure you have correctly completed GBC controller training at this point! If you lack the corresponding controller and training configuration under the GBC environment, this will not run.

python -m GBC_MUJICA.controller.isaacsim_controller \
    --lab_config <your_gbc_lab_config> \
    --model_path <your_rl_model_path> \
    --pkl_file <your_generated_motion.pkl> \

Similarly, you can also run this simulation in MuJoCo.

5. 🤖 Using VLM to Generate Task-Based Text Prompts (Not Yet Released)​

Please follow our new project GBC-BTR (Behavior-clone Task Reasoning) to get VLM-related functions. (Repository will be available soon)

🙏 Acknowledgments​

We thank the following frameworks that provide core support for this project. Without their support, we could not achieve the goals of this project:

📄 LICENSE​

This project follows the MIT License. You are free to use, modify, and distribute this code, but please retain the original author's copyright statement and license information.

The dependencies used in this project, including the original LICENSE of MDM, the LICENSE of GBC, etc., all follow their respective open source license agreements. Please comply with the relevant license terms when using.

📚 Citation​

If you use GBC-MUJICA in your research or project, please cite the following literature:

@misc{GBC_MUJICA,
  title = {GBC-MUJICA: Motion Diffusion and Trajectory Planning Model for Cross-Mophology Humanoid Architectures Based On GBC Framework},
  author = {Yifei Yao},
  year = {2025},
    url = {
    https://github.com/sjtu-mvasl-robotics/GBC-MUJICA
    },
    note = {Accessed: 2025-01-01}
}

(Will be replaced with Arxiv version citation after publication)

📧 Contact Us​

If you have any questions or suggestions, please contact us through the following ways: