Bench2Dex Documentation

Tasks, Training & Evaluation Guide

Policy

Bench2Dex supports four imitation learning policy families. Each has its own framework, environment setup, training procedure, and evaluation commands.

1. ACT (Action Chunking with Transformers)

ACT is a behavior-cloning policy that uses a CVAE (Conditional Variational Autoencoder) transformer architecture to predict action chunks. It is built on a ResNet18 backbone with a DETR-style transformer encoder–decoder. The implementation is adapted from the original ACT repository.

Environment Setup

Create a uv-managed Python 3.9 virtual environment with PyTorch and all dependencies:

cd policy/ACT
bash env.sh                          # Build uv virtual environment
cd ../..

The script automatically installs uv if missing, creates .venv/, installs PyTorch 2.0.0 (CUDA 11.8), and all required packages (numpy, opencv-python, mujoco, h5py, etc.).

A conda alternative is also provided via conda_env.yaml:

conda env create -f policy/ACT/conda_env.yaml
conda activate act

Training

The training script supports positional arguments and automatic path resolution.

Basic Usage

# Default: uses outputs/ur5_rh56dfx/scenes/<TASK>/replay/
bash policy/ACT/train.sh 03_wine_glass_plate_balance

# Specify GPU and seed
bash policy/ACT/train.sh 03_wine_glass_plate_balance 0 1

# With custom dataset and output paths
bash policy/ACT/train.sh 03_wine_glass_plate_balance 0 1 /path/to/replay /path/to/output

# Train on generalized data (covariate / inv-covariate)
bash policy/ACT/train.sh 03_wine_glass_plate_balance 0 1 cov_only
bash policy/ACT/train.sh 03_wine_glass_plate_balance 0 1 inv_cov

Positional Arguments

Key Defaults

Evaluation & Inference

ACT provides three evaluation modes, each suited for different use cases.

Direct Evaluation (Recommended for Quick Validation)

Runs inference directly without a server/client — ideal for rapid testing:

bash policy/ACT/eval_direct.sh 03_wine_glass_plate_balance \
  ./outputs/logs/act/03_wine_glass_plate_balance/exp_20250101_120000/ \
  policy_last.ckpt 0 50 100000000 --max-steps 2400

Server/Client Evaluation

Connects to an existing model server (or starts one):

bash policy/ACT/eval.sh 03_wine_glass_plate_balance \
  ./outputs/logs/act/03_wine_glass_plate_balance/exp_20250101_120000/ \
  policy_last.ckpt 0 50 100000000

Double-Environment Evaluation (Full Channel Support)

Starts both a model server and an Isaac Sim client, supporting generalization profiles and Isaac restarts:

bash policy/ACT/eval_double_env.sh 03_wine_glass_plate_balance \
  ./outputs/logs/act/03_wine_glass_plate_balance/exp_20250101_120000/ \
  policy_best.ckpt 0 50 100000000 --headless \
  --generalization-profile none

Available Evaluation Flags

2. DP (Diffusion Policy)

Diffusion Policy uses a diffusion model (DDPM) over a U-Net architecture to generate action sequences conditioned on visual observations. It employs a ResNet18 multi-image encoder with 4 RGB cameras and a cosine-scheduled noise scheduler. The implementation is adapted from the Diffusion Policy and RoboTwin projects.

Environment Setup

Two environment options are available:

Option A: uv Virtual Environment

cd policy/DP
bash env.sh                          # Build uv virtual environment
cd ../..

Option B: Conda Environment

conda env create -f policy/DP/dp_env.yml
conda activate dp

Training

Training takes a task number and auto-discovers dataset paths. It uses Hydra for configuration management.

Basic Usage

# Single GPU (defaults to all 8 GPUs, use --gpu to restrict)
bash policy/DP/train.sh 03

# 4-GPU DDP training with custom learning rate
bash policy/DP/train.sh 03 --gpu 0,1,2,3 --lr 1e-4

# Foreground training (no tmux)
bash policy/DP/train.sh 03 --no-tmux --epochs 300

# Custom dataset directory
bash policy/DP/train.sh 03 --dataset /path/to/episodes

# Dry-run to inspect the command
bash policy/DP/train.sh 03 --dry-run

Training Flags

Key Architecture Defaults

Evaluation & Inference

Server/Client Evaluation

bash policy/DP/eval.sh 03_wine_glass_plate_balance \
  ../policy_ckpt/03/multi_ur5_rh56dfx_with_flange/dp_home/checkpoints/ \
  bs512_ep300.ckpt 0 50 100000000

Double-Environment Evaluation (Channel Support)

# Single channel (no generalization)
bash policy/DP/eval_double_env.sh 03

# Covariate generalization
bash policy/DP/eval_double_env.sh 03 cov

# Full generalization (invariant + covariate)
bash policy/DP/eval_double_env.sh 03 inv_cov

# All 4 channels sequentially
bash policy/DP/eval_double_env.sh 03 all

Available Flags for eval_double_env.sh

3. Pi0.5

Pi0.5 is a JAX-based vision-language-action (VLA) model from Physical Intelligence. Bench2Dex integrates Pi0.5 via the openpi library with full fine-tuning and WebSocket-based model serving. It uses the pi0_fast architecture with a Gemma + SigLIP backbone.

Environment Setup

Pi0.5 uses a uv-managed virtual environment (JAX-based, not PyTorch):

cd policy/pi05
GIT_LFS_SKIP_SMUDGE=1 uv sync         # Build uv virtual environment (JAX)
cd ../..

Training (Full Fine-Tuning)

Pi0.5 fine-tunes a pretrained base model with full-parameter training, loading the base weights from ModelScope/S3.

Basic Usage

# 8-GPU training with all defaults
bash policy/pi05/finetune.sh --task 03_wine_glass_plate_balance \
  --gpu-use 0,1,2,3,4,5,6,7 --exp-name my_exp \
  --dataset-dir /path/to/replay --output-dir /path/to/logs/pi05

# Specify robot key explicitly (overrides auto-detection)
bash policy/pi05/finetune.sh --task 03_wine_glass_plate_balance \
  --gpu-use 0,1,2,3,4,5,6,7 --exp-name my_exp \
  --dataset-dir /path/to/replay --output-dir /path/to/logs/pi05 \
  --robot-key multi_ur5_rh56dfx_with_flange

Training Flags

Key Training Defaults (from deploy_policy.yml)

Evaluation & Inference

Pi0.5 uses a split server/client architecture: a JAX model server runs in the openpi environment, while the Isaac Sim client runs in the dex2bench environment.

Single-Channel Evaluation

# All optional positionals (train_config, gpu_id, num_episodes, seed) use script defaults
bash policy/pi05/eval_double_env.sh 03_wine_glass_plate_balance \
  multi_ur5_rh56dfx_with_flange \               # ← positional robot key
  ./outputs/logs/pi05/pi05_base_dex2bench_full/my_exp/2000/ \
  --channel none \
  --anchor-dir /path/to/replay-generalization

All-Channel Evaluation (Recommended)

# robot key is the 2nd positional argument (multi_ur5_rh56dfx_with_flange)
# All optional positionals (train_config, gpu_id, num_episodes, seed) use script defaults
bash policy/pi05/eval_all_channels.sh 03_wine_glass_plate_balance \
  multi_ur5_rh56dfx_with_flange \             # ← positional robot key
  ./outputs/logs/pi05/pi05_base_dex2bench_full/my_exp/2000/ \
  /path/to/replay-generalization

The eval_all_channels.sh script runs all four generalization channels (none, cov, inv, inv_cov) sequentially and produces a single success_rates.tsv summary.

Key Evaluation Flags

4. GR00T N1.5

GR00T N1.5 is NVIDIA's open foundation model for generalist robot manipulation. It uses a vision-language backbone (Eagle 2.5) with a diffusion-transformer action head. Bench2Dex integrates a vendored copy of Isaac-GR00T with HDF5-native data loading and LoRA fine-tuning support. The default base model is the N1.5-derived WaveHand checkpoint from ModelScope.

Environment Setup

The setup script creates a conda environment, installs dependencies, and downloads the base model:

# Full setup (conda env + dependencies + model download)
bash policy/GR00T_n15/setup_env.sh

# Skip model download (environment only)
bash policy/GR00T_n15/setup_env.sh --skip-model-download

# Use a different model
bash policy/GR00T_n15/setup_env.sh \
  --model-id nv-community/GR00T-N1.5-3B-WaveHand

Setup Options

Training

GR00T N1.5 fine-tunes using HDF5-native mode — reading .hdf5 files directly without converting to LeRobot format. Training scripts support LoRA-based fine-tuning.

Basic Usage

# Single GPU (default)
bash policy/GR00T_n15/train.sh 03

# Multi-GPU with custom batch size
bash policy/GR00T_n15/train.sh 03 --gpu 0,1,2,3 --batch 64

# Custom hyperparameters with tag
bash policy/GR00T_n15/train.sh 03 --lr 2e-4 --steps 10000 --tag v1

# Foreground training (no tmux)
bash policy/GR00T_n15/train.sh 03 --no-tmux

Training Flags

Key Architecture Defaults

Evaluation & Inference

Server/Client Evaluation

bash policy/GR00T_n15/eval.sh 03_wine_glass_plate_balance \
  ../policy_ckpt/03/multi_ur5_rh56dfx_with_flange/gr00t_n15_tr00tbs64_step20000_gpu1_lr1/ \
  0 50 100000000

Double-Environment Evaluation (Channel Support)

# Single channel (no generalization)
bash policy/GR00T_n15/eval_double_env.sh 03

# Covariate generalization
bash policy/GR00T_n15/eval_double_env.sh 03 cov

# Full generalization
bash policy/GR00T_n15/eval_double_env.sh 03 inv_cov

# All 4 channels sequentially
bash policy/GR00T_n15/eval_double_env.sh 03 all

Available Flags for eval_double_env.sh

5. Tasks

Bench2Dex provides 26 long-horizon bimanual dexterous manipulation tasks spanning 12 robot embodiments. Each task involves coordinated two-hand operations such as precision grasping, tool use, articulated-object interaction, pouring, and multi-stage assembly. The table below lists all available scenes grouped by embodiment, with descriptions extracted from the scene configuration files in scenes/.