Cross-Embodiment Robot Training: Can You Transfer Across Different Robots?

The Cross-Embodiment Hypothesis

The central claim of cross-embodiment learning: training a robot policy on data from multiple different robot types — different arm designs, different DOF counts, different kinematics — produces a model that generalizes better to new robots and new tasks than a model trained on a single robot type alone. This hypothesis was intuitive from the language model analogy (training on more diverse text helps) but non-obvious for robotics, where action spaces, kinematics, and physical capabilities differ fundamentally across platforms.

The hypothesis matters practically because robot data is expensive. At SVRC, collecting 100 expert demonstrations on an OpenArm takes approximately 4 hours of operator time at $75/hour — that's $300 per task. If you can leverage 100,000 demonstrations collected on other robot types by other labs, your effective cost per useful training example drops by orders of magnitude. The question is whether data from a WidowX actually helps train a policy for an OpenArm, and if so, how much.

Open X-Embodiment: The Evidence

The Open X-Embodiment project (Padalkar et al., 2023; RTX collaboration paper) is the most comprehensive test of this hypothesis. The dataset contains 527 skills across 22 robot types, representing 160,000+ robot episodes contributed by labs around the world. The key result: RT-X, trained on the full multi-embodiment dataset, outperformed single-robot specialists by approximately 50% on held-out generalization tasks.

This is a striking result. The multi-embodiment model was not given any additional information about the target robot — it was simply trained on more diverse data. The diversity itself was the advantage.

Dataset Statistics in Detail

The OXE dataset is unevenly distributed: roughly 60% of episodes come from just 3 robot types (WidowX, Franka, Everyday Robot). The remaining 19 robot types each contribute 1,000–5,000 episodes. This imbalance matters because the multi-embodiment benefit is partly driven by the most data-rich platforms. When you train on the full OXE dataset, you're mostly learning WidowX/Franka/Everyday Robot manipulation with a diversity bonus from the other 19 platforms.

RT-2-X: The VLA That Proved Cross-Embodiment Works

RT-2-X extended the RT-2 vision-language-action model to consume the full OXE dataset. The architecture is a 55B parameter PaLM-E model that takes images and language instructions as input and outputs discretized robot actions. The key results:

• +50% on emergent evaluations: Tasks requiring novel object combinations or spatial relationships that weren't directly in the training data for any single robot.
• +25% on in-distribution tasks: Even for tasks well-represented in single-robot datasets, the multi-embodiment model was better.
• Zero-shot transfer: RT-2-X could control robots not in its training set at above-chance success rates, though performance was substantially below fine-tuned models.

The 55B parameter count is important context: this is not a model you can run on a lab GPU. Inference requires multiple A100s or H100s. The practical version for most teams is a smaller model (1–7B) fine-tuned on the OXE data, which captures most of the cross-embodiment benefit at deployable scale.

Octo: The Practical Cross-Embodiment Foundation Model

Octo (Ghosh et al., 2024) is the model most teams should start with for cross-embodiment transfer. It's a 93M parameter transformer trained on 800K robot episodes from the OXE dataset, designed specifically for efficient fine-tuning to new robots and tasks. Key properties:

• Architecture: Transformer with modality-specific tokenizers for images (ViT), language (T5), and proprioception (MLP). Action heads are embodiment-specific — you swap or add an action head when fine-tuning to a new robot.
• Fine-tuning cost: 50–200 demonstrations + 2–4 hours on a single A100 GPU. This is within reach of most research labs.
• Performance: Octo fine-tuned with 100 target-robot demos outperforms training from scratch with the same 100 demos by 25–40% on pick-and-place tasks across WidowX, Franka, and other platforms tested.
• Inference speed: 93M parameters runs at ~15 Hz on a single RTX 4090 or at ~8 Hz on a Jetson AGX Orin — fast enough for real-time manipulation control.

# Fine-tuning Octo on your own robot data (simplified)
# Requires: pip install octo-model jax[cuda]

from octo.model.octo_model import OctoModel
from octo.data.dataset import make_single_dataset

# Load pre-trained cross-embodiment model
model = OctoModel.load_pretrained("hf://rail-berkeley/octo-base")

# Load your target robot dataset (LeRobot HDF5 or RLDS format)
target_dataset = make_single_dataset(
    dataset_kwargs={
        "name": "my_openarm_dataset",
        "data_dir": "/data/openarm_pick_place/",
        "image_obs_keys": {"primary": "image_wrist", "secondary": "image_overhead"},
        "state_obs_keys": ["joint_positions"],
        "language_key": "language_instruction",
    },
    traj_transform_kwargs={"window_size": 2},
    frame_transform_kwargs={"image_size": (256, 256)},
)

# Fine-tune with new action head for your robot's action space
# OpenArm: 6 joint positions + 1 gripper = 7D action space
model = model.finetune(
    target_dataset,
    action_dim=7,
    action_head="diffusion",       # or "mse" for deterministic
    learning_rate=3e-4,
    batch_size=256,
    num_steps=50_000,              # ~2 hours on A100
    save_dir="./octo_openarm_finetuned/",
)

DROID: Scale Within Cross-Embodiment

The DROID paper (Khazatsky et al., 2024) extended this analysis with a focus on scale. 76K trajectories across 22 robots and 564 environments. The key DROID finding relevant to cross-embodiment: adding demonstrations from different robot types improved Diffusion Policy and ACT performance on a target robot even when the additional robot types were kinematically quite different (WidowX vs. UR5, for example).

The improvement was not uniform — adding data from very similar robots (WidowX + WidowX-XL) helped more than adding data from very different robots (WidowX + mobile manipulator). But even distant cross-embodiment data provided a positive signal, suggesting that shared visual and semantic representations carry useful information across different physical platforms.

Transfer Benefit by Robot Similarity

The pattern is clear: transfer benefit correlates with kinematic similarity, but even moderate-similarity robots provide meaningful improvement. The practical takeaway is that if you have 100 demonstrations on your target robot, adding 1,000+ demonstrations from kinematically similar robots can give you performance equivalent to 150–200 demonstrations on your target robot alone — a 50–100% effective data multiplication at zero marginal collection cost.

Why Cross-Embodiment Transfer Works

Three mechanisms appear to be responsible:

1. Shared visual features: Regardless of which robot is executing a task, the visual inputs (objects, workspace, task structure) are similar. A model trained on 22 robot types develops richer visual representations for manipulation-relevant features — edge detection around graspable surfaces, object state estimation (open/closed, full/empty), spatial relationship encoding (on top of, inside, next to) — than one trained on a single robot. The visual encoder benefits from data diversity even when the action decoder does not.

2. Action space abstraction in VLAs: Vision-Language-Action models like OpenVLA and RT-2-X represent actions in a tokenized, abstract space that partially decouples task knowledge from specific robot kinematics. When actions are discretized into bins (e.g., 256 bins per dimension), a "move right" token from a WidowX carries similar semantic meaning to a "move right" token from a Franka, even though the underlying joint trajectories differ completely. This abstraction allows task-level knowledge transfer across kinematically different platforms.

3. Language grounding: When language instructions are part of the model input (as in RT-2-X, Octo, OpenVLA), the model learns a shared language-to-behavior mapping across embodiments. "Pick up the red cup" means the same thing regardless of which robot is executing it. This shared linguistic grounding provides a universal task representation that transfers perfectly across embodiments, even when the motor execution strategies differ.

Where Transfer Fails

• Very different kinematics: Transfer between wheeled mobile platforms and fixed-base arms is near zero. The action space mismatch is too large for the shared visual features to overcome. A mobile base's (vx, vy, omega) actions provide no useful gradient information for a 6-DOF arm's joint position actions.
• Gripper type mismatch: Data from parallel jaw grippers transfers poorly to suction cup robots and vice versa. The contact interaction model is too different. Parallel jaw data transfers to other parallel jaw robots well (even with different finger geometries), but the grasp-phase behavior learned from parallel jaw demonstrations is actively harmful for suction-based grasping.
• DOF scale mismatch: Data from 7-DOF arms transfers to other 7-DOF arms well, but to 4-DOF arms poorly. The dimensionality difference creates action space coverage problems: a 7-DOF arm can approach objects from many angles, while a 4-DOF arm is constrained. Policies trained on 7-DOF data develop approach strategies that are physically unreachable for 4-DOF hardware.
• Speed and dynamics mismatch: Data from high-speed industrial arms (cycle times <0.5s) transfers poorly to research arms with 3–5s cycle times for the same task. The temporal dynamics of demonstrations — velocity profiles, acceleration patterns, timing of grasp closure — are fundamentally different. The visual observations may look similar frame-by-frame, but the action sequences have different rhythms.
• Contact-rich vs. contact-free: Data from contact-rich tasks (screwing, insertion, polishing) provides minimal transfer benefit for contact-free tasks (reaching, visual inspection) and vice versa. The learned policies develop contact anticipation behaviors (slowing before expected contact, stiffening specific joints) that are task-type-specific rather than embodiment-general.

Practical Co-Training Setup

If you want to leverage cross-embodiment data for your specific robot and task, here is the workflow that produces the best results in our experience at SVRC:

Step 1: Select Your Base Datasets

From the OXE collection, select datasets from robots with similar DOF and gripper type to your target. For a 6-DOF arm with parallel jaw gripper (like OpenArm), the best sources are: Bridge V2 (WidowX, 60K episodes), DROID subset (6-DOF arms, ~15K episodes), and the SVRC shared pool (OpenArm + similar, ~10K episodes). Avoid mobile manipulation, dexterous hand, and high-DOF datasets — they add training cost without benefit.

Step 2: Normalize Action Spaces

Different robots have different joint configurations, so raw joint-space actions don't transfer. The standard approach is to convert all actions to end-effector space: (dx, dy, dz, droll, dpitch, dyaw, gripper). This 7D delta end-effector representation is universal across all single-arm manipulators with any DOF count and any gripper. The conversion uses each robot's forward kinematics (URDF + FK solver) and is a one-time preprocessing step per dataset.

# Action space normalization: joint space -> EE delta space
import numpy as np
from lerobot.common.robot_devices.robots.configs import OpenArmConfig

def joint_to_ee_delta(joint_actions, joint_states, robot_config):
    """Convert joint-space actions to end-effector delta actions.

    This normalization is required for cross-embodiment training:
    different robots have different joints but share the same
    6D end-effector space.
    """
    ee_deltas = []
    for i in range(len(joint_actions)):
        # FK at current state
        ee_current = robot_config.forward_kinematics(joint_states[i])
        # FK at next state (current + action)
        ee_next = robot_config.forward_kinematics(
            joint_states[i] + joint_actions[i, :robot_config.n_joints]
        )
        # Delta in EE space
        delta_pos = ee_next[:3] - ee_current[:3]
        delta_rot = quaternion_to_axis_angle(
            quaternion_multiply(ee_next[3:7], quaternion_inverse(ee_current[3:7]))
        )
        gripper = joint_actions[i, -1]  # gripper action passes through
        ee_deltas.append(np.concatenate([delta_pos, delta_rot, [gripper]]))
    return np.array(ee_deltas)

Step 3: Dataset Mixing Strategy

The mixing ratio between your target-robot data and cross-embodiment data matters significantly. Too much cross-embodiment data overwhelms target-specific fine details; too little provides no benefit. Our recommended ratios based on target dataset size:

• <50 target demos: Mix 1:10 (target:cross). The model needs the diversity because it has very little target signal. Weight cross-embodiment data from the most similar robots higher.
• 50–200 target demos: Mix 1:5. This is the sweet spot where cross-embodiment provides the most relative benefit — the model has enough target data to learn the specific robot's dynamics while leveraging cross-embodiment for visual and task representations.
• 200–1,000 target demos: Mix 1:2. Cross-embodiment data helps but the marginal benefit is decreasing. Focus on the highest-quality cross-embodiment datasets only.
• >1,000 target demos: Cross-embodiment pre-training is still beneficial (use it as initialization), but mixing during fine-tuning provides diminishing returns. Train from an Octo or OpenVLA checkpoint, then fine-tune on target data only.

Step 4: Training Configuration

Use embodiment-conditional training: add a learnable embodiment token (one per robot type) to the model's input sequence. This allows the model to learn embodiment-specific behavior patterns while sharing visual and task representations across all robots. During inference on your target robot, the model uses your robot's embodiment token. Cost: ~8 hours on an A100 for a full co-training run with 200K mixed episodes.

Co-Training with HuggingFace LeRobot

The most accessible way to run cross-embodiment co-training today is through HuggingFace's LeRobot framework, which provides standardized data loading, model training, and evaluation infrastructure:

# LeRobot co-training config (YAML)
# Save as: configs/co_train_openarm.yaml

dataset:
  # Primary: your target robot data
  repo_id: "svrc/openarm_pick_place_v2"
  mix_datasets:
    # Cross-embodiment data, weighted by similarity
    - repo_id: "lerobot/bridge_v2"
      weight: 0.3    # WidowX - moderate similarity
    - repo_id: "lerobot/droid_franka"
      weight: 0.2    # Franka - moderate-high similarity
    - repo_id: "svrc/shared_pool_6dof"
      weight: 0.4    # SVRC pool - high similarity
  action_space: "ee_delta_7d"  # Normalized EE space

policy:
  name: "diffusion"
  pretrained: "lerobot/diffusion_oxe_base"  # OXE pre-trained
  chunk_size: 16
  n_obs_steps: 2
  n_diffusion_steps: 100

training:
  batch_size: 256
  learning_rate: 1e-4
  num_epochs: 200
  eval_freq: 10
  device: "cuda"

# Run: python lerobot/scripts/train.py --config configs/co_train_openarm.yaml

Measuring Cross-Embodiment Benefit

To know whether cross-embodiment data is actually helping your specific case, run a controlled experiment:

1. Baseline: Train on your target-robot data only. Record success rate on 50+ evaluation episodes.
2. +Cross-embodiment: Train on your target data mixed with cross-embodiment data. Same model architecture, same hyperparameters, same evaluation protocol.
3. +Pre-trained init: Initialize from an Octo/OpenVLA checkpoint, then fine-tune on your target data only.
4. Full pipeline: Initialize from pre-trained checkpoint AND mix cross-embodiment data during fine-tuning.

In our experience, option 3 (pre-trained init + target-only fine-tuning) beats option 2 (random init + mixed training) for most teams. The pre-trained visual representations are the primary source of transfer benefit, and they come "for free" from the checkpoint. Option 4 provides a further 5–10% improvement over option 3 but doubles training time and complexity.

Current Limitations

• Negative transfer is real: Adding data from very dissimilar robots (different DOF, different gripper type, different task domain) can hurt performance compared to target-only training. Always validate with the controlled experiment above.
• Action space normalization loses information: Converting to EE delta space discards redundancy resolution information (how the arm's elbow/shoulder configuration changes). For tasks where arm configuration matters (reaching into cluttered environments, avoiding obstacles with the elbow), joint-space fine-tuning after EE-space co-training is sometimes necessary.
• Dataset quality variance: OXE and DROID datasets vary enormously in demonstration quality. Some contributing labs had expert operators; others used novice operators or partially automated systems. Low-quality demonstrations from cross-embodiment sources can introduce noise that degrades your target policy. Curate your cross-embodiment mix carefully.
• Computational cost scales linearly: More cross-embodiment data means proportionally more training compute. For a startup with a single GPU, a full OXE co-training run takes 3–5 days. Pre-trained checkpoints (Octo, OpenVLA) amortize this cost — the open-source community has already paid the compute bill.
• Sim-to-real gap compounds: If your cross-embodiment data includes simulation data (which some OXE datasets do), the sim-to-real gap adds to the cross-embodiment gap. Filter simulation-derived datasets from your cross-embodiment mix unless you've verified they help on your specific task.

The Future: Universal Robot Foundation Models

The trajectory is clear: the robotics field is moving toward foundation models trained on data from hundreds of robot types, millions of episodes, and thousands of tasks — analogous to how GPT was trained on billions of web pages. Projects like GR-2 (ByteDance), pi0 (Physical Intelligence), and the next generation of RT-X models are scaling toward this vision. When these models mature, cross-embodiment transfer will shift from "careful co-training recipe" to "download checkpoint, fine-tune 30 minutes, deploy." We are not there yet in 2026, but the gap is closing fast.

For teams building today, the practical advice is: collect your data in standardized formats (LeRobot HDF5 or RLDS), with full calibration metadata, so you can benefit from these foundation models as they become available. The data you collect today in the right format will be more valuable in 12 months than it is now.

Data Sharing Through SVRC

SVRC maintains a shared cross-embodiment dataset pool that clients can contribute to and draw from. When you collect demonstration data through SVRC's data services, you have the option to contribute anonymized demonstrations to the shared pool in exchange for access to cross-embodiment pre-training data from other platforms. This reduces your effective per-task data requirement by leveraging the cross-embodiment transfer effect. The pool currently contains 25,000+ episodes across OpenArm, WidowX, Franka, UR5e, and Unitree Z1 platforms, with new contributions added weekly.

For teams that want to contribute data collected on their own hardware, SVRC provides a validation pipeline that checks format compliance, demonstration quality (smoothness, success rate, camera calibration), and privacy (no identifiable information in camera views). Validated contributions earn data credits redeemable against any data in the pool.

Related Reading