The Robot LIBERO Dataset

Liao Jiayi Liao Jiayi

Notes from exploring the LIBERO robotics dataset.

Translated from Chinese with AI · Read the original

HDF5 Data Format

  • A simple structure of datasets and groups, corresponding to data collections and folders.
  • The all-in-one file format feels cumbersome.
  • Perhaps mainly older datasets use this format? Object storage might be better.

A data-parsing demo:

# 文件结构遍历
with h5py.File(hdf5_file, "r") as f:
    # Top-level keys (Groups)
    print(f"\nTop-level keys: {list(f.keys())}")


    # 类似 posix 方式访问子目录数据
    agentview_rgb = f["data/demo_0/obs/agentview_rgb"]
    if isinstance(agentview_rgb, h5py.Group):
        print("agentview_rgb is a Group, keys:", list(agentview_rgb.keys()))
    elif isinstance(agentview_rgb, h5py.Dataset):
        print("agentview_rgb is a Dataset")
        print(f"  Shape: {agentview_rgb.shape}")
        print(f"  Dtype: {agentview_rgb.dtype}")
        print(f"  Size: {agentview_rgb.size}")


    # read metadata
    data_attrs = f["data"].attrs
    print("Data group attributes:")
    for key in data_attrs.keys():
        value = data_attrs[key]
        # Handle bytes/string conversion
        if isinstance(value, bytes):
            value = value.decode('utf-8')
        elif isinstance(value, np.ndarray) and value.dtype.kind == 'S':
            value = value.tobytes().decode('utf-8')
        print(f"  {key}: {value}")

An example LIBERO HDF5 structure, using open_the_middle_drawer_of_the_cabinet_demo.hdf5 from LIBERO-GOAL:

-data/demo_0
  -actions
  -dones
  -obs
    ---agentview_rgb
    ---ee_ori
    ---ee_pos
    ---ee_states
    ---eye_in_hand_rgb
    ---gripper_states
    ---joint_states
  -rewards
  -robot_states
  -states
  • Most arrays begin with shape (T, …). Does T represent time or an action episode?
  • actions has action_dim=7, apparently seven degrees of freedom: x,y,z position; roll, pitch, and yaw of the end effector (EE); and the gripper.
  • dones indicates whether the current task has ended, perhaps used by robomimic to split videos?
  • obs: What the policy sees or receives.
    • Vision
      • agentview_rgb: (T,H,W,3) images/frames from an external camera.
      • eye_in_hand_rgb: (T,H,W,3) images from the wrist-mounted camera.
    • End-effector state
      • ee_ori: (T,3), corresponding to roll, pitch, and yaw.
      • ee_pos: x,y,z position.
      • ee_stats: Combined EE state, including velocity, force, and pose.
      • gripper_states: (T,1), gripper open/closed state.
      • joint_state: (T,n-Joints), all joint states.
  • rewards: Rewards during training.
  • robot_states: A narrower robot-centered view, (T,9).
  • states: An omniscient view of everything in the environment, (T,79).

Note: I initially expected ee_pos[T+1] - ee_pos[T] to match actions, but that overlooks two points:

  1. ee_pos and actions do not share the same coordinate system.
  2. Actions are commands and need not match the resulting ee_pos, given controller gains, damping, and other effects.

BDLL Task Description

  • Define the problem.
  • Use robosuite as the simulator.
  • Define the instruction.
  • regions: Spatial definitions for plates, bowls, bottles, and so on.
  • fixtures: Static scene objects the robot does not grasp.
  • objects: Manipulable objects.
  • obj_of_interest: Objects relevant to the target.
  • init: Initial state; On describes relative relationships between objects.
  • target: Goal conditions.
(define (problem LIBERO_Tabletop_Manipulation)
  (:domain robosuite)
  (:language Open the middle layer of the drawer)
    (:regions
      (plate_region
          (:target main_table)
          (:ranges (
              (0.04 -0.03 0.060000000000000005 -0.01)
            )
          )
      )
      (akita_black_bowl_region
          (:target main_table)
          (:ranges (
              (-0.09999999999999999 -0.01 -0.08 0.01)
            )
          )
      )
      (wine_bottle_region
          (:target main_table)
          (:ranges (
              (-0.21000000000000002 -0.060000000000000005 -0.19 -0.04)
            )
          )
      )
      (cream_cheese_region
          (:target main_table)
          (:ranges (
              (-0.060000000000000005 0.12000000000000001 -0.04 0.14)
            )
          )
      )
      (stove_front_region
          (:target main_table)
          (:ranges (
              (-0.09 0.16999999999999998 -0.010000000000000002 0.25)
            )
          )
      )
      (cabinet_region
          (:target main_table)
          (:ranges (
              (0.02 -0.25 0.04 -0.23)
            )
          )
          (:yaw_rotation (
              (3.141592653589793 3.141592653589793)
            )
          )
      )
      (stove_region
          (:target main_table)
          (:ranges (
              (-0.42 0.2 -0.4 0.22)
            )
          )
      )
      (wine_rack_region
          (:target main_table)
          (:ranges (
              (-0.27 -0.27 -0.25 -0.25)
            )
          )
          (:yaw_rotation (
              (3.141592653589793 3.141592653589793)
            )
          )
      )
      (top_region
          (:target wooden_cabinet_1)
      )
      (middle_region
          (:target wooden_cabinet_1)
      )
      (bottom_region
          (:target wooden_cabinet_1)
      )
      (top_side
          (:target wooden_cabinet_1)
      )
      (cook_region
          (:target flat_stove_1)
      )
      (right_region
          (:target bowl_drainer_1)
      )
      (left_region
          (:target bowl_drainer_1)
      )
      (top_region
          (:target wine_rack_1)
      )
    )


  (:fixtures
    main_table - table
    wooden_cabinet_1 - wooden_cabinet
    flat_stove_1 - flat_stove
    wine_rack_1 - wine_rack
  )


  (:objects
    akita_black_bowl_1 - akita_black_bowl
    cream_cheese_1 - cream_cheese
    wine_bottle_1 - wine_bottle
    plate_1 - plate
  )


  (:obj_of_interest
    wooden_cabinet_1_middle_region
  )


  (:init
    (On wine_bottle_1 main_table_wine_bottle_region)
    (On akita_black_bowl_1 main_table_akita_black_bowl_region)
    (On plate_1 main_table_plate_region)
    (On cream_cheese_1 main_table_cream_cheese_region)
    (On wooden_cabinet_1 main_table_cabinet_region)
    (On flat_stove_1 main_table_stove_region)
    (On wine_rack_1 main_table_wine_rack_region)
  )


  (:goal
    (And (Open wooden_cabinet_1_middle_region))
  )
)

Rendering a Demo Locally

Using open_the_middle_drawer_of_the_cabinet.bddl from LIBERO-GOAL as an example:

from libero.libero.benchmark import get_benchmark
import os


bm = get_benchmark("libero_goal")()
print(bm.get_task_names())


print(len(bm.tasks))
for t in bm.tasks[:5]:
    print(t.name)






from libero.libero.envs import OffScreenRenderEnv, DemoRenderEnv


task = bm.tasks[0]
print("----")
print("name:", task.name)
print("language:", task.language)
print("problem:", task.problem)
print("problem_folder:", task.problem_folder)
print("bddl_file:", task.bddl_file)
print("init_states_file:", task.init_states_file)
print("----")


# For GUI rendering on Mac, use ControlEnv with has_renderer=True
# DemoRenderEnv uses offscreen rendering (no GUI window)
#
# Note: On Mac, if you get OpenGL errors, you may need to:
# 1. Set environment variable: export PYOPENGL_PLATFORM=osmesa (for headless)
#    OR use: export PYOPENGL_PLATFORM=glfw (for GUI - requires XQuartz or similar)
# 2. Install: brew install glfw (if using glfw backend)
# 3. Alternative: Use OffScreenRenderEnv and visualize frames with matplotlib (see below)


from libero.libero.envs.env_wrapper import ControlEnv


env = ControlEnv(
    bddl_file_name='./libero/libero/bddl_files/libero_goal/open_the_middle_drawer_of_the_cabinet.bddl',
    camera_names=["agentview"],
    has_renderer=True,          # Enable GUI window for visualization
    has_offscreen_renderer=True, # Required for camera observations
    render_camera="frontview",   # Camera view for rendering
)


# Alternative: If GUI doesn't work on Mac, use offscreen rendering and visualize:
# from libero.libero.envs import OffScreenRenderEnv
# import matplotlib.pyplot as plt
# env = OffScreenRenderEnv(
#     bddl_file_name='./libero/libero/bddl_files/libero_goal/open_the_middle_drawer_of_the_cabinet.bddl',
#     camera_names=["agentview"],
# )
# obs = env.reset()
# plt.imshow(obs['agentview_image'][::-1])  # Flip vertically for display
# plt.show()


obs = env.reset()
print(obs.keys())


# Render the environment (this opens the GUI window on Mac)
# Access the underlying robosuite environment's render method
env.env.render()


import numpy as np
import time
import h5py
from libero.libero import get_libero_path


# Load actions from HDF5 demonstration file
print("\n" + "=" * 60)
print("Loading demonstration from HDF5 file...")
print("=" * 60)


# Get the demonstration file path for this task
demo_file_path = os.path.join(
    get_libero_path("datasets"),
    bm.get_task_demonstration(0)  # Get demo for task 0
)
print(f"Loading demo from: {demo_file_path}")


# Load actions from HDF5 file
with h5py.File(demo_file_path, "r") as f:
    # Get first demo (you can change demo_0 to demo_1, demo_2, etc.)
    demo_key = "demo_0"
    actions = f[f"data/{demo_key}/actions"][()]  # Load all actions into memory
    states = f[f"data/{demo_key}/states"][()]  # Load initial states


    print(f"Loaded {len(actions)} actions from {demo_key}")
    print(f"Action shape: {actions.shape}")
    print(f"Action range: [{actions.min():.4f}, {actions.max():.4f}]")


    # Optionally set initial state from the demo
    if len(states) > 0:
        print(f"Setting initial state from demonstration...")
        obs = env.set_init_state(states[0])  # set_init_state returns observations
    else:
        obs = env.reset()


# Action format for OSC_POSE: [dx, dy, dz, droll, dpitch, dyaw, gripper]
# All values are typically in range [-1, 1]
# - First 3: position delta (translation)
# - Next 3: orientation delta (rotation)
# - Last 1: gripper action (-1=close, 1=open)


print(f"\nReplaying demonstration with {len(actions)} steps...")
print("=" * 60)


# Replay the demonstration actions
print(f"\nReplaying demonstration actions...")
for t, action in enumerate(actions):
    # Optionally inject random actions during demo
    if np.random.random() < 0.1:
        # Replace with random action
        action = np.random.uniform(
            -1,
            1,
            size=7
        )
        action_type = "random"
    else:
        action_type = "demo"


    obs, reward, done, info = env.step(action)


    # Render after each step to see the animation
    env.env.render()


    # Small delay to make animation visible (adjust speed here)
    # 0.01 = real-time, 0.05 = slower, 0.001 = faster
    time.sleep(0.01)


    # Print progress every 50 steps
    if (t + 1) % 50 == 0:
        print(f"Step {t+1}/{len(actions)} ({(t+1)/len(actions)*100:.1f}%)")


    if done:
        print(f"\nTask completed at step {t+1}!")
        break


print(f"\nDemonstration replay complete!")


# Close the environment when done
env.close()