(copy) MIT IAP 2025 / 6.S186

• Source: https://mp.weixin.qq.com/s/W1tUVwkWH2s3GQsbp6F_lA

Observation Mismatch

Observation mismatch refers to differences between what is observed in human videos and what a robot actually observes. The video’s viewpoint is not the robot’s viewpoint.

Missing Actions

1. Extract actions Human pose estimation in computer vision can accomplish this. Define keypoints on the body, usually joints such as elbows, wrists, and finger joints. Detect and connect them to form a skeleton, yielding a low-dimensional state space, typically around 17 degrees of freedom depending on the joints defined. Differences in joint states between consecutive frames then provide human action information.

2. Retarget actions This field is called action retargeting. Determine the positions of human hand joints or terminal links, define reference coordinate frames on the robot, and calculate relative distances between corresponding joints. Inverse kinematics then computes the robot’s kinematic parameters to produce robot actions.

Learning from Human Video Data

1. Pretraining and fine-tuning: Pretrain a policy network on human videos and extracted human actions, then fine-tune it with teleoperated robot data, using robot observations as inputs and robot actions as outputs.
2. Joint training: Feed human and robot images into the same network, which outputs both human and robot actions. Train all data in the same batch to force the network to learn embodiment-independent action patterns, such as how to move an end effector regardless of whether it belongs to a human or a robot.
3. Coarse actions with human-in-the-loop correction: Train a policy on human observations and coarse robot actions. Because these actions cannot be applied directly to real robots, use human-in-the-loop correction, essentially a variation of pretraining and fine-tuning.

world model

How Can Actions Be Recovered When a World Model Lacks Action Information?

1. Learn an inverse dynamics model: Collect a robot dataset through teleoperation. Train a network to take observations at two time steps (two images) and predict the action that moves the robot from the first state to the second. The drawback is that it still requires teleoperated robot data, contrary to the original goal of learning from video alone.
2. Learn correspondences from optical flow: Use mature computer-vision libraries such as OpenCV to compute optical flow between two frames. Use that flow to establish robot-arm correspondences, constructing a matrix that lets the robot reproduce the predicted flow. The advantage is that no dedicated training is needed; existing techniques can directly recover actions.

Teleoperation vs. Simulation vs. Video

1. Teleoperation: Expensive, requiring many people to collect data. It is inefficient and limited: teleoperation imposes a mental burden on operators, reducing efficiency, and many tasks are difficult to complete.
2. Simulation: Building environments is extremely expensive. Real-world environments are diverse, and recreating them in simulation requires substantial effort from PhDs or highly paid engineers. Task scalability is poor: it is impossible to build nearly as many simulated environments as we encounter in daily life, making complex-task training difficult. Task scalability is poor: it is impossible to build nearly as many simulated environments as we encounter in daily life, making complex-task training difficult.
3. Video: Observations are limited to vision and lack tactile feedback. Occlusion is a problem: severe occlusion prevents actions from being recovered visually. Generative models are unreliable: AI systems such as GPT may generate robot videos that fail to follow instructions. For example, a request for a robot arm pushing a red block may instead produce a shaking table without the pushing action.