Paper Notes

Liao Jiayi Liao Jiayi

Notes on papers I have read.

Translated from Chinese with AI · Read the original

Notes on papers I have read.

Model Architectures

GR00T N1.5

gr00t

RoboBrain 2.5: Depth in Sight, Time in Mind.

  • A recent update from the Beijing Academy of Artificial Intelligence, based on Qwen-VL.
  • Configuration: model.config; inference code: https://github.com/FlagOpen/RoboBrain2.5/blob/main/inference.py. It appears to train directly on Qwen’s architecture. If the architecture is unchanged, is the main difference the data?
  • The paper claims precise 3D reasoning and dense temporal value estimation, but I did not see how the code implements them.

OpenVLA

  • Proposed by Stanford over a year ago.
  • A 7B model using Open X-Embodiment. From today’s perspective, little seems unusual; it looks similar to UniVLA.

We train OpenVLA by fine-tuning a pretrained Prismatic-7B VLM. Our model consists of three key elements:
(1) a fused visual encoder, consisting of a SigLIP and a DinoV2 backbone, that maps image inputs to a number of ``image patch embeddings’’,
(2) a projector that takes the output embeddings of the visual encoder and maps them into the input space of a large language model, and
(3) a Llama 2 7B language model backbone that predicts tokenized output actions.
These tokens get decoded into continuous output actions that can be directly executed on the robot.

openvla

SONIC: Supersizing Motion Tracking for Natural Humanoid Whole-Body Control

  • NVIDIA introduces motion tracking to address customized RL around individual models, policies, and rewards.
  • Combined with the GR00T VLA model, it resembles Helix’s division between higher- and lower-level control.
  • I will examine it more closely later.

π0 model by PI

  • Released in mid-2025: Internet data for pretraining (VLA), robot demonstration data for post-training (VLM), as noted here.
  • github: https://github.com/Physical-Intelligence/openpi
    • Looking through pi0.py, training in compute_loss uses flow matching around x_t = time_expanded * noise + (1 - time_expanded) * actions. Add random noise and use the difference between the v_t predicted by action_out_proj and actual u_t as loss. Inference reverses the process, solving an ODE from noise toward actions.
  • PaliGemma is the core model; other linear layers convert dimensions. action_dim maps into the action expert and an MLP maps back.
    • action_dim: The robot’s action-space dimension, defined by data and task.
    • action_expert: A 300M Gemma subnetwork with width 1024, defined by the architecture.
      • This expert differs from models predicting directly in a latent action space.
  • Cross-embodiment training: How are differently formatted embodiment datasets combined?

openpi0

GR-2: A Generative Video-Language-Action Model with Web-Scale Knowledge for Robot Manipulation

  • Released by Seed in October 2024.
  • Very few details.

Genie

  • A 2024 Google DeepMind world model. Genie 3 is also attracting attention now; interesting work.
  • Characteristics:
    • Trained only on video without supervision; action labels are inferred from frames.
    • For robotics, removing RT-1 action data and training on video alone still produced consistent action latents: frame(n) -> latent space -> frame(n+1). If this generalizes, could robot action latents come entirely from world models?
  • Architecture:
    • Latent action model: A VQ-VAE with an eight-entry action codebook, mainly generating labels for training and unused during inference.
    • Video tokenizer: Separately trained VQ-VAE. How is it trained? Latent z contains video information from 1.
    • Main model: Input z(1) + a(1), predict frame(T), and compare with ground-truth video.
  • Inference predicts the next frame from video tokens and user actions, currently forward, backward, left, and right.

Helix02

  • 26.1.29 Helix02
  • System 2 (S2) reasons slowly about goals: scene interpretation, language understanding, and behavior sequencing. A 7B model.
  • System 1 (S1) translates perception into full-body joint targets at 200 Hz. An 80M model.
    • Adds a Transformer over S2’s latent space. What is the action-space dimension?
  • System 0 (S0) executes at 1 kHz, handling balance, contact, and whole-body coordination. Together they connect pixels to torque. A 10M model.
    • I do not fully understand S0’s role. Is coordination not already the result of joint actions? Why separate it from S1?
  • Too little material is available.

UnifoLM-VLA-0

  • Open-sourced by Unitree on January 29, 2026. Fine-tunes Qwen-VL-7B with 340 hours of real-robot data.
  • Some code looks odd, such as TensorFlow structures wrapped inside PyTorch in the data loader.

Lingbot-VLA

  • Open-sourced by Ant in January 2026, with 20,000 training hours and a data-scaling study.
  • Observations combine three camera views, task instructions, and robot state.
  • Flow matching trains predicted actions toward real actions.
  • A projection layer supervises depth from the three images using LingBot-Depth outputs.
  • Too few model details. Architecture optimization only discusses HSDP and fusion. Is there more substance?
  • The GitHub evaluation section also seems thin.

Cosmos Policy: Fine-Tuning Video Models for Visuomotor Control and Planning

  • Proposed on January 22, 2026, using NVIDIA Cosmos. It took me some time to understand.
  • A Transformer-based diffusion model with two notable points:
    1. Uses Cosmos-Predict2’s video world model and its understanding of the physical world.
    2. Latent-frame injection adds modalities without changing the architecture, directly injecting robot state, actions, and value functions into latent space.
  • This seems applicable to many robot datasets, suggesting good generalization.

cosmos

CLAP: Contrastive Latent Action Pretraining for Learning Vision-Language-Action Models from Human Videos

  • Proposed by Tsinghua on January 7, aligning video and robot trajectories through a codebook.
  • Seems very theoretical; I am not convinced.

V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning

  • Meta’s June 2025 work on multimodal representations.
  • I found it through Action100M and will inspect the details later.

Flow Matching for Generative Modeling

  • Published in 2023. I studied it with GPT after seeing flow matching used for Qwen’s multimodal features.
  • It can be compared with a continuous, infinitely deep Transformer.
  • In multimodal settings, diffusion noise can damage token semantics, unlike image noise.
  • Separate encoders are moved into a unified tokenizer.
  • Compared with diffusion: (1) FM uses a simpler linear formulation; (2) FM predicts a derivative or velocity, x_t = t*noise + (1-t)*action;v_t(dx_t/dt) = noise - action; loss = model(action,noise,ts) - v_t, while diffusion compares predicted noise with added noise, loss = (noise - noise_pred).

Dream2Flow: Bridging Video Generation and Open-World Manipulation with 3D Object Flow

  • From Fei-Fei Li’s lab, related to PointWorld.
  • Converts task descriptions into 3D-flow trajectories for robots to execute, loosening the coupling between tasks and embodiments.
  • Recovering 3D trajectories from 2D video is technically interesting. I want to inspect the implementation once open-sourced.

mHC: Manifold-Constrained Hyper-Connections

  • Released by DeepSeek on December 31, 2025.
  • Residual and hyperconnection architectures add inputs back to outputs. Across many layers, this can lead to exploding gradients or unstable training.
  • mHC constrains learnable HC matrices through orthogonality-related methods to preserve norms: length stays constant while representations rotate or transform, preventing uncontrolled growth across layers.

mhc

Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Models

  • ACL 2024, a video-conversation model.
  • Also provides a dataset and benchmark: https://github.com/mbzuai-oryx/Video-ChatGPT?tab=readme-ov-file.
  • Adds CLIP ViT-L/14, representing video spatially as N patch tokens and temporally as T frames, then pools and projects into language space.
  • Data generation combines manual and semiautomatic annotation.
    • Manual: Enrich ActivityNet-200 descriptions.
    • Semiautomatic:
      • Key frames: BLIP-2 generates frame-level captions.
      • Detailed frame descriptions: GRiT provides dense captions of scene objects.
      • Tags: Tag2Text, followed by noise removal.
      • GPT-3.5 enriches the combined content.

video-chatgpt

Unified Vision-Language-Action Model

  • Released in June 2025.
  • Based on the Emu3 pretrained model, described here as from Zhipu. Language, vision, and actions share a token space, though with separate tokenizers, and an autoregressive Transformer predicts them.
  • Uses extensive video data and Physical Intelligence’s FAST tokenizer.
  • Unlike common VLA pipelines with at least two stages, vision -> language -> action, where semantics may constrain actions.
  • Two training stages:
    • Post-training without actions: Lt, the task description, plus initial vision Lv predicts Lv(t+1), Lv(t+2), and onward.
    • Fine-tuning: Add actions.

uniVLA

PointWorld: Scaling 3D World Models for In-The-Wild Robotic Manipulation

  • Given a robot description in URDF and RGB images, infer subsequent motion trajectories.
  • Represent the environment and robot actions as 3D point flow.
  • Real-time inference around 0.1 seconds; some scenarios need no post-training.
  • Dataset: DROID followed by the annotation pipeline below.
  • Training suppresses noise through pixel-level weights.
  • annotation pipeline:
    1. Foundation-Stereo produces pixel depths, working well around 30-80 cm.
    2. VGGT, Visual Geometry Grounded Transformer, estimates global pose: relative camera/robot orientation and translation (R,t).
    3. CoTracker3 predicts future pixel trajectories.

Datasets

AgiBot World Colosseo: A Large-scale Manipulation Platform for Scalable and Intelligent Embodied Systems

  • Released by AgiBot in March 2025. Strengths include the hardware platform and rich, high-quality data with over one million trajectories. Since the company mainly focuses on data, I will skip the GO-1 model.
  • Failure recovery during collection: Teleoperators manually mark mistakes for later model learning, about 1% of the data.

agibot

Action100M: A Large-scale Video Action Dataset

  • Released by Meta on January 15, 2026: https://github.com/facebookresearch/Action100M.

  • An impressive automated pipeline over 1.2 million YouTube videos.

  • Action100M is generated by a fully automated pipeline that

    • Segmentation: Hierarchical temporal segmentation using V-JEPA 2 embeddings.
    • Captioning: Multilevel frame and segment captions organized into a Tree-of-Captions.
    • Aggregation: GPT-OSS-120B combines evidence through multiround Self-Refine to produce structured action, actor, and caption annotations.

  • Data Pipeline

    1. Segmentation: window_size=64 and step=8 create overlapping frames with different representations. These are accumulated, though I am unsure how, into a final embedding. Visual tokens come from ViT-based V-JEPA 2. Hierarchical segmentation produces both long and short segments. Example: https://github.com/facebookresearch/Action100M/blob/main/data/hySSAAw4t24.json.
    2. Captioning: Llama captions the middle frame, while Meta’s Perception-LM-3B captions a 32-frame segment video, generating captions, questions, and descriptions.
    3. Aggregation: GPT-OSS-120B extracts five fields: brief action, detailed action, actor, brief video caption, and detailed video caption.

VLMEvalKit: An Open-Source Toolkit for Evaluating Large Multi-Modality Models

  • Released in August 2025. It does not seem fundamentally different from other evaluation frameworks.
  • Followed the command through the code: uv run python run.py --model Qwen3-VL-2B-Instruct --data COCO_VAL.

QUANTIPHY: A Quantitative Benchmark Evaluating Physical Reasoning Abilities of Vision-Language Models

  • A VLM benchmark from Fei-Fei Li’s lab, released December 22, 2025. Almost all models fall below human understanding.
    • An interesting final observation: when estimating kinematic quantities, existing VLMs hallucinate by relying heavily on pre-trained world knowledge while hardly inferring from the actual reference video and text..
  • Unlike common VQA description or multiple-choice benchmarks, this tests whether models understand video and apply physics to concrete problems. Description may reflect memorization without understanding or application.
  • Dataset categories:
    • 2D/3D: Mainly distinguished by depth.
    • Static/dynamic: For example, fixed length versus velocity at time t.
    • Supplies prior knowledge, mathematical questions, and reference answers.
  • Sources:
    • Blender simulation: A simulation tool used in research.
    • Laboratory captures.
    • Internet data.
  • Key Findings
    • VLMs rely more on learned prior knowledge than visual inputs for physical reasoning.
      • Removing video while retaining the question and prior still lets the model answer.
    • VLMs (mostly) do not reason but memorize
      • Multiplying numerical priors by a factor changes the scale, yet outputs remain close to familiar real-world values.

quantiphy.png

ALOHA(A Low-cost Open-source Hardware Arm)

  • A bimanual manipulation dataset addressing scarce real-world two-arm robot data.
  • Diverse tasks, including folding cloth and opening containers.

DROID(Distributed Robot Interaction Dataset)

  • A substantial Stanford dataset with 76,000 episodes.
  • Includes a visualization.

Calvin(Composing Actions from Language and Vision)

  • Proposed in 2022, completing long-horizon tasks through language.

We present CALVIN (Composing Actions from Language and Vision), an open-source simulated benchmark to learn long-horizon language-conditioned tasks. Our aim is to make it possible to develop agents that can solve many robotic manipulation tasks over a long horizon, from onboard sensors, and specified only via human language. CALVIN tasks are more complex in terms of sequence length, action space, and language than existing vision-and-language task datasets and supports flexible specification of sensor suites.·

LIBERO

  • Proposed in 2023, with broad scenarios and potentially unlimited manipulation-task generation.

Label

FoundationMotion: Auto-Labeling and Reasoning about Spatial Movement in Videos

  • An annotation framework from MIT and NVIDIA in late 2025, focused on spatial movement.
  • Pipeline:
    1. Video preprocessing: Split into segments and calculate a VGGT motion score. Discard excessive motion, since rapid camera movement impairs tracking.
    2. object detection:
      • Open vocabulary: Qwen2.5-VL-7B recognizes objects; Grounding DINO localizes them.
      • Human-centered hands: Person detection with Cascade Mask R-CNN, body keypoints with ViTPose+, then hands with Hands23. Quite complex.
    3. Temporal tracking: Run SAM2 every five frames.
    4. Caption Generation: GPT-4o-mini by prompt
    5. same as caption

Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection

  • A widely used annotation framework that partitions images and matches regions with text by similarity.
Grounding DINO accepts an (image, text) pair as inputs.
It outputs 900 (by default) object boxes. Each box has similarity scores across all input words. (as shown in Figures below.)
We defaultly choose the boxes whose highest similarities are higher than a box_threshold.
We extract the words whose similarities are higher than the text_threshold as predicted labels.
If you want to obtain objects of specific phrases, like the dogs in the sentence two dogs with a stick., you can select the boxes with highest text similarities with dogs as final outputs.
Note that each word can be split to more than one tokens with different tokenlizers. The number of words in a sentence may not equal to the number of text tokens.
We suggest separating different category names with . for Grounding DINO.

caption-1 caption-2

Infra

Conditional Memory via Scalable Lookup: A New Axis of Sparsity for Large Language Models

  • 26.1.12 by deepseek
  • An interesting idea, Since standard Transformers (Vaswani et al., 2017) lack a native knowledge lookup primitive, current LLMs are forced to simulate retrieval through computation: Transformers compute answers to some knowledge questions that could instead be served by a huge lookup table.
  • n-gram demo: https://github.com/deepseek-ai/Engram/blob/main/engram_demo_v1.py
  • Insert an Engram into the Transformer, then:
    1. Project tokenized inputs into a smaller vocabulary.
    2. Use multihead hashing to reduce collisions and look up n-grams.
    3. Add context to retrieved embeddings through attention, using inputs as queries over embedding keys and values.
  • Training distributes n-grams through GPU all-to-all, probably with hierarchical caching; serving uses external multilevel storage. If n is modest, memory requirements may be manageable, though the numbers need calculating.

Engram Architecture

Lance: Efficient Random Access in Columnar Storage through Adaptive Structural Encodings

  • Published in April 2025, mainly by LanceDB. Lance format is open source; LanceDB is the commercial product.

  • Addresses Parquet’s limitations with large values: separate metadata/data causes multiple I/Os, and page metadata can grow enough to cause OOM.

  • My test with a Hugging Face image dataset was unimpressive. Compression was worse than Parquet, contrary to the paper but consistent with other users’ experiences. Embedding search is useful, but scalability remains uncertain.

  • Provides an evaluation environment rather than a framework, reducing the high cost of evaluation in real scenarios.