An Overview of Classic RL Paradigms

I spent some time reading about classic RL approaches and paradigms. Different forms of RL also change how data is collected. In particular, current RL paradigms for RSI and long-horizon tasks place increasing demands on synthetic data, environments, exploration efficiency, and more.

Classic RL Paradigms

DQN: Playing Atari with Deep Reinforcement Learning, Mnih et al., 2013-12-19.

• Background and challenges: Learning action values from pixels requires extensive interaction. Consecutively collected samples are highly correlated, which can make direct training unstable. How experience is organized is fundamental to making deep RL work.
• Approach: Approximate action values with a CNN, combining Q-learning with experience replay.
• Results: On the original seven Atari games, the method outperformed prior approaches on six and human experts on three. These results refer to the 2013 paper.
• Data characteristics: Pixels, actions, rewards, and next states form transitions. Random replay reduces correlations between sampled experiences.
• Pipeline: Game interaction → store transitions → random minibatches → TD updates → new interactions.

PPO: Proximal Policy Optimization Algorithms, Schulman et al., 2017-07-20.

• Background and challenges: Standard policy gradient methods have low data efficiency. Large policy updates can quickly create a mismatch between freshly collected data and the new policy. TRPO’s stable updates are relatively complex to implement.
• Approach: Constrain updates with a clipped surrogate objective, performing multiple epochs of minibatch optimization on the same collected batch.
• Results: Achieved a favorable balance of sample complexity, implementation complexity, and runtime in simulated robotics and Atari.
• Data characteristics: Full or truncated trajectories, behavior-policy log probabilities, rewards, values, and advantages. How much the data can be reused is limited by policy changes.
• Pipeline: Rollouts with the current policy → returns / advantages → multiple epochs of constrained updates → fresh rollouts.

SAC: Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor, Haarnoja et al., 2018-01-04.

• Background and challenges: Data for real-world control is expensive, and on-policy methods require extensive interaction. Deterministic policies or policies with limited exploration can also converge to brittle solutions.
• Approach: Combine off-policy actor-critic learning with a maximum-entropy objective to reuse past experience while maintaining exploration.
• Results: Outperformed several contemporary on-policy and off-policy methods on continuous-control benchmarks, with relatively stable performance across random seeds.
• Data characteristics: Transitions from different past policies share a replay buffer. Past experience can be used repeatedly for training.
• Pipeline: Execute the policy → replay buffer → update Q and the actor → feed back new experience.

Conservative Q-Learning for Offline Reinforcement Learning, Kumar et al., 2020-06-08.

• Background and challenges: With only a fixed dataset, a policy may choose actions rarely represented in the data. The value network can be overly optimistic about these actions, and no new interactions are available to correct the errors. Standard off-policy methods therefore cannot necessarily be applied directly to offline RL.
• Approach: Add conservative Q regularization to the Bellman error, lowering value estimates that lack support in the data.
• Results: Outperformed existing methods on various offline-control tasks in the original paper. On some complex data distributions, final returns reached 2–5 times those of the baselines.
• Data characteristics: Fixed experience from multiple policies with multimodal distributions. Coverage matters more than sheer data volume.
• Pipeline: Offline transitions → Bellman fitting and conservative regularization → policy improvement → independent evaluation.

Hindsight Experience Replay, Andrychowicz et al., 2017-07-05.

• Background and challenges: With sparse goal rewards, many attempts fail to achieve the specified goal and appear to offer no positive signal. Yet a trajectory may have reached another meaningful goal.
• Approach: Treat a state actually reached as an alternative goal, recompute the rewards, and replay the experience.
• Results: Learned tasks such as pushing, sliding, and pick-and-place using only binary rewards. Some policies trained in simulation could be deployed on real robots.
• Data characteristics: Retain both desired and achieved goals. Relabel the goals without fabricating state transitions.
• Pipeline: Collect trajectories → select achieved goals → recompute rewards → original and relabeled transitions → off-policy updates.

RL Rewards and Update Methods

RLHF: Training language models to follow instructions with human feedback, Ouyang et al., 2022-03-04.

• Background and challenges: The pretraining language-modeling objective does not align with user preferences. Human demonstrations show desired behavior, but candidates generated by the model itself still need to be compared and evaluated.
• Approach: Use human demonstrations for SFT, train a reward model on candidate rankings, and then optimize the policy with PPO.
• Results: Under the paper’s human evaluations and prompt distribution, outputs from 1.3B InstructGPT were preferred to those from 175B GPT-3.
• Data characteristics: SFT demonstrations, preference rankings, RL prompts, and policy rollouts are distinct layers of data.
• Pipeline: Prompts → human demonstrations → SFT → candidate rankings → RM → score rollouts → PPO.

GRPO: DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models, Shao et al., 2024-02-05.

• Background and challenges: PPO training for LLMs typically requires an additional value model, increasing memory and compute costs. Token-level value estimation is also difficult when only an outcome reward for the entire response is available. Multiple responses to the same question offer a more direct relative comparison.
• Approach: Sample multiple responses to each question and estimate relative advantages using the mean and standard deviation of rewards within the group. This removes the separate value model while retaining clipped policy updates and a KL constraint.
• Results: After applying the paper’s RL recipe to 7B Instruct, GSM8K increased from 82.9% to 88.2% and MATH from 46.8% to 51.7% (CoT, without tools). This is a before-and-after comparison of the full RL recipe.
• Data characteristics: About 144K questions, with 64 outputs sampled per question in the original experiments. Groups of responses to the same question, rewards, and behavior-policy probabilities jointly contribute to the updates.
• Pipeline: Questions → grouped rollouts for each question → reward-model scoring → group-relative advantages → policy updates → resampling. Sections 4.1–4.2, Table 5

GSPO: Group Sequence Policy Optimization, Zheng et al., 2025-07-24.

• Background and challenges: Long sequences, multiple epochs of minibatch updates, and changes in MoE routing amplify differences between past rollouts and the current policy. The paper observed training instability caused by fluctuations in token-level importance ratios, while rewards are usually assigned to entire responses.
• Approach: Construct weights from length-normalized sequence likelihood ratios, apply clipping at the whole-response level, and use group-relative advantages.
• Results: At equal training compute and question consumption, the Qwen3-30B-A3B cold-start model outperformed the paper’s GRPO baseline on math and coding evaluations. Its MoE experiments did not require Routing Replay.
• Data characteristics: Responses are still grouped by question. Retain token log probabilities from the old and new policies and aggregate them into a sequence weight for each response.
• Pipeline: Grouped rollouts → verifier scoring → within-group advantages → length-normalized sequence ratios → sequence-level clipping and updates. Sections 4–5, Figure 1

Building RL Data

DAPO: An Open-Source LLM Reinforcement Learning System at Scale, Yu et al., 2025-03-18.

• Background and challenges: When all responses to a question are correct or all are wrong, within-group advantages are zero. As training progresses, the number of useful prompts declines. Loss aggregation for long responses and truncation penalties can also amplify reward noise. Simply increasing the number of rollouts does not guarantee effective learning.
• Approach: Dynamically filter out zero-advantage groups and sample additional groups, together with Clip-Higher, token-level loss aggregation, and overlong reward shaping.
• Results: On AIME 2024, Qwen2.5-32B’s avg@32 increased from 30% with naive GRPO to 50% with the full recipe. This is not the isolated gain from any single technique.
• Data characteristics: DAPO-Math-17K converts answers to integers for grading. With 16 outputs per question, retain groups containing both correct and incorrect responses, and record lengths and truncation status.
• Pipeline: Questions and integer answers → grouped rollouts → rule-based grading → filter all-correct / all-incorrect groups and sample replacements → length handling → token-level aggregated updates. Sections 3–4, Table 1

DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning, DeepSeek-AI, 2025-01-22.

• Background and challenges: High-quality reasoning demonstrations are difficult to collect. R1-Zero applies RL directly to a base model, but its outputs suffer from poor readability and language mixing. R1 needs to balance reasoning capabilities with general response quality.
• Approach: Introduce cold-start SFT, then filter responses generated by reasoning RL for correctness and readability to build data for subsequent SFT, followed by a final RL stage.
• Results: R1 achieved 79.8% pass@1 on AIME 2024. This is a result of the complete training pipeline.
• Data characteristics: Several thousand cold-start CoT examples, about 600K filtered reasoning samples, and 200K general-purpose samples. Rule-based or learned rewards are used at different stages.
• Pipeline: Cold-start SFT → reasoning RL → generation and rejection sampling → SFT on mixed data → final RL. Sections 2.3–3

DreamerV3: Mastering Diverse Domains through World Models, Hafner et al., first released on 2023-01-10.

• Background and challenges: Real-world interaction is expensive, and observations, reward scales, and dynamics vary widely across tasks. A training approach is needed that both reuses experience and reduces task-specific tuning.
• Approach: Learn a latent world model, train the actor and value function on imagined trajectories, and use a unified set of stabilization techniques.
• Results: A single configuration covered more than 150 tasks. In Minecraft, it collected diamonds from scratch without human data or a curriculum.
• Data characteristics: Real temporal data trains the dynamics model; imagined data trains behavior. The two serve different purposes.
• Pipeline: Environment sequences → replay → world model → latent rollouts → actor / value → real interactions.

π*0.6: a VLA That Learns From Experience, Amin et al., 2025-11-18.

• Background and challenges: Demonstration learning struggles to cover errors that actually occur during deployment or to optimize speed. Real-world rollouts contain successes, failures, and human takeovers. Treating them all as equally good demonstrations confuses the learning objective.
• Approach: Estimate advantages with a value function, train the VLA with advantage conditioning, and iteratively incorporate autonomous experience and corrections.
• Results: On some difficult tasks, throughput more than doubled and failure rates were roughly halved. These improvements are limited to the tasks studied in the paper.
• Data characteristics: Demonstrations, autonomous trajectories from multiple policy versions, and expert interventions coexist. Suboptimal actions are retained with advantage labels.
• Pipeline: Offline VLA / value pretraining → real-robot rollouts and takeovers → outcome labeling → value updates → advantage-conditioned policy training → redeployment. Sections IV–VI