A Closer Look at the Recent Buzz Around RSI

As long-horizon tasks and AI for Science gain traction, recursive self-improvement (RSI) is coming up more often in online discussions. But after digging into the research, I found that progress is slower than we might imagine, and getting RSI to work perfectly is extremely difficult, even within a single domain.

Introduction

People mean different things by RSI and approach it in different ways. Some build loops around multi-agent pipelines using a harness; others incorporate model training so the model itself improves. Broadly, I would group these approaches as follows:

• L0: Automatically update training data: Use a mechanism that produces fresh training data each round to drive successive model improvements.
• L1: Use an external database for memory and feedback: Engineer the harness to enrich the model’s context with each round, leading to better rollouts.
• L2: Improve the agent itself: Define a verifier and modify the agent framework directly each round, gradually adapting the agent to a particular domain.
• L3: Iteratively improve exploration: Exploration is important and expensive in long-horizon tasks; the goal is to make each round of exploration less costly.
• L4: Pose and solve your own problems: In mathematics or science, start from scratch by constructing problems and environments, or adapting environments to problems, then use the resulting suitable samples to train the model.

L0: Automatically Updating Training Data

STaR: Self-Taught Reasoner — Bootstrapping Reasoning With Reasoning

• Published by Stanford in 2022: https://arxiv.org/html/2203.14465v2
• Background: At the time, many QA datasets contained only questions and reference answers, with no reasoning steps. STaR aims to automatically update the training data by adding reasoning generated by the model, creating a fine-tuning loop.

The process works as follows:

1. The model reasons through the question/answer examples. When it reaches the correct answer, its reasoning is added to the training data; when it gets the answer wrong, the reasoning is repaired. This produces question-rationale-answer samples.
2. The loop begins: Fine-tune on the data from step 1, then have the fine-tuned model reason through the question-rationale-answer examples again. Keep the rationales for correct answers and continue refining those for incorrect answers by giving the same model the answer and asking it to work backward to a rationale.
3. Repeat step 2.

My take: The overall idea is to improve model capabilities by repairing the training data, specifically by strengthening its reasoning. But producing the reasoning itself is the key to intelligence. Providing the answer first to try to elicit that capability is one possible approach.

Self-Rewarding Language Models

• Published by Meta in 2024: Self-Rewarding Language Models
• Background: Post-training relied heavily on human feedback in RLHF to train reward models. But once a reward model stops improving, it can become easy to exploit. Meta’s aim was to use an LLM as a judge to continually generate DPO preference pairs and drive the RL loop.
• Approach: Two models share the same weights. In each round:
  1. The model generates a batch of candidate responses.
  2. An LLM judge scores the candidates to form DPO preference pairs.
  3. Run DPO optimization, then return to step 1.

My take: This is still a process of iteratively generating DPO training data. But there is a major flaw: the premise of “generate your own responses → judge them yourself → train yourself on your own preferences” does not necessarily hold up.

L1: Finding Better Solutions Through Memory and Feedback

Self-Refine: Iterative Refinement with Self-Feedback

• Proposed in 2023: https://arxiv.org/html/2303.17651v2. This resembles today’s harness approaches: there is no model training, and feedback from each round becomes input for the next.

My take: The method is straightforward; I do not have much to add.

Reflexion: Language Agents with Verbal Reinforcement Learning

• Also from 2023, proposed by Shunyu Yao: https://arxiv.org/html/2303.11366v4. Compared with Self-Refine, it adds a memory step, storing past experience for retrieval in the next round.

FunSearch: Making new discoveries in mathematical sciences using Large Language Models

• Proposed by DeepMind in late 2023. It builds on an asymmetry in mathematics: verifying a solution is easy, but discovering a new way to solve a problem is difficult.
• Approach: Humans provide the problem framework, an initial program, and an evaluation function. Is that essentially what we now call an RL environment? A database stores the solutions explored by the LLM, which keeps searching for new approaches and adding successful candidates back to the database.

A key question is how to keep the search directions diverse enough to avoid wasting tokens endlessly tuning parameters in a dead end. FunSearch’s answer is to use an LLM to cluster the solutions, then introduce some randomness when selecting a program from a cluster as the starting point for the next iteration.

L2: Making the Agent Itself the Target of Improvement

Darwin Gödel Machine: Open-Ended Evolution of Self-Improving Agents

• Proposed in May 2025: https://arxiv.org/html/2505.22954v1. The motivation is that agent architectures generally remain fixed during iteration, constraining what the LLM can do. This approach makes the coding agent’s own logic part of what can evolve.
• Approach: The key is to establish a clear evaluation setup and iterate on the code against it. Programs that regress in the short term are also retained, since they may offer greater gains over the longer term.

L3: Searching for Better Exploration Strategies

Auto-TTS: LLMs Improving LLMs: Agentic Discovery for Test-Time Scaling

• Another paper published this year: https://arxiv.org/html/2605.08083v1. Test-time scaling means spending more compute on generating a response at inference time to improve answer quality.
• Background: Current test-time scaling relies on manually designed strategies. One way to scale more efficiently is to sample large numbers of long-horizon rollout trajectories and turn each strategy into a sequence of multiple-choice decisions. During reasoning, a probe is performed every so many tokens, and intermediate states are saved for replay when making later decisions.

Here is a practical example:

Suppose you run a math problem-solving service. A fixed policy of generating 64 complete solutions for every problem and then voting can waste compute on easy problems.
A controller discovered by AutoTTS can schedule work using intermediate signals. This is an illustrative flow, not the exact rules of the paper's controller:
Receive a problem
    ↓
Start 4 reasoning chains and advance each a little
    ↓
Probe: ask each chain for a provisional answer based on its progress so far
    ↓
┌─ Answers agree and remain stable in subsequent checks → Stop early
│
└─ Answers diverge substantially → Continue some branches or add new ones
                         ↓
                  Check again until stopping
A probe does not read the reference answer or tell the system which answer is correct. It obtains a provisional answer from the current reasoning prefix. Agreement is only a signal for decision-making: all chains could still agree on a wrong answer.

My take: I am not particularly convinced by this direction. It effectively turns test-time scaling into offline trajectory collection and selection. In real applications, identical problems are rare, and this also limits the model’s ability to explore new paths.

Dream-RSI: Recursive Self-Improvement through Evolving Worlds

• Proposed by DeepMind on September 14, 2026: https://arxiv.org/html/2609.14858v1.
• Background: The LLM field is pursuing long-horizon agent tasks, while post-training data relies heavily on synthesis. This makes exploration for data synthesis a critical question: how do we find exploration opportunities with a better return on investment?
• Approach: Treat each exploration as a node in a discovery tree, recording its corresponding simulation environment, inputs, outputs, and process as completely as possible.
• For example:

Suppose you want to optimize an operator from 10 ms to 5 ms. Here is an illustration:
A fixed coding agent tries three directions in parallel:

A: Change memory access patterns   10 → 8 → 7.8 → 7.8 ms
B: Adjust parallel organization    10 → 9 → 7   → 6   ms
C: Fuse computation steps          Correctness tests keep failing
There are now several scheduling choices:
- A has stalled. How many more revisions should it get?
- B keeps improving. Should it continue?
- Should C get another chance at a fix, or should it stop?
- Should a new direction be started from scratch?
Writing the next version of the kernel is the coding agent's job. Deciding which attempts continue and when to change direction is the exploration strategy's job.
Dream-RSI uses historical exploration records to compare these scheduling rules, then applies the selected rules to subsequent real exploration. Stopping C does not establish that C will never be valuable, so the strategy must still balance short-term gains against opportunities to explore.

L4: Pose Your Own Problems, Then Solve Them?

R-Zero: Self-Evolving Reasoning LLM from Zero Data

• Published by Tencent AI Lab: https://arxiv.org/html/2508.05004v4
• Background and approach: Returning to mathematics, R-Zero uses two models:
  • Challenger: Its main reward is for posing sufficiently difficult problems.
    • Training uses an uncertainty reward and a repetition penalty. Difficulty is assessed by holding the solver fixed and measuring its success rate. The key is to avoid problems that are too hard, but this kind of RSI can also generate invalid problems, especially if the solver starts producing unreliable answers and the loop deteriorates.
  • Solver: Its main reward is for solving problems correctly.

EnvHarness: Awakening Static Worlds for Agent Learning

• Published by Google AI Research: https://arxiv.org/html/2608.19880v1
• Background: Agents can practice in browsers, code repositories, or operating environments, but human-built environments are often fixed: the same tasks, starting points, and feedback. Collecting more trajectories may simply repeat behaviors the agent has already mastered. Generating environments from scratch creates another burden: environment logic can be faulty, and rewards and verifiers must be designed again. EnvHarness takes a different approach: retain the existing environment and verification infrastructure, then add a programmable wrapper around the standard interface so that practice conditions adapt to the agent’s weaknesses.
• Approach: The core idea is to adjust the practice conditions the agent encounters through an environment wrapper, based on its current abilities and weaknesses. There are two modules:
  • EnvHarness: Modifies the environment directly through a wrapper around the original environment.
  • EnvRigger: Observes the agent policy’s performance, analyzes weaknesses, and suggests changes.

The agent performs tasks in the current environment
        ↓
EnvRigger analyzes successful and failed trajectories
        ↓
Diagnose: What can the agent not do? What is too easy? Is it relying on fragile shortcuts?
        ↓
Write EnvHarness wrapper components
        ↓
Have the agent try the new environment to verify that tasks are solvable and offer useful practice
        ↓
The agent learns from practice → Diagnose again and adjust the environment

My take: This is actually a familiar approach in RL. If a model is too weak, tailor the labeling to its capabilities to obtain so-called positive samples. Here, the idea is applied to long-horizon agent tasks.