Continual Learning: The Next Direction for Intelligence?

I recently encountered this idea repeatedly, including in a leaked PDF about Liang Wenfeng and DeepSeek’s fundraising. It argued that continual learning should have much higher priority than new modalities, and that this capability is a key condition for practical deployment.

My Experience

I Do Not Have an AI That Is Truly Mine

Outside coding, my biggest frustration is that AI cannot continuously acquire and internalize information. Markdown files alone do not provide deep connections across memories or make the model understand me well enough.

For example, asking ChatGPT about Li Auto’s July sales creates two problems:

1. By default, it may predict 2025 figures because the requested data is absent from pretraining.
2. Even if tools retrieve current figures, it does not know I have already asked about June, May, and every month back to January because I follow this company continuously.

I want it to set aside general knowledge I do not need, such as CAD drawing, which I may never use. Instead, like a colleague or friend with a similar background, it should solve my actual problems, grow through them, and share my context.

I Tried Distilling Myself, with Limited Results

To help Codex understand me, I created a GitHub project and scheduled it to summarize all my sessions each day into a daily Markdown file.

Even with this harness, the result is essentially RAG: retrieve similar material and tell me when I previously encountered it. That offers limited help.

What Counts as Continual Learning?

Here I mean internalizing knowledge from everyday use into model intelligence, rather than harness-based self-evolution such as AK’s autoresearch project, although the approaches may eventually converge.

Attention connects information throughout an input context and uncovers implicit relationships. A short, typo-filled prompt can therefore produce unexpectedly rich results. By continual learning, I mean training on every input and response from daily interactions in real time, perhaps as new hidden states or a small model’s weights. Two aspects matter:

1. Real time.
2. Incorporated into a model.

A Parallel with Recommendation Systems

This reminds me of my recommendation-system experience. Everything a user views and does, such as clicking or following, passes through a real-time pipeline into positive and negative training samples. Frequent parameter-server updates refresh preferences within minutes, making the product seem to understand the user.

What Would This Enable?

Imagine a general base model continuously evolving through your work and life, absorbing the same knowledge and skills. Would that not become your own digital counterpart?

1. Real Productivity Gains

Much software work now involves a person consulting AI, passing the conclusion to another person, who then consults their AI. This is cumbersome. Ideally, I could ask your AI directly, or my AI could contact yours. Why keep people relaying messages in the middle?

I also think the lack of continual learning limits multi-agent collaboration. If several general-purpose agents have roughly identical abilities, what does cooperation add? Communication overhead may outweigh the benefit, just as with people.

2. Even Greater Potential for Businesses

I worked at an early Chinese SaaS startup in 2017. I later found that standardizing complex systems as software-as-a-service often failed for small domestic customers:

1. Intense competition forces even small businesses to differentiate. Their customization demands range from changing an icon to rebuilding the system around their business model.
2. Small customers cannot pay enough to justify customization, and do not need 90% of a complex system’s features. Those systems often encode successful large-company practices, while small customers mainly need survival and growth.

With continual learning, a small model could internalize a company’s actual workflows. Combined with an agent framework such as DeepSeek-Harness, this might give each business software tailored closely to its needs.

Current Technology and Startups

TTT（Test-Time Training）

Starting with research, see https://keyurramoliya.com/posts/Test-Time-Training/. Test-time training means a model can learn during testing or inference, rather than limiting weight updates to the usual training phase.

It combines ideas from RNNs and attention:

• RNNs compress history into a fixed-dimensional recurrent unit. Information can be forgotten, leading to mechanisms such as LSTM gates, though those gates depend on human architectural choices.
• Attention performs O(L^2) computation over all tokens, avoiding that compression loss but becoming expensive for very long contexts.

TTT combines these properties in linear and MLP variants:

• TTT-Linear: A small linear model over hidden states is updated recurrently from new states, without backward propagation in this description. Its weights compress the history of inputs, or hidden states.
• TTT-MLP: Adds nonlinear layers to capture deeper interactions among hidden states and updates weights through backward propagation.

TML（ThinkingMachineLab）

Details: https://thinkingmachines.ai/blog/lora/#what-matters-for-lora

TML’s article on LoRA argues that a small model, or a base model with LoRA, can match full-parameter fine-tuning. Perhaps LoRA can also learn what large-model pretraining learns.

Other key findings:

• Applying LoRA to all layers works better than applying it only to attention layers.
• SFT learning efficiency is O(N), while RL’s is O(1): each SFT token provides a correct target for backward propagation, whereas an RL episode supplies one outcome.
• LoRA and full fine-tuning require different learning rates.

Its released products still appear aimed at businesses. (https://thinkingmachines.ai/blog/on-policy-distillation/?utm_source=chatgpt.com)

MindLab

MindLab could be described as a Chinese counterpart to TML’s technology, with several supporting papers, though its practical deployment remains unclear to me. Two ideas stand out:

• Very long memory and adapters: Small models or matrices absorb growing histories, while adapters update the model in real time.
• Millions of LoRAs: A mixture-of-LoRA architecture compresses information for individuals or groups into many LoRA matrices.

Notes on several key papers:

1. δ-mem: A tiny online matrix absorbs an ever-growing history.
   • Addresses absent model memory and hallucinations introduced by context harnesses.
   • First train a memory matrix with SFT for memory reads and writes. Add a mutable matrix on top, updating only this addition during inference to retain user-specific associations, with an update after every token.
2. PEFT（Parameter-efficient fine-tuning）
   • This paper presents MindLab’s overall architecture: the base model supplies prior knowledge, while adapters personalize preferences, habits, and memory.
   • Three concepts:
     • Scale Up: Trillion-parameter MoE models can receive repeated adapter updates; stronger base models give LoRA more potential to unlock.
     • Scale Down: Small models suffice for personalized memory; higher ranks add storage and training cost.
     • Scale Out: Once Scale Down is validated, add more adapters to serve different groups’ needs.
   • A sensible business pipeline:
     • Put the current conversation in explicit context. Move reusable information into online memory. Distill only confirmed, desensitized, repeatedly validated preferences into long-term adapters. Keep exact facts in external storage that supports retrieval, deletion, and auditing.
3. Mint (MindLab Toolkit): MindLab’s inference and training framework, explaining the PEFT training architecture above.
4. LongStraw：Long-Context RL Beyond 2M Tokens under a Fixed GPU Budget
   • MindLab’s RL system mainly optimizes GRPO. Standard GRPO calculates advantages and backpropagates through earlier token states, including the prompt and preceding response tokens, to update gradients and parameters.
   • Here, prompt hidden states are frozen, eliminating their backward pass and saving substantial computation. Total work can be viewed as prompt forward/backward plus response forward/backward.
     • Responses may diverge from their first token, preventing reuse of their hidden states.
   • LongStraw demonstrates that response-side RL updates on very long contexts are feasible within a fixed GPU budget. It does not establish equivalence to full-gradient training or prove task-level benefits from long-context RL.

RiverAI

Founded by an xAI cofounder, River reportedly raised $1.1 billion a few days ago. Its statement at https://river.ai/ describes personal AI owned and shaped by each individual, clearly pointing toward personalized consumer models.