LLM Data Processing from Kimi K1.5 to K3

Mainly based on Kimi’s technical reports. K1.5 provides more detail; K2 and K3 largely extend its data work. This took three days to organize, including reviewing the open-source projects cited by Kimi.

• Kimi K1.5: Scaling Reinforcement Learning with LLMs
• Kimi K2: Open Agentic Intelligence
• Kimi K3: Open Frontier Intelligence

Pretraining Data

K1.5

English and Chinese Textual Data:

1. Rule-based filtering: Remove unsuitable data with explicit rules.
2. FastText-based classification: Train FastText classifiers for categories and quality. CPU inference provides lightweight supervised filtering between rules and LLMs, outputting label probabilities. Quality labels reuse its classification capability. DataComp-LM describes a similar methodology.
3. Embedding-based similarity: Use document embeddings to remove semantically similar documents, following BGE-M3, https://huggingface.co/BAAI/bge-m3）
   • BGE-M3 is BAAI’s open text-embedding model. M3 means multilinguality, supporting over 100 languages and cross-language retrieval; multifunctionality, supporting dense, sparse, and multi-vector retrieval; and multigranularity, supporting sentences, paragraphs, and documents up to 8,192 tokens. Compared with MinHashLSH over n-grams and bands, it also captures semantic duplication.
4. LLM-based quality assessment: Score coherence, informativeness, and potential educational value, as in FineWeb.
   • FineWeb distills Llama-3-70B-Instruct’s 0-5 scores into a smaller classifier, covering coherence, information, and educational value. An open model is HuggingFaceFW/fineweb-edu-classifier.

Code Data

1. Pure code: Follow BigCode’s rule-based cleaning and junk removal, adding programming-language resampling for balance.
   1. Identify languages with go-enry, https://github.com/go-enry/go-enry, focusing on 32 languages.
   2. Apply rules such as filtering extremely long lines.
   3. Remove near-duplicate code with MinHashLSH, 5-grams, and a Jaccard threshold of 0.7.
   4. Address safety, privacy, and benchmark contamination.
2. Interleaved text and code: Technical documentation and tutorials are common examples. Embedding retrieval selects high-quality, diverse samples, implying an ongoing vector database. It also lets algorithm teams retrieve supplemental training data.

Mathematics and Reasoning Data, Following OpenWebMath

OpenWebMath’s process (https://huggingface.co/datasets/open-web-math/open-web-math):

1. High-recall coarse filtering: Crawl 200 billion HTML pages and retain likely mathematics using simple symbol checks.
2. Parse HTML while preserving formulas.
3. Filter: FastText identifies English; MathScore, trained from LaTeX positives and random nonmath negatives, detects mathematical content (open FastText model at https://huggingface.co/kenhktsui/math-fasttext-classifier); a roughly 1.53 GB KenLM trained on ProofPile assesses text quality.
4. Use text-dedup’s SimHash implementation with a 0.7 threshold to remove near-duplicate pages.

Kimi’s main approach:

1. Remove irrelevant content with FastText.
2. Clean the remainder with a fine-tuned model, retaining high-quality data.

Knowledge Data

1. Specialized internal academic OCR, optimized for formulas and special symbols.
2. Multidimensional annotation by an internal LM, including OCR quality, educational value, and document type, similar to FineWeb.
3. OCR threshold filtering to remove common errors.
4. Rank by educational value.
5. Run isolated experiments by document type, upsampling valuable subsets while retaining others for generalization.

Caption Data

• Open Chinese/English image-text datasets such as LAION-5B, which uses CLIP similarity to filter roughly 50 billion candidates into six billion pairs, and DataComp, whose filters include FAISS clustering into 100,000 groups to find ImageNet-like distributions. Limit synthetic captions to reduce hallucinations, deduplicate by hashes/embeddings, ensure image-text relevance, and vary resolution.

Image-Text interleaving data

• MMC4 matches images and sentences with CLIP rather than trusting DOM structure; OBELICS preserves DOM order while cleaning text-image sequences. Add textbooks, webpages, tutorials, and synthetic data. Beyond filtering and deduplication, reorder image-text content to preserve knowledge flow, using LLM rewriting, and support multiple images and long contexts.
• Interleaved image-text data partly restores language ability harmed by simplistic caption data, such as a cat label paired with an image.

OCR Data

• Open and internal OCR data covers multilingual text, dense layouts, webpages, handwriting, charts, geometry, Mermaid, and scene text. Apply rotation, deformation, color, and noise augmentation. Following OCR 2.0, extend recognition to broader visual-text understanding, including formulas, music notation, and graphics.

Multi-Modal Knowledge Data

• Textbooks, papers, geometry, and web image-text: Balance taxonomy, use layout parsing and OCR, and separately extract figure text to avoid OCR shortcuts where answers are simply read from the image.
• How can image-text ordering be preserved?
  1. DOM structure.
  2. Image-text embedding matching.
  3. Captions and references.
  4. Rule checks after ordering; strictly prohibit shuffling within sequences.

General QA Data

• Grounding, table/chart QA, web agents, and general QA use open and internal datasets, scoring models for difficulty, and manually designed fine-grained taxonomies to balance difficulty and diversity.
• Kimi-VL also discusses these QA datasets:
  • Cooldown: Filter and rewrite academic visual/vision-language material into QA, keeping its proportion low to avoid overfitting a fixed question-answer format.
  • Instruction tuning: Build human-authored seed data, train a seed model, generate multiple answers for varied prompts, then have humans rank and polish the best.
  • Verifiable reasoning: Expand SFT data through rejection sampling with rule- or model-based verifiers.

K2

K2 emphasizes improving token utility through rephrasing. After absorbing large-scale data, the next priority is exploiting high-quality material, especially using it as seeds for further synthesis.

Knowledge Data Rephrasing

• Style- and perspective-diverse prompting: Rewrite with varied prompts, inspired by WRAP.
• Chunk-wise autoregressive generation: Rewrite long documents in chunks and concatenate them.
• Fidelity verification: Compare original and rewritten content. Possible methods:
  • LLM-as-a-judge, perhaps Kimi’s approach.
  • Existing models such as MiniCheck, AlignScore, and DeBERTa-v3-large-mnli. Typically extract claims and test consistency or missing information.
  • Embedding similarity, such as BGE-M3.

Kimi’s conclusions about repeated training on high-quality data:

1. One epoch is insufficient.
2. Synthesizing multiple rephrasings brings a modest improvement.

Math Data Rephrasing

The SwallowMath project provides a reference:

1. FineMath-4 plus mathematical webpages.
2. Rewrite with Llama-3.3-70B-Instruct.
3. Filter, complete context, and organize derivations. Temperature=0.2 keeps rewriting stable and faithful, reducing invention.
4. Save as mathematical text.

Kimi’s report additionally describes:

1. Rewrite high-quality mathematical documents as learning notes, making implicit derivations, definitions, and learning structure explicit.
2. Translate valuable mathematics from other languages into English to expand knowledge coverage.

K3

K3 discusses data more briefly, highlighting:

• Further long-context synthesis: Arrange and combine multimodal documents and subtasks into problems requiring the full context, preventing short-window shortcuts.
• Substantially more multimodal code data, including SVG, 3D assets, games, and CAD. Sources likely include crawls and open datasets.
  • SVG: https://starvector.github.io/starvector/
  • 3D Asset: https://huggingface.co/datasets/allenai/objaverse-xl

Note: I found that much hard-to-obtain data is readily available on Taobao.

SFT/RL Data

K1.5 chain-of-thought generation, similar to DeepSeek-R1: Building SFT data to initialize RL reasoning follows a similar process:

1. Select a diverse subset of RL prompts.
2. Carefully prompt the model to produce plan-evaluate-reflect-explore reasoning.
3. Filter and verify through rejection sampling.
4. Train the model with SFT.

1
Solve the following problem carefully.
2

3
Your solution should demonstrate:
4

5
- Planning: identify a promising strategy before carrying it out.
6
- Exploration: consider alternative approaches when appropriate.
7
- Evaluation: check important intermediate claims and calculations.
8
- Reflection: if an approach produces a contradiction or dead end,
9
  identify the mistake and revise the approach.
10
- Verification: independently verify the final result.
11

12
Do not assume the answer in advance.
13
Conclude with a clearly formatted final answer.
14

15
Problem:
16
{question}

K3 RL Initialization

• Earlier domain-specialized Kimi models synthesize trajectories, followed by multistage verification and human-in-the-loop annotation. All use an XTML chat template. Coverage expands from ordinary instructions to long-horizon agents: adaptive reasoning, precise tool calls, sustained execution, and error recovery.

Other Data Types

Vision data:

• Real-world data: Image understanding, location understanding, perception, and simple multimodal reasoning.
• Synthetic data: Targeted reasoning for spatial understanding, geometry, and object interaction.
• Text-rendered data: Convert documents into images to improve understanding of text in images.

Code-Verifier Data

For web-sourced programming problems lacking tests, K1.5 uses CYaRon:

1. The base model generates fifty test cases from the problem and CYaRon instructions.
2. Randomly run ten ground-truth submissions on each case.
3. Accept a case only if at least seven outputs agree.
4. Include the problem in RL training only if at least nine of ten submissions pass every selected case.

Agentic Data: A Separate, Important Topic

Building RL Environments

K3’s Unified White-Box RL Environment deserves attention.

1
4.2.1 Unified White-Box RL Environment
2
Training with a single fixed agent harness can cause a model to overfit to a particular tool schema, system prompt, context
3
management mechanism, or interaction protocol. To address this, we develop a unified white-box RL environment
4
that represents an agent harness as a collection of configurable, composable modules, including tool interfaces, system
5
prompts, context management strategies, skills, memories, subagents, and other components. Composing these modules
6
through configuration, the environment can instantiate mainstream harnesses such as Kimi Code [57], Claude Code [15],
7
Codex [20], OpenClaw [87], and Hermes [45], as well as entirely new ones. During RL training, we dynamically construct different harness configurations for different task groups, exposing Kimi K3 to diverse combinations of these
8
modules rather than the conventions of any single harness. The same abstraction also readily supports RL across various
9
task domains, providing a scalable foundation for training more general-purpose agents.

• Underlying environment: Move from Kubernetes to Rust-based AgentENV. Agents may need to run Docker themselves, making AgentENV more flexible.
• Configuration: Agents exploring environments through the knowledge graph below may also generate environment configurations for RL setup.

AgentENV

Since most companies still use Docker for RL environments, I examined why Kimi built a VM-based alternative. Docker’s issues include:

• Startup overhead: Seconds-long cold starts accumulate when every RL task needs a fresh environment. AgentENV reduces startup to milliseconds.
• Idle waste: Agentic rollout spends much time waiting for models while containers occupy memory. AgentENV can pause environments and heavily overcommit resources.
• Fork semantics: Branching multiple states from an intermediate trajectory, as in best-of-N, is not directly supported by Docker. AgentENV provides snapshot and fork.

They can serve similar purposes but use different mechanisms:

• Docker: dockerd/containerd manages images and lifecycle, using Linux namespaces and cgroups while sharing the host kernel.
• AgentENV: Rust orchestration -> Firecracker microVMs on KVM -> each sandbox can have its own kernel, effectively a complete OS.

For an RL-environment abstraction, see Prime Intellect’s EnvironmentHub and its wiki-search example (https://app.primeintellect.ai/dashboard/environments/primeintellect/wiki-search):

• tool
• verifier
• Dataset: Many environments include benchmarks, but upstream-generated data may need only the environment’s tools and verifier.

Data Classification

Classification directly shapes RL’s training direction.

K1.5

A simple tagging system classifies by domain and rules, then samples across categories. References include From Quantity to Quality: Boosting LLM Performance with Self-Guided Data Selection for Instruction Tuning and What Makes Good Data for Alignment? A Comprehensive Study of Automatic Data Selection in Instruction Tuning.

K2: Greater Focus on Agent Trajectories

1. Tool repository: Collect over 3,000 MCP tools from GitHub, then recursively expand domain categories in a knowledge tree to synthesize over 20,000 tools, citing WizardLM’s Evol-Instruct.
   • Evol-Instruct evolves prompts. Starting from about 52,000 Alpaca instructions, WizardLM runs four rounds, selecting equally among five depth operations and one breadth operation. This produces about 250,000 instructions, of which 70,000 train WizardLM. The approach can also build a tool knowledge graph.
2. Agent diversification: Combine system prompts and tool sets into thousands of roles and capability profiles.
3. Rubric-based tasks: Generate tasks from simple to complex with success conditions, expected tool patterns, and evaluation checkpoints.
4. Multiturn user simulation: An LLM generates personas, communication styles, and preferences for natural conversations.
5. Stateful tool simulation: Maintain world state, update it after calls, and inject controlled successes, partial failures, and edge cases.
6. Trajectory filtering: Agents execute tasks, an LLM judge evaluates rubrics, and only successful trajectories survive, effectively large-scale rejection sampling.
7. Real execution: For code/SWE, use real repositories, development sandboxes, and unit-test pass rates to compensate for simulator fidelity limits.

Related points:

1. t-SNE checks tool diversity and functional coverage. Tool descriptions may be serialized and projected into two dimensions, perhaps with UMAP, to inspect category structure and excessive duplication.
2. ACEBench may have inspired Kimi’s classification tree, with eight domains and 68 subdomains.

K3: A Knowledge Graph for Broader and Deeper Agent Tasks

1. Build a hierarchical DAG from predefined coarse seed nodes.
   • How many? The report’s fine-grained concepts suggest thousands or even tens of thousands.
2. Assign agents to nodes and repeatedly search the web to explore concepts.
3. Before adding nodes, search the graph for equivalent or related concepts and reuse them to reduce duplication.
   • How are duplicates such as RoPE and Rotary Position Embedding recognized?
4. Edges always run from broad to narrow concepts. Stop expanding when the agent judges a concept sufficiently atomic.
   • How is sufficient atomicity determined?
5. Sample levels or combinations of related nodes according to target domain and task distributions.
6. Combine node keywords and ancestor context into web queries for real public material.
7. Synthesis agents generate four task types from that material:
   • Knowledge / Coding / Debugging / Vision / Agentic search

Engineering Questions

1. How Do We Deduplicate at Scale?

Cost and performance generally suggest coarse and fine stages:

1. Coarse filtering: MinHashLSH, using Spark and object storage.
   • Extract n-grams, hash them with multiple functions, commonly 128, and retain each minimum. Split the 128-dimensional signature into bands, such as sixteen, and hash each band. One matching band makes two documents possible duplicates.
2. Fine filtering: Embedding retrieval with k-means, embedding clusters, and Spark.
   • Sample data to train centroids, assign all data to clusters, then classify and compare embeddings within each. I need to look further into multistage k-means.
   • With a persistent service, write embeddings into an indexed cluster and call it for deduplication.

The performance-accuracy tradeoffs are complex enough to deserve a separate article.

2. How Is Prompt Difficulty Controlled?

Use a smaller SFT model for ten rollouts at a relatively high temperature, then estimate difficulty from pass rate.

3. How Can Reward Hacking Be Avoided?

1. Exclude easily guessed formats such as multiple choice and true/false.
2. Ask the model to guess without chain of thought; eight attempts can filter easily hacked prompts. Note: Better methods are still needed to judge whether reasoning itself is sound.

On reward hacking:

• K2 combines verification methods: code execution for software, LLM-as-a-judge for semantics, and a K2 critic when no reference answer exists.
  • The K2 critic compares scores without ground truth, likely trained using open preference datasets and human feedback. Its evaluation dimensions are worth examining:

• K3 focuses on agents, so more verification happens inside sandboxes.