Tracing Large Models in Recommendation Systems

Liao Jiayi Liao Jiayi #AI#Recommendation systems#LLM

New wine bottles for familiar ideas: how can LLM techniques systematically reshape recommendation systems?

Translated from Chinese with AI · Read the original

Since ChatGPT appeared, Facebook, Alibaba, Kuaishou, and ByteDance have explored combining recommendation systems with the technology behind LLMs, making rule-based architectures easier to scale.

Introduction

Background:

1. Recommendation Systems Enter the Large-Model Era

Meta 2024.02: Actions Speak Louder than Words: Trillion-Parameter Sequential Transducers for Generative Recommendations

  • At the data level, handcrafted features are largely abandoned, leaving time sequences of (item, ts, action), with causal attention and cross-attention against target candidates.
  • At the model level, HSTU introduces two main changes from the LLM Transformer (are they well justified?):
    • HSTU adds a gate U after attention to further cross the aggregated result. (The original wording is This design is motivated by the difficulty of approximating dot products with learned MLPs..)
    • The nonlinearity changes from Softmax to SiLU, preserving the strength of the user’s interest representation, similar to DIN pooling.

image

2. Lightweight Recommendation Upgrades with LLMs

Bytedance 2024.09: HLLM: Enhancing Sequential Recommendations via Hierarchical Large Language Models for Item and User Modeling

  • ByteDance’s early approach to adapting recommendation systems with LLMs can be understood from the item and user sides:
    • Item: Describe each item’s attributes with an LLM, then feed the descriptions into an LLM and take its final-layer item embedding as their representation.
    • User: Describe the user’s entire behavior history as text and feed it into the LLM as well.
    • There are two ways to combine users and items:
      • Early Fusion: Concatenate the item after the user representation at the input, then use cross-attention to calculate similarity between each item and historical user behavior.
      • Late Fusion: Compress user history into an interest embedding, as on the item side, then use a prediction head to predict its similarity to the item.

image

3. Combining GR and DLRM

Alibaba 2025.02: Unlocking Scaling Law in Industrial Recommendation Systems with a Three-step Paradigm based Large User Model（LUM）

  • Alibaba proposed this combination in 2025 as GR and DLRM developed separately. The motivations were:
    • GR scales well, but its generation objective, the next item, does not necessarily align with the actual CTR prediction objective.
    • The extensive sparse features accumulated in DLRM are difficult for GR to replace.
    • Repeatedly processing the long histories required by GR is expensive for training and inference. To accommodate existing systems, it proposes three stages:
  1. Autoregressively train on user sequences and current search queries to obtain an interest embedding conditioned on the query.
  2. When a user request arrives, combine the interest representation with the current context, such as the search query, to obtain current interests.
  3. Feed this information into DLRM for joint prediction and feature interactions.

image

5. Attempts to Remove IDs from Recommendation

OneRec Technical Report

  • Video representation:
    • Tokenize into Semantic IDs, first obtaining embeddings from multimodal information and contrastive learning.
    • Apply RQ-k-means to embeddings to generate three semantic IDs.
    • Store the semantic-ID-to-video mapping.
  • Pretraining:
    • Inputs are semantic IDs and other interaction features.
    • Outputs are the target video’s three semantic IDs.
  • Reinforcement learning:
    • ECPO: Use user feedback as labels to increase positive-sample probabilities and suppress negative-sample probabilities.
    • ECPO vs. GRPO: Both compare rewards within groups, but GRPO clips only within-group deviations, leaving the KL-divergence contribution unclipped. Negative samples were found to produce large gradient fluctuations that hurt training.

image

OneRec-V2 Technical Report

  • V2’s biggest change replaces OneRec’s encoder-decoder architecture with a decoder-only model. This unifies the architecture, enables shared K/V reuse, and improves scalability.

6. Applying LLM Ideas to Retrieval

Alibaba 2025.07: RecGPT Technical Report

  • Alibaba’s three-tower retrieval system, proposed in July 2025, mainly uses LLMs for their representations, then converts to embeddings and dot products for retrieval.

    • User interests: Generate an interest description from user attributes and historical behavior with an LLM.
    • Product tags: Predict user needs from those interests, corresponding to product tags.
    • Three-tower retrieval: User interests x product tags x products.

  • How is the LLM trained? Fine-tune Qwen-14B using data synthesized by DeepSeek-R1 and real online user interactions.

  • Finally, jointly train the three towers through a DNN and token-embedding weights.

image

7. LLM-Style Latent Transformers for Temporal Events

Meta 2026.01: LLaTTE: Scaling Laws for Multi-Stage Sequence Modeling in Large-Scale Ads Recommendation

  • Meta again! Its 2026 paper discusses bringing the benefits of user-sequence models into low-latency ad ranking.

The overall approach uses two model stages:

  • Async: Take the user’s long history and produce an interest representation, or user embedding.
  • Sync: Take recent history, candidates, and context, and predict CTR.

The models are cascaded. The advantage is that long histories can be reused and processed less frequently. The drawback is that the two-stage structure is less fundamental, limiting interactions between recent and historical behavior.

image