Inside a General-Purpose LLM Evaluation Framework

Dec 28, 2025

Translated from Chinese with AI · Read the original

I mainly examined OpenCompass and LightEval. Their overall logic is similar, centered on models and datasets.

LightEval

A typical command:

1
lighteval accelerate \
2
    "model_name=openai-community/gpt2,batch_size=1" \
3
    hellaswag

1. Select the Evaluation Backend

lighteval-backend

2. Using Accelerate as an Example, Build a Complete Evaluation Pipeline

1
pipeline = Pipeline(
2
    tasks=tasks,
3
    pipeline_parameters=pipeline_params,
4
    evaluation_tracker=evaluation_tracker,
5
    model_config=model_config,
6
)

It includes:

Tasks: provide inference requirements such as datasets, models, and evaluation rules;
EvaluationTracker: tracking information, integration with external hubs, result formatting, and so on;

3. Build the Dataset

Tasks specify datasets. In the hellaswag task, the hf_xx attributes identify the dataset and its splits. The Metrics.exact_match setting specifies how predictions are compared with targets: exact matching here.

1
hellaswag = LightevalTaskConfig(
2
    name="hellaswag",
3
    prompt_function=hellaswag_prompt,
4
    hf_repo="Rowan/hellaswag",
5
    hf_subset="default",
6
    hf_avail_splits=["train", "test", "validation"],
7
    evaluation_splits=["validation"],
8
    few_shots_split=None,
9
    few_shots_select=None,
10
    generation_size=1,
11
    metrics=[
12
        Metrics.exact_match,
13
    ],
14
    stop_sequence=["\n"],
15
    version=0,
16
)

In the pipeline, extract the dataset’s validation portion and split it according to sampling_method. sampling_method, how we query/sample information from the model, has three forms:

GENERATIVE: Sample text from the distribution
LOGPROBS: Sample/retrieve probability scores for choices
PERPLEXITY: Sample/retrieve probability scores across sequences

4. Evaluate the Model

Take SamplingMethod.GENERATIVE as an example:

1
case SamplingMethod.GENERATIVE:
2
    model_outputs = self.model.greedy_until(docs)
3
    outputs[sampling_method] = model_outputs

This also has several steps:

Divide the dataset into several splits, fine enough to track inference progress
Construct a Torch DataLoader
Preprocess data and prompts using the model’s tokenizer
Run inference and return responses

5. Compute Metrics

For the earlier hellaswag example, the metric is exact_match. The key method is:

1
@abstractmethod
2
def compute(self, doc: Doc, model_response: ModelResponse, **kwargs):
3
    raise NotImplementedError

doc: contains the original gold/target text;
model_response: the model’s inference output, or prediction;

Compare after normalization. Metrics use different comparison methods, then collect and aggregate results through aggregation_function. Here is the implementation of exact_match:

1
def compute(self, doc: Doc, model_response: ModelResponse, **kwargs) -> float:
2
    """Computes the metric over a list of golds and predictions for one single sample.
3

4
    Args:
5
        doc (Doc): The document containing gold references.
6
        model_response (ModelResponse): The model's response containing predictions.
7
        **kwargs: Additional keyword arguments.
8

9
    Returns:
10
        float: Aggregated score over the current sample's items.
11
    """
12
    results = []
13
    # We might need to flatten golds if they are a list of lists
14
    golds = doc.get_golds()
15
    for gold in golds:
16
        for pred in model_response.final_text:
17
            results.append(self.compute_one_item(gold=gold, pred=pred))
18
    return self.aggregation_function(results)

OpenCompass

OpenCompass is similar: users configure a model and dataset. For example:

1
opencompass --models hf_qwen2_0_5b_instruct_cpu --datasets demo_gsm8k_chat_gen

model: a small Qwen model
dataset: GSM8K

1. Partition the Dataset

1
partitioner = PARTITIONERS.build(cfg.infer.partitioner)
2
tasks = partitioner(cfg)

Users can specify a Partitioner; by default, splitting follows max_number_workers. Each Partitioner constructs task configurations from its partitions and returns a list, for example NumWorkerPartitioner:

1
for dataset in chunks:
2
    tasks.append(
3
        Config({
4
            'models': [model],
5
            'datasets': [[dataset]],
6
            'work_dir': work_dir,
7
            **add_cfg
8
        }))

2. Construct and Execute the Runner

1
runner = RUNNERS.build(cfg.infer.runner)
2
runner(tasks)

Initialize a runner with task configurations and invoke launch. For example, LocalRunner uses multiple processes for parallel execution, unlike the LightEval flow above:

1
pbar = tqdm(total=len(tasks))
2
lock = Lock()
3

4
def submit(task, index):
5
    task = TASKS.build(dict(cfg=task, type=self.task_cfg['type']))
6
    num_gpus = task.num_gpus
7
    assert len(gpus) >= num_gpus
8

9
    while True:
10
        lock.acquire()
11
        if sum(gpus > 0) >= num_gpus:
12
            gpu_ids = np.where(gpus)[0][:num_gpus]
13
            gpus[gpu_ids] -= 1
14
            lock.release()
15
            break
16
        lock.release()
17
        time.sleep(1)
18

19
    res = self._launch(task, gpu_ids, index)
20
    pbar.update()
21

22
    with lock:
23
        gpus[gpu_ids] += 1
24

25
    return res
26

27
with ThreadPoolExecutor(
28
        max_workers=self.max_num_workers) as executor:
29
    status = executor.map(submit, tasks, range(len(tasks)))

Because execution is parallel, GPU resource records are locked, and ThreadPoolExecutor submits work concurrently.

3. Evaluation

OpenCompass evaluation differs from LightEval in these ways:

OpenCompass designs evaluation with the same Runner/Task architecture as inference. One benefit is that users can run evaluation separately;
LightEval integrates evaluation into its pipeline, so it cannot run independently;

The overall flow resembles LightEval, with preprocessing, scoring, and saving.