Evaluating LLMs: Beyond Accuracy and Perplexity

The Evaluation Crisis

Traditional metrics (accuracy, F1, perplexity) fail to capture LLM capabilities. 2025 has introduced richer evaluation frameworks.

Beyond Static Benchmarks

HELM (Holistic Evaluation of Language Models)

Stanford's framework evaluates across:

• Accuracy, calibration, robustness, fairness, bias, toxicity, efficiency
• Multiple scenarios (QA, reasoning, instruction following)
• All major models regularly updated

BIG-bench (Beyond the Imitation Game)

204 tasks designed to probe model capabilities:

• Logical reasoning, common sense, creativity, deception detection
• Multi-step tasks requiring planning
• Language understanding across 50+ languages

LLM-as-Judge

Using LLMs to evaluate LLM outputs. Surprisingly effective:

prompt = f"""
Evaluate the following response on:
1. Correctness (1-5)
2. Helpfulness (1-5)
3. Harmlessness (1-5)

Response: {response}

Output as JSON: {{"correctness": x, "helpfulness": y, "harmlessness": z}}
"""

judgment = gpt4.generate(prompt)

Calibration needed: Compare to human judgments first.

Task-Specific Metrics 2025

• Code generation: Pass@k (passes unit tests), HumanEval, MBPP
• Math reasoning: GSM8K, MATH, TheoremQA
• Instruction following: IFEval, MT-Bench
• Hallucination detection: SelfCheckGPT, FactScore

Evaluating RAG Systems

RAGAS framework components:

• Faithfulness: Does answer contradict context?
• Answer relevance: Is answer relevant to question?
• Context recall: Did we retrieve necessary info?

from ragas import evaluate
from ragas.metrics import faithfulness, answer_relevancy, context_recall

result = evaluate(
    dataset=rag_dataset,
    metrics=[faithfulness, answer_relevancy, context_recall]
)
print(result)

Production Evaluation

• A/B testing: Compare model versions in production
• Human feedback loops: Collect thumbs up/down
• Monitoring dashboards: Track metrics over time