arxiv:2409.00844

Report Cards: Qualitative Evaluation of Language Models Using Natural Language Summaries

Published on Sep 1

· Submitted by

loveblairsky on Sep 6

Upvote

Authors:

Blair Yang ,

Fuyang Cui ,

Pashootan Vaezipoor ,

Silviu Pitis ,

Abstract

The rapid development and dynamic nature of large language models (LLMs) make it difficult for conventional quantitative benchmarks to accurately assess their capabilities. We propose report cards, which are human-interpretable, natural language summaries of model behavior for specific skills or topics. We develop a framework to evaluate report cards based on three criteria: specificity (ability to distinguish between models), faithfulness (accurate representation of model capabilities), and interpretability (clarity and relevance to humans). We also propose an iterative algorithm for generating report cards without human supervision and explore its efficacy by ablating various design choices. Through experimentation with popular LLMs, we demonstrate that report cards provide insights beyond traditional benchmarks and can help address the need for a more interpretable and holistic evaluation of LLMs.

View arXiv page View PDF Add to collection

Community

loveblairsky

Paper author Paper submitter 14 days ago

🔍 Current LLM evaluations fall short:
• Lack nuanced understanding of model capabilities
• Overly focused on quantitative metrics
• Difficult for humans to interpret

Introducing LLM Report Cards: A novel approach for qualitative, interpretable model evaluation.
Report Cards provide concise, fine-grained descriptions of a model characteristic behaviors, including its strengths and weaknesses, with respect to specific topics, such as math, biology, and safety-focused questions.