Measurement to Meaning: A Validity-Centered Framework for AI Evaluation

The paper provides a structured approach for reasoning about the types of evaluative claims that can be made given the available evidence.

Proposes a validity-centered framework for AI evaluation that reasons explicitly about which evaluative claims the evidence actually supports, with detailed vision and language case studies. The operational companion to the Wallach/Jacobs position paper.

The applied-framework companion to the Wallach/Jacobs position paper. Where the position paper diagnoses the field, this paper hands you a tool: map the evidence you have to the claim you want to make, and refuse to make claims the evidence can't support.

• Read the original →

Where the Wallach/Jacobs position paper argues that generative-AI evaluation is a social-science measurement challenge, this paper supplies the operational framework. Two case studies (vision and language model evaluations) demonstrate how explicitly reasoning about validity strengthens or weakens the claims an evaluation can support. The central move is refusing the shortcut from benchmark score to capability claim. A model that does well on a math benchmark may be good at that benchmark, not good at math. A model that does well on graduate-exam-style questions may be good at graduate-exam-style questions, not good at reasoning. Useful against • "Reward richness is the lever" framings — this paper asks which construct the reward even measures. • "Thin harness, fat skills" — reminds that the skills you think you are pushing into the model are defined by the evaluations you check them with. Useful for • Anyone who wants to score a library entry on institutional_ratification or observability with conceptual grounding rather than intuition.

Title, authors, and URL verified; date is best-estimate from the arxiv ID (2505 = May 2025) — confirm exact first-submission date before citing.