"And what matters is if it works."

I discuss evaluation of general-purpose search results in my dissertation, largely in Ch. 4. Extending searching—which as a whole gets at a key position I have on evaluating the performance of LLMs: we have to extend our observations and analysis well-beyond the raw outputs of these systems. In that chapter I discuss “Running workable code”, “Decoupling performance from search”, and note that not all qualities desirable in code are so easily testable as simply ‘running it’ (but the processes and practices of the organization is the final evaluator, for my professional data engineer participants at least):

But workable in the moment isn’t enough. There is other information that isn’t immediately testable by running it through a compiler or interpreter and seeing if it “works”. This is particularly the case for non-functional properties (normally including things like security, reliability, and scalability but could also include considerations of harm both in the implementation and use of the code (Widder et al., 2022, p. 8)).