Rethinking Rethinking Search: Making Domain Experts out of Dilettantes

The title of the Metzler et al. (2021) paper, “Rethinking Search: Making Domain Experts out of Dilettantes”, is prone to multiple interpretations. They write within of the existing models¹ as dilettantes, but my orientation towards understanding generative applications in search-in-practice makes me think rather of finding places where human dilettantes (like myself in many domains that I search) might be able to successfully interact with (or even interact as) domain experts.

Here is how Metzler et al. write of dilettantes—

In their abstract (p. 1):

Pre-trained language models, by contrast, are capable of directly generating prose that may be responsive to an information need, but at present they are dilettantes rather than domain experts – they do not have a true understanding of the world, they are prone to hallucinating, and crucially they are incapable of justifying their utterances by referring to supporting documents in the corpus they were trained over.

In their introduction (p. 2):

However, such models are dilettantes – they do not have a true understanding of the world, they are prone to hallucinating, and crucially they are incapable of justifying their utterances by referring to supporting documents in the corpus they were trained over.

In § 3 Model-Based Information Retrieval > 3.1 Beyond Language Models (p. 8):

This is why we refer to pre-trained LMs as dilettantes – they are perceived to know a lot but their knowledge is skin deep.

By this usage, surely most of us are “dilettantes”, perhaps minus negative connotations or the pejorative sense, about most things. Even the professional² data engineers that I interviewed in my dissertation research felt like they had to confess to searching all the time and talked about how little they really knew given the wide scope of their work (“there are going to be gaps”). Crucially, the fullness of their expertise was only exhibited and accessible in interaction with their search tools and work practices.