"We really need to talk more about monitoring search quality for public interest topics."

    [cat:weblinks]
    August 31st, 2023

      Dave Guarino (website | Twitter; “the founding engineer (and then Director) of GetCalFresh.org at Code for America”)
      @allafarce via Twitter on Jan 16, 2020

      We really need to talk more about monitoring search quality for public interest topics.

      Guarino has been regularly sharing examples of “Google Search results for public benefits” on Twitter and discussing “public interest search quality” and “#PublicInterestSEO”, see: Twitter[from:allafarce public interest search]. Here are a list of such queries, linked to his tweets about them:

      He has shared about asking questions of various LLM models, including in ChainForge (Arawjo et al., 2023): [What is the SNAP asset limit in Illinois?] and [north carolina dss fair hearing request phone number]

      See also work looking at new evaluations needed with search engines shifting to look at “societal relevance” (Sundin et al., 2021)

      Here are examples just a few pieces doing some public interest search quality evaluation and search audits (though not necessarily providing ongoing monitoring and sometimes evaluating things much more contentious and broad):

      References

      Arawjo, I., Vaithilingam, P., Swoopes, C., Wattenberg, M., & Glassman, E. (2023). ChainForge. https://www.chainforge.ai/. [arawjo2023chainforge]

      Guendelman, S., Pleasants, E., Cheshire, C., & Kong, A. (2022). Exploring google searches for out-of-clinic medication abortion in the united states during 2020: Infodemiology approach using multiple samples. JMIR Infodemiology, 2(1), e33184. https://doi.org/10.2196/33184 [guendelman2022exploring]

      Lurie, E., & Mulligan, D. K. (2021). Searching for representation: A sociotechnical audit of googling for members of U.S. Congress. https://arxiv.org/abs/2109.07012 [lurie2021searching_facctrec]

      Mejova, Y., Gracyk, T., & Robertson, R. (2022). Googling for abortion: Search engine mediation of abortion accessibility in the united states. JQD, 2. https://doi.org/10.51685/jqd.2022.007 [mejova2022googling]

      Mustafaraj, E., Lurie, E., & Devine, C. (2020). The case for voter-centered audits of search engines during political elections. FAT* ’20. [mustafaraj2020case]

      Noble, S. U. (2018). Algorithms of oppression how search engines reinforce racism. New York University Press. https://nyupress.org/9781479837243/algorithms-of-oppression/ [noble2018algorithms]

      Sundin, O., Lewandowski, D., & Haider, J. (2021). Whose relevance? Web search engines as multisided relevance machines. Journal of the Association for Information Science and Technology. https://doi.org/10.1002/asi.24570 [sundin2021relevance]

      Urman, A., & Makhortykh, M. (2022). “Foreign beauties want to meet you”: The sexualization of women in google’s organic and sponsored text search results. New Media & Society, 0(0), 14614448221099536. https://doi.org/10.1177/14614448221099536 [urman2022foreign]

      Urman, A., Makhortykh, M., & Ulloa, R. (2022). Auditing the representation of migrants in image web search results. Humanit Soc Sci Commun, 9(1), 5. https://doi.org/10.1057/s41599-022-01144-1 [urman2022auditing]

      Urman, A., Makhortykh, M., Ulloa, R., & Kulshrestha, J. (2022). Where the earth is flat and 9/11 is an inside job: A comparative algorithm audit of conspiratorial information in web search results. Telematics and Informatics, 72, 101860. https://doi.org/10.1016/j.tele.2022.101860 [urman2022earth]

      Zade, H., Wack, M., Zhang, Y., Starbird, K., Calo, R., Young, J., & West, J. D. (2022). Auditing google’s search headlines as a potential gateway to misleading content. Journal of Online Trust and Safety, 1(4). https://doi.org/10.54501/jots.v1i4.72 [zade2022auditing]