The latest generation of plugins for IDEs that some people¹²¹ suggest might replace web search are those that support code generation directly from comments written within the code. This is a variant of a larger class of tools, called code generation tools.¹²² One such plugin is GitHub’s Copilot, based on the OpenAI Codex model, itself based on the Generative Pre-trained Transformer (GPT) models from OpenAI. My interview research did not generally directly address GitHub’s Copilot.¹²³ The data engineers I asked about it had not used it. I will not go into the technical mechanisms of these systems, other than note that they are designed to take a prompt and predict the most likely text strings to follow. If I type “Mary had a” into the OpenAI GPT-3 Playground¹²⁴ , the system completes the nursery rhyme.

GitHub Copilot, free to GitHub verified students, teachers, and maintainers of popular open source projects¹²⁵ , is trained on portions of the significant amount of code uploaded to GitHub, which raises legal, ethical, and security concerns.¹²⁶ can It perhaps be imagined as an advanced autocomplete. Rather than suggesting the completion of a function name or command in your code, the plugin will suggest an entire block of code, perhaps the entire function. When the user types a comment or a line of code, the plugin will suggest a completion. Some users appear to have been satisfied with these suggestions. GitHub wrote a blog post in July 2022 reporting on a survey of Copilot users combined with data on their shown and accepted Copilot suggestions (Ziegler, 2022) . They claim “[u]sing GitHub Copilot correlates with improved developer productivity”. GitHub continues to publish reports along this line (Kalliamvakou, 2022) , while there is also intense interest from external researching on the use and effects of GitHub Copilot.

External researchers are particularly examining security vulnerabilities (some in a manner similar to that of Fischer et al. (2017) and others mentioned in Extending searching ). A team from NYU and the University of Calgary examined code suggested by GitHub copilot scenarios developed relevant to MITRE’s “Top 25” Common Weakness Enumeration (a regularly updated list of significant software vulnerabilities). Across the 89 scenarios they had Copilot produce over 1,500 programs of which they found approximately 40% to be vulnerable (Pearce et al., 2021) . They recommend that Copilot “should be paired with appropriate security-aware tooling during both training and generation to minimize the risk of introducing security vulnerabilities.”

Security concerns, and such precautions, are also acknowledged by GitHub¹²⁷ :

You should take the same precautions as you would with any code you write that uses material you did not independently originate. These include rigorous testing, IP scanning, and checking for security vulnerabilities. You should make sure your IDE or editor does not automatically compile or run generated code before you review it.

These suggested responses to vulnerabilities in Copilot mirror some of what I discussed in Extending searching around the evaluation of search results and the decoupling from search.

The plugins for IDEs, and the voice-search, free-text, as in the OpenAI GPT-3 Playground¹²⁸ and chat-based, ex. ChatGPT¹²⁹ , language interfaces are user interface components that, in the language of Handoff, provide distinct engagements for interactions. One of the benefits of using a general-purpose search engine is the contestability and interrogatability, perhaps not of the ranking of the websites on the SERP, but of the results. Data engineer searchers can look at the websites where they find information to gain clues as to its provenance and trustworthiness. GitHub Copilot and ChatGPT are black boxed and do not currently provide access for that, though GitHub has announced future product changes that will allow some interrogation¹³⁰ , beyond directly engaging with the system for alternatives.

The tool designers will likely continue to improve the tool, and IDEs and companies may adapt practices to pull in such untested code in way primed for effective testing. OpenAI and others continue to do research looking into the hazards posed by such tools (Khlaaf et al., 2022) . Copilot does and will likely continue to replace some subset of searching done by some data engineers. And also just like promises of automatic programming in the past (Ensmenger, 2010) , if it does lower the cost of programming it will likely only increase the demand for more programmers. The programming languages used by my research participants are far simpler to use and understand than even the “automatic programming” languages of the past, like FORTRAN and COBOL. The hard problems remain, how to use a tool to do something you or someone else wants.

A web search engine provides some means to find what others, not only the search engine, say about top ranking search results, in addition to viewing the source website. This is a capacity of the configuration of web search that is leveraged in misinformation research. Some search engines provide interface options to learn general information about a website, to, for instance, see if the website presenting itself as a news organization is identified by Google as one. Mike Caulfield’s SIFT model for basic fact-checking practices has four moves: Stop, Investigate the source, find better coverage, trace the original context (Caulfield, 2019b) . Those moves are not supported by the Copilot configuration itself. Users of Copilot and other such tools will still find it helpful to refer to web search, and turn to sources of search repair, for learning things they do not already know.¹³¹ GitHub’s investments in Copilot come alongside a significant redesign of their search platform, for searching for code within GitHub (GitHub, 2022) ¹³² , suggesting a recognition from the creators of the tool that search will not be fully replaced.

Bibliography