Here is the ‘OWASP Top 10 for Large Language Model Applications’. Overreliance is relevant to my research.
(I’ve generally used the term “automation bias”, though perhaps a more direct term like overreliance is better.)
You can see my discussion in the “Extending searching” chapter of my dissertation (particularly the sections on “Spaces for evaluation” and “Decoupling performance from search”) as I look at how data engineers appear to effectively address related risks in their heavy use of general-purpose web search at work. I’m very focused on how the searcher is situated and what they are doing well before and after they actually type in a query (or enter a prompt).
Key lessons in my dissertation: (1) The data engineers are not really left to evaluate search results as they read them and assigning such responsibility could run into Meno’s Paradox (instead there are various tools, processes, and other people that assist in evaluation). (2) While search is a massive input into their work, it is not tightly coupled to their key actions (instead there are useful frictions (and perhaps fictions), gaps, and buffers).
I’d like discussion explicitly addressing “inadequate informing” (wc?), where the information generated is accurate but inadequate given the situation-and-user.
The section does refer to “inappropriate” content, but usage suggests “toxic” rather than insufficient or inadequate.
The OWASP Top 10 for Large Language Model Applications project aims to educate developers, designers, architects, managers, and organizations about the potential security risks when deploying and managing Large Language Models (LLMs). The project provides a list of the top 10 most critical vulnerabilities often seen in LLM applications, highlighting their potential impact, ease of exploitation, and prevalence in real-world applications. Examples of vulnerabilities include prompt injections, data leakage, inadequate sandboxing, and unauthorized code execution, among others. The goal is to raise awareness of these vulnerabilities, suggest remediation strategies, and ultimately improve the security posture of LLM applications. You can read our group charter for more information
OWASP Top 10 for LLM version 1.0
LLM01: Prompt Injection
This manipulates a large language model (LLM) through crafty inputs, causing unintended actions by the LLM. Direct injections overwrite system prompts, while indirect ones manipulate inputs from external sources.LLM02: Insecure Output Handling
This vulnerability occurs when an LLM output is accepted without scrutiny, exposing backend systems. Misuse may lead to severe consequences like XSS, CSRF, SSRF, privilege escalation, or remote code execution.LLM03: Training Data Poisoning
This occurs when LLM training data is tampered, introducing vulnerabilities or biases that compromise security, effectiveness, or ethical behavior. Sources include Common Crawl, WebText, OpenWebText, & books.LLM04: Model Denial of Service
Attackers cause resource-heavy operations on LLMs, leading to service degradation or high costs. The vulnerability is magnified due to the resource-intensive nature of LLMs and unpredictability of user inputs.LLM05: Supply Chain Vulnerabilities
LLM application lifecycle can be compromised by vulnerable components or services, leading to security attacks. Using third-party datasets, pre-trained models, and plugins can add vulnerabilities.LLM06: Sensitive Information Disclosure
LLM’s may inadvertently reveal confidential data in its responses, leading to unauthorized data access, privacy violations, and security breaches. It’s crucial to implement data sanitization and strict user policies to mitigate this.LLM07: Insecure Plugin Design
LLM plugins can have insecure inputs and insufficient access control. This lack of application control makes them easier to exploit and can result in consequences like remote code execution.LLM08: Excessive Agency
LLM-based systems may undertake actions leading to unintended consequences. The issue arises from excessive functionality, permissions, or autonomy granted to the LLM-based systems.LLM09: Overreliance
LLM10: Model Theft
Systems or people overly depending on LLMs without oversight may face misinformation, miscommunication, legal issues, and security vulnerabilities due to incorrect or inappropriate content generated by LLMs.
This involves unauthorized access, copying, or exfiltration of proprietary LLM models. The impact includes economic losses, compromised competitive advantage, and potential access to sensitive information.