Add LLM VRT Entries

[drunkrhin0]

Add VRT entries for LLMs based on the OWASP LLM Top 10 v1.1

Add to the following existing categories:

• P1: Sensitive Data Exposure > Disclosure of Secrets > LLM Output Handling
• Varies: Application Level DoS > Excessive Resource Consumption > Injection (Prompt)

Add a LLM Security Misconfiguration category:

• P1: LLM Security Misconfiguration > Prompt Injection
• P1: LLM Security Misconfiguration > Training Data Poisoning
• P2: LLM Security Misconfiguration > Excessive Agency > Permission Manipulation via LLM
• P4: LLM Security Misconfiguration > Insecure Application Logic > Overreliance on LLM Output

I've prepared an updated json file with the relevant changes pending approval and creating a PR - Branch

[GangGreenTemperTatum]

Hey @drunkrhin0 , this is awesome thank you, definitely needed!! The file in master shows that you haven't commited to the VRT? Also looks like your PR failed?

Do you have a visual aid for your implementation? Sorry my old eyes 👀

Also, for reference I'd personally recommend you change wording ~"LLM Security" to "LLM Application Security" - Since red teaming a model (I.E prompt injection to return biased or falsified information) is different than red teaming an LLM application (I.E Direct Prompt Injection to trigger Excessive Agency or Insecure Plugins which could cause XSS) - Check this out as an example.

Additionally, I strongly assume that BugCrowd bug bounty programs and VRT will only focus on LLM applications, not LLMs themselves. As such, your "Prompt Injection" narrative could scope soley on "direct Prompt Injection" in mostly 99.9% of cases since its active participation in the program

[arcwhite]

Some of these sub-categories and specific items might apply to other machine learning implementations that are not LLMs. Is it worth broadening the top-level category slightly to be something like "Machine Learning Security"?

Even if we don't, I think "LLM Security Misconfiguration" might need tweaking - some of these aren't necessarily misconfigurations, they could be design or implementation flaws (insert philosophical debate about what is a "misconfiguration" here)

[drunkrhin0]

@GangGreenTemperTatum @arcwhite what are your thoughts on something like AI/ML Application Security?

[arcwhite]

Yeah if we want to go broad, that might be even better...

[GangGreenTemperTatum]

Couldn't agree more, I guess the ultimate question of what this delves down to, what is the immediate VRT scope required to support BB programs? I'd personally assume (on behalf of BC) that this is LLM Application Security (IE a chat bot - take OpenAI ChatGPT as a real world example), which at a high-level, is a model that sits inside infrastructure and is accessible via an API (which the current defacto VRT caters for - Web Application Security or AppSec)

ML Security (one example, training data poisoning via front running or split-view data poisoning) is not feasible or scalable to have as a bug bounty VRT entry. However, examples for LLM Application Security such as triggering Escalation of Privilege via a chat bot plugin (I.E RAG) - See "confused deputy" is feasible to exploit and submit a valid PoC for. Machine Learning security can also vary for different types of model developers, I.E Foundation Models, Derivative Models etc.

However LLM Application security can encompass all of the factors with a general web application that sits as a wrapper.

... Apologies, missing example hyperlinks as I'm on cell and AFK right now. Hope that helps with clarity?

[GangGreenTemperTatum]

Additionally with ML Security, this may become difficult territory to map as you then start the conversation of LLM Vulnerability Scans (IE jailbreak prompts) and Model Scans (IE serialization attacks), these themselves are risk but outputs that are biased, falsified or erroneous are problems tailored around generative AI safety. As a security engineer, I'd be more concerned about how a jailbreak prompt injection may link to excessive agency and therefore a plugin exploit.

Also thinking out loud, the VRT should have an explicit statement outlining the rules around hallucinations (which are going to be super common submissions for Remote Code Execution attacks).

Additionally, for the OWASP Top 10 for LLM Applications next steps of the project, we plan to deliver a mapping of known CWE's and additional frameworks to the top 10.

[GangGreenTemperTatum]

@drunkrhin0 @arcwhite , what do you guys think?

[drunkrhin0]

I thought as much. ML security can always have it's own category if required. I'll seek thoughts around it when the next meeting occurs. I.e. I would likely prefer to gravitate towards "AI Application Security". While they share similarities/the spotlight they are still a bit too different to group together.

[TimmyBugcrowd]

AI applications often integrate complex interactions with users and other systems that may not be present in ML contexts.

Therefore, I agree with @drunkrhin0 's perspective on leaning towards "AI Application Security" as a distinct category as a start.

We can have changes in the near future while actively observing the development of AI/ML and act accordingly once new categories are needed. For the immediate solution, I agree with "AI Application Security" as mentioned above.

[drunkrhin0]

Great to hear @TimmyBugcrowd I'll refine it a bit more and then update it soon

[GangGreenTemperTatum]

Fully respect that, I assume your including all of the major types of AI here also