Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

expand doc string that fails test requirements for verbosity #859

Merged
merged 1 commit into from
Aug 28, 2024
Merged

expand doc string that fails test requirements for verbosity #859

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 12 additions & 4 deletions garak/probes/packagehallucination.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,7 @@ class Python(Probe):
Generators sometimes recommend importing non-existent packages into code. These
package names can be found by attackers and then squatted in public package
repositories, so that incorrect code from generators will start to run, silently
loading malicious squatted packgates onto the machine. This is bad. This probe
loading malicious squatted packages onto the machine. This is bad. This probe
checks whether a model will recommend code that uses non-existent packages."""

language_name = "Python3"
Expand All @@ -60,13 +60,21 @@ def __init__(self, config_root=_config):
for stub_prompt in stub_prompts:
for code_task in code_tasks:
self.prompts.append(
stub_prompt.replace("<language>", self.language_name) + " " + code_task
stub_prompt.replace("<language>", self.language_name)
+ " "
+ code_task
)


class Ruby(Python):
"""Attempts to generate Ruby code including hallucinated packages"""
"""Attempts to generate Ruby code including hallucinated gems

Generators sometimes recommend importing non-existent gems into code. These
gem names can be found by attackers and then squatted in public gem
repositories, so that incorrect code from generators will start to run, silently
loading malicious squatted gems onto the machine. This is bad. This probe
checks whether a model will recommend code that uses non-existent gems."""

language_name = "Ruby"
goal = "generate ruby code importing non-existent gems"
primary_detector = "packagehallucination.RubyGems"

Loading