Lawful Good

Get Started

Clone the repo:

git clone [email protected]:dluo96/lawful-good.git

Install the inspect_ai and inspect_evals Python packages:

pip install inspect_ai
pip install git+https://github.com/UKGovernmentBEIS/inspect_evals

Install the package locally

cd lawful-good
pip install -e .

Example Evaluation with Ollama

• Install ollama. On Linux, simply run

  curl -fsSL https://ollama.com/install.sh | sh

• Run (say) Llama2 3B:

  ollama run llama3.2
  

  This will automatically pull (download) Llama2 3B for you and then run.
• To run Inspect evals with Ollama, you will need to install the openai package:

  pip install openai
  

• Run the example evaluation script:

  inspect eval lg/lawful_good.py --model ollama/llama3.2

• NOTE: if you receive the error PermissionError: [Errno 13] Permission denied: '/run/user/1000', try setting the following environment variable:

  export XDG_RUNTIME_DIR=/tmp

  and re-running.

Analyse evaluation runs with Inspect view

inspect view --log-dir lg/logs