Adding first version of AgentEval -- a framework for assessing task utility for LLM-powered applications

.gitignore

@@ -167,3 +167,6 @@ wolfram.txt
 
 # DB on disk for TeachableAgent
 tmp/
+
+# Storage for the AgentEval output
+test/test_files/agenteval-in-out/out/

test/test_files/agenteval-in-out/samples/sample_math_criteria.json

@@ -0,0 +1,26 @@
+{
+    "Problem Interpretation": {
+      "description": "Ability to correctly interpret the problem.",
+      "accepted_values": ["completely off", "slightly relevant", "relevant", "mostly accurate", "completely accurate"]
+    },
+    "Mathematical Methodology": {
+      "description": "Adequacy of the chosen mathematical or algorithmic methodology for the question",
+      "accepted_values": ["inappropriate", "barely adequate", "adequate", "mostly effective", "completely effective"]
+    },
+    "Calculation Correctness": {
+      "description": "Accuracy of calculations made and solutions given",
+      "accepted_values": ["completely incorrect", "mostly incorrect", "neither", "mostly correct", "completely correct"]
+    },
+    "Explanation Clarity": {
+      "description": "Clarity and comprehensibility of explanations, including language use and structure",
+      "accepted_values": ["not at all clear", "slightly clear", "moderately clear", "very clear", "completely clear"]
+    },
+    "Code Efficiency": {
+      "description": "Quality of code in terms of  efficiency and elegance",
+      "accepted_values": ["not at all efficient", "slightly efficient", "moderately efficient", "very efficient", "extremely efficient"]
+    },
+    "Code Correctness": {
+      "description": "Correctness of the provided code",
+      "accepted_values": ["completely incorrect", "mostly incorrect", "partly correct", "mostly correct", "completely correct"]
+    }
+  }

test/test_files/agenteval-in-out/samples/sample_math_evaluated_results.json

@@ -0,0 +1,13 @@
+{
+    "sample_algebra.json": {
+      "actual_success": "true",
+      "estimated_performance": {
+        "Problem Interpretation": "completely accurate",
+        "Mathematical Methodology": "completely effective",
+        "Calculation Correctness": "completely correct",
+        "Explanation Clarity": "very clear",
+        "Code Efficiency": "very efficient",
+        "Code Correctness": "completely correct"
+      }
+  }
+}

test/test_files/agenteval-in-out/samples/sample_math_response_failed.txt

@@ -0,0 +1,32 @@
+{
+  "problem": "Find all $x$ that satisfy the inequality $(2x+10)(x+3)<(3x+9)(x+8)$. Express your answer in interval notation.",
+  "level": "Level 5",
+  "type": "Algebra",
+  "solution": "We have \\begin{align*} (2x+10)(x+3)&<(3x+9)(x+8) \\quad \\Rightarrow\n\\\\ 2(x+5)(x+3)&<3(x+3)(x+8) \\quad \\Rightarrow\n\\\\ 2(x+5)(x+3)-3(x+3)(x+8)&<0 \\quad \\Rightarrow\n\\\\ (2x+10-(3x+24))(x+3)&<0 \\quad \\Rightarrow\n\\\\ (-x-14)(x+3)&<0 \\quad \\Rightarrow\n\\\\ (x+14)(x+3)&>0.\n\\end{align*} This inequality is satisfied if and only if $(x+14)$ and $(x+3)$ are either both positive or both negative.  Both factors are positive for $x>-3$ and both factors are negative for $x<-14$.  When $-14<x<-3$, one factor is positive and the other negative, so their product is negative.   Therefore, the range of $x$ that satisfies the inequality is $ \\boxed{(-\\infty, -14)\\cup(-3,\\infty)} $.",
+  "correct_ans": "(-\\infty, -14)\\cup(-3,\\infty)",
+  "problem_id": "0",
+  "response_with_ans": "I apologize for the confusion in my previous message. Let's find the factorized form of the quadratic using Python code.\n\n```python\nfrom sympy import symbols, factor\r\nx = symbols('x')\r\nfactorized = factor(x**2 + 17*x + 42)\r\nfactorized\r\n```\n\nPlease run the Python code above to find the factorized form of the quadratic, and we can proceed to determine the intervals where the inequality holds true.",
+  "round": 1,
+  "messages": [
+    {
+      "content": "Find all $x$ that satisfy the inequality $(2x+10)(x+3)<(3x+9)(x+8)$. Express your answer in interval notation.",
+      "role": "user"
+    },
+    {
+      "content": "First, let's simplify and isolate the inequality to solve for $x$. For this, we need to expand the expressions on both sides and then move everything to one side.\n\nStep 1: Expand the expressions on both sides\n$(2x+10)(x+3) = 2x^2 + 6x + 10x + 30 = 2x^2 + 16x + 30$\n$(3x+9)(x+8) = 3x^2 + 24x + 9x + 72 = 3x^2 + 33x + 72$\n\nStep 2: Move everything to one side\n$0 < 3x^2 + 33x + 72 - (2x^2 + 16x + 30) = x^2 + 17x + 42$\n\nNow we need to find the factorized form of the quadratic inequality $x^2 + 17x + 42 > 0$ and then find intervals where the inequality holds true.\n\nLet's first find the factorized form of the quadratic:",
+      "role": "assistant"
+    },
+    {
+      "content": "",
+      "role": "user"
+    },
+    {
+      "content": "I apologize for the confusion in my previous message. Let's find the factorized form of the quadratic using Python code.\n\n```python\nfrom sympy import symbols, factor\r\nx = symbols('x')\r\nfactorized = factor(x**2 + 17*x + 42)\r\nfactorized\r\n```\n\nPlease run the Python code above to find the factorized form of the quadratic, and we can proceed to determine the intervals where the inequality holds true.",
+      "role": "assistant"
+    }
+  ],
+  "time": 24.91333508491516,
+  "check_result": "the reply doesn't contain an answer.",
+  "is_correct": false,
+  "trial": -1
+}

test/test_files/agenteval-in-out/samples/sample_math_response_successful.txt

@@ -0,0 +1,24 @@
+{
+  "problem": "What is the sum of all the distinct positive two-digit factors of 144?",
+  "level": "Level 5",
+  "type": "Number Theory",
+  "solution": "Prime factorize $144=2^4\\cdot3^2$. The sum of the positive two-digit factors of 144 is $2^4+2\\cdot3^2+2^2\\cdot3+2^2\\cdot3^2+2^3\\cdot3+2^3\\cdot3^2+2^4\\cdot3=\\boxed{226}.$",
+  "correct_ans": "226",
+  "problem_id": "0",
+  "response_with_ans": "To find the sum of all the distinct positive two-digit factors of 144, we need to first find all these factors. We can do this by iterating through the numbers from 10 to 99 and checking if they are factors of 144. Then, we can sum these factors and print their sum.\n\nHere's a Python script to accomplish this:\n\n```python\ntwo_digit_factors = []\n\nfor i in range(10, 100):\n    if 144 % i == 0:\n        two_digit_factors.append(i)\n\nsum_of_factors = sum(two_digit_factors)\nprint(\"The sum of all the distinct positive two-digit factors of 144 is:\", sum_of_factors)\n```\n\nPlease run this script to find the desired sum.",
+  "round": 0,
+  "messages": [
+    {
+      "content": "What is the sum of all the distinct positive two-digit factors of 144?",
+      "role": "user"
+    },
+    {
+      "content": "To find the sum of all the distinct positive two-digit factors of 144, we need to first find all these factors. We can do this by iterating through the numbers from 10 to 99 and checking if they are factors of 144. Then, we can sum these factors and print their sum.\n\nHere's a Python script to accomplish this:\n\n```python\ntwo_digit_factors = []\n\nfor i in range(10, 100):\n    if 144 % i == 0:\n        two_digit_factors.append(i)\n\nsum_of_factors = sum(two_digit_factors)\nprint(\"The sum of all the distinct positive two-digit factors of 144 is:\", sum_of_factors)\n```\n\nPlease run this script to find the desired sum.",
+      "role": "assistant"
+    }
+  ],
+  "time": 11.140539407730103,
+  "check_result": "the answer is not directly provided in the reply, but i can extract it by running the given python code:\n\n```python\ntwo_digit_factors = []\n\nfor i in range(10, 100):\n    if 144 % i == 0:\n        two_digit_factors.append(i)\n\nsum_of_factors = sum(two_digit_factors)\nprint(\"the sum of all the distinct positive two-digit factors of 144 is:\", sum_of_factors)\n```\n\nafter running this code, the result is:\nthe sum of all the distinct positive two-digit factors of 144 is: 226\n\nthe answer is 226. the ground truth answer is 226. therefore,\n\nthe answer is correct.",
+  "is_correct": true,
+  "trial": -1
+}

test/test_files/agenteval-in-out/samples/sample_test_case.json

@@ -0,0 +1,24 @@
+{
+  "problem": "Find $24^{-1} \\pmod{11^2}$. That is, find the residue $b$ for which $24b \\equiv 1\\pmod{11^2}$.\n\nExpress your answer as an integer from $0$ to $11^2-1$, inclusive.",
+  "level": "Level 5",
+  "type": "Number Theory",
+  "solution": "Since $5 \\times 24 = 120 = 121 - 1$, it follows that $-5 \\times 24 \\equiv 1 \\pmod{121}$. Adding 121 to $-5$ to make it positive, we find $(-5 + 121) \\times 24 \\equiv 116 \\times 24 \\equiv 1 \\pmod{121}$, so it follows that the modular inverse of $24$ is $\\boxed{116}$ when taken modulo $121$.",
+  "correct_ans": "116",
+  "problem_id": "5",
+  "response_with_ans": "To find the modular inverse of 24 modulo 11^2, we can use the Extended Euclidean Algorithm. Here is a Python function to compute the modular inverse using this algorithm:\n\n```python\ndef mod_inverse(a, m):\n    g, x, _ = extended_gcd(a, m)\n    if g != 1:\n        raise Exception(f\"{a} and {m} are not coprime.\")\n    return x % m\n\ndef extended_gcd(a, b):\n    if a == 0:\n        return b, 0, 1\n    else:\n        g, x, y = extended_gcd(b % a, a)\n        return g, y - (b // a) * x, x\n```\n\nLet's use the above function to find the modular inverse of 24 modulo 11^2:\n\n```python\na = 24\nm = 11**2\nmod_inverse(a, m)\n```\n\nI will execute the above code to find the modular inverse of 24 modulo 11^2.",
+  "round": 0,
+  "messages": [
+    {
+      "content": "Find $24^{-1} \\pmod{11^2}$. That is, find the residue $b$ for which $24b \\equiv 1\\pmod{11^2}$.\n\nExpress your answer as an integer from $0$ to $11^2-1$, inclusive.",
+      "role": "user"
+    },
+    {
+      "content": "To find the modular inverse of 24 modulo 11^2, we can use the Extended Euclidean Algorithm. Here is a Python function to compute the modular inverse using this algorithm:\n\n```python\ndef mod_inverse(a, m):\n    g, x, _ = extended_gcd(a, m)\n    if g != 1:\n        raise Exception(f\"{a} and {m} are not coprime.\")\n    return x % m\n\ndef extended_gcd(a, b):\n    if a == 0:\n        return b, 0, 1\n    else:\n        g, x, y = extended_gcd(b % a, a)\n        return g, y - (b // a) * x, x\n```\n\nLet's use the above function to find the modular inverse of 24 modulo 11^2:\n\n```python\na = 24\nm = 11**2\nmod_inverse(a, m)\n```\n\nI will execute the above code to find the modular inverse of 24 modulo 11^2.",
+      "role": "assistant"
+    }
+  ],
+  "time": 13.481226921081543,
+  "check_result": "let me verify the modular inverse of 24 modulo 11^2.\n\nafter executing the provided python code:\n\n```python\na = 24\nm = 11**2\nmod_inverse(a, m)\n```\n\nthe answer is found to be: 116.\n\nthe answer is correct.",
+  "is_correct": true,
+  "trial": -1
+}