Adding first version of AgentEval -- a framework for assessing task utility for LLM-powered applications

.gitignore

@@ -167,3 +167,7 @@ wolfram.txt
 
 # DB on disk for TeachableAgent
 tmp/
+
+# Storage for the AgentEval output
+test/test_files/agenteval-in-out/out/
+

test/test_files/agenteval-in-out/agentchat_results/prealgebra/0.json

@@ -0,0 +1,40 @@
+{
+  "problem": "Triangle $ABC$ is a right triangle. If the measure of angle $PAB$ is $x^\\circ$ and the measure of angle $ACB$ is expressed in the form $(Mx+N)^\\circ$ with $M=1$, what is the value of $M+N$?\n\n[asy]\ndraw((-10,0)--(20,0),linewidth(1),Arrows);\ndraw((0,0)--(10,10/sqrt(3))--(10+10/3,0),linewidth(1));\n\ndraw((10,10/sqrt(3))+dir(-150)--(10,10/sqrt(3))+dir(-150)+dir(-60)--(10,10/sqrt(3))+dir(-60),linewidth(1));\n\ndot((-3,0));\n\ndraw(dir(180)..dir(105)..dir(30),linewidth(1));\n\nlabel(\"P\",(-3,0),NW);\nlabel(\"A\",(0,0),S);\nlabel(\"$x^\\circ$\",(-1,1),N);\nlabel(\"B\",(10,10/sqrt(3)),N);\nlabel(\"C\",(10+10/3,0),NE);\n\n[/asy]",
+  "level": "Level 5",
+  "type": "Prealgebra",
+  "solution": "Since $\\angle PAB$ and $\\angle BAC$ are supplementary, $\\angle BAC = 180^{\\circ} - x^\\circ$.  Since the three angles of a triangle add up to $ 180^{\\circ} $, we have $\\angle ACB = 180^{\\circ} - 90^{\\circ} - (180^{\\circ} - x^\\circ) = x^\\circ - 90^{\\circ}$.  Thus, $M + N = \\boxed{-89}$.",
+  "correct_ans": "-89",
+  "problem_id": "0",
+  "response_with_ans": "We know that $x + y = 180^\\circ$. From this equation, we can express $y$ in terms of $x$: $y = 180^\\circ - x$.\n\nNow we substitute the expression for $y$ in the expression of angle $ACB$:\n$ACB = 90^\\circ - y = 90^\\circ - (180^\\circ - x) = x - 90^\\circ$.\n\nComparing this expression with the given form $(Mx + N)^\\circ$, we can see that $M = 1$ and $N = -90$. Therefore, $M + N = 1 + (-90) = -89$.\n\nThe value of $M + N$ is $-89$.",
+  "round": 2,
+  "messages": [
+    {
+      "content": "Triangle $ABC$ is a right triangle. If the measure of angle $PAB$ is $x^\\circ$ and the measure of angle $ACB$ is expressed in the form $(Mx+N)^\\circ$ with $M=1$, what is the value of $M+N$?\n\n[asy]\ndraw((-10,0)--(20,0),linewidth(1),Arrows);\ndraw((0,0)--(10,10/sqrt(3))--(10+10/3,0),linewidth(1));\n\ndraw((10,10/sqrt(3))+dir(-150)--(10,10/sqrt(3))+dir(-150)+dir(-60)--(10,10/sqrt(3))+dir(-60),linewidth(1));\n\ndot((-3,0));\n\ndraw(dir(180)..dir(105)..dir(30),linewidth(1));\n\nlabel(\"P\",(-3,0),NW);\nlabel(\"A\",(0,0),S);\nlabel(\"$x^\\circ$\",(-1,1),N);\nlabel(\"B\",(10,10/sqrt(3)),N);\nlabel(\"C\",(10+10/3,0),NE);\n\n[/asy]",
+      "role": "user"
+    },
+    {
+      "content": "To solve this problem, we need to find the relationship between angle $x$ and angle $ACB$. We know that triangle $ABC$ is a right triangle, so the sum of angles $ACB$ and $ABC$ must be $90^\\circ$. Let's call the angle $ABC$ as $y$. Therefore, angle $ACB = 90^\\circ - y$.\n\nNow, we need to find the relationship between angles $x$ and $y$. Since $PAB$ and $ABC$ share the same vertex $A$ and angle $PAB$ is opposite angle $ABC$, they are explementary (their sum is $180^\\circ$). So, we have $x + y = 180^\\circ$.\n\nNow, we can express $y$ in terms of $x$ and then substitute the expression for $y$ in the expression of angle $ACB$. Finally, we can compare angle $ACB$'s expression with the given form $(Mx + N)^\\circ$ to find the values of $M$ and $N$.\n\nLet me do the calculations.",
+      "role": "assistant"
+    },
+    {
+      "content": "",
+      "role": "user"
+    },
+    {
+      "content": "We know that $x + y = 180^\\circ$. From this equation, we can express $y$ in terms of $x$: $y = 180^\\circ - x$.\n\nNow we substitute the expression for $y$ in the expression of angle $ACB$:\n$ACB = 90^\\circ - y = 90^\\circ - (180^\\circ - x) = x - 90^\\circ$.\n\nComparing this expression with the given form $(Mx + N)^\\circ$, we can see that $M = 1$ and $N = -90$. Therefore, $M + N = 1 + (-90) = -89$.\n\nThe value of $M + N$ is $-89$.",
+      "role": "assistant"
+    },
+    {
+      "content": "",
+      "role": "user"
+    },
+    {
+      "content": "TERMINATE",
+      "role": "assistant"
+    }
+  ],
+  "time": 28.305670976638794,
+  "check_result": "the answer is -89.\n\nthe answer is correct.",
+  "is_correct": true,
+  "trial": -1
+}

test/test_files/agenteval-in-out/agentchat_results/prealgebra/1.json

@@ -0,0 +1,32 @@
+{
+  "problem": "How many positive and negative integers is $12$ a multiple of?",
+  "level": "Level 5",
+  "type": "Prealgebra",
+  "solution": "The number $12$ is a multiple of $-12, -6, -4, -3, -2, -1, 1, 2, 3, 4, 6,$ and $12,$ for a total of $\\boxed{12}$ integers.",
+  "correct_ans": "12",
+  "problem_id": "1",
+  "response_with_ans": "```python\nnum = 12\ndivisors = [i for i in range(1, num + 1) if num % i == 0]\n\npositive_divisors_count = len(divisors)\nnegative_divisors_count = positive_divisors_count\ntotal_divisors_count = positive_divisors_count + negative_divisors_count\n\nprint(f\"Positive divisors of 12: {divisors}\")\ndivisors_negative = [-x for x in divisors]\nprint(f\"Negative divisors of 12: {divisors_negative}\")\nprint(f\"Total divisors count: {total_divisors_count}\")\n```\n\nExecute this code snippet to find the divisors of 12 and count how many positive and negative integers 12 is a multiple of.",
+  "round": 1,
+  "messages": [
+    {
+      "content": "How many positive and negative integers is $12$ a multiple of?",
+      "role": "user"
+    },
+    {
+      "content": "To determine how many positive and negative integers 12 is a multiple of, we need to find the divisors of 12. Then we will double the number of divisors, as each positive divisor has a corresponding negative divisor.\n\nLet's use a Python code snippet to find all the divisors of 12 and count them.",
+      "role": "assistant"
+    },
+    {
+      "content": "",
+      "role": "user"
+    },
+    {
+      "content": "```python\nnum = 12\ndivisors = [i for i in range(1, num + 1) if num % i == 0]\n\npositive_divisors_count = len(divisors)\nnegative_divisors_count = positive_divisors_count\ntotal_divisors_count = positive_divisors_count + negative_divisors_count\n\nprint(f\"Positive divisors of 12: {divisors}\")\ndivisors_negative = [-x for x in divisors]\nprint(f\"Negative divisors of 12: {divisors_negative}\")\nprint(f\"Total divisors count: {total_divisors_count}\")\n```\n\nExecute this code snippet to find the divisors of 12 and count how many positive and negative integers 12 is a multiple of.",
+      "role": "assistant"
+    }
+  ],
+  "time": 17.360238790512085,
+  "check_result": "upon running the provided code snippet, it gives the following output:\n\npositive divisors of 12: [1, 2, 3, 4, 6, 12]\nnegative divisors of 12: [-1, -2, -3, -4, -6, -12]\ntotal divisors count: 12\n\nanswer extracted: 12\n\nsince our answer extracted matches the given ground truth answer, the final result is:\n\nthe answer is correct.",
+  "is_correct": true,
+  "trial": -1
+}

test/test_files/agenteval-in-out/agentchat_results/prealgebra/10.json

@@ -0,0 +1,32 @@
+{
+  "problem": "How many integers $n$ satisfy both of the inequalities $4n + 3 < 25$ and $-7n + 5 < 24$?",
+  "level": "Level 5",
+  "type": "Prealgebra",
+  "solution": "Subtract 3 and divide by 4 on both sides of the first inequality to obtain \\begin{align*}\n4n + 3 &< 25 \\\\\n\\Rightarrow\\qquad 4n &< 22 \\\\\n\\Rightarrow\\qquad n &< 5.5.\n\\end{align*}Similarly, the second inequality yields \\begin{align*}\n-7n + 5 &< 24 \\\\\n\\Rightarrow\\qquad -7n &< 19 \\\\\n\\Rightarrow\\qquad n &> -\\frac{19}{7}.\n\\end{align*}Therefore, we are looking for all the integers between $-\\frac{19}{7}$ and $5.5$. Since $-\\frac{19}{7}$ is between $-3$ and $-2$ and the largest integer less than $5.5$ is 5, we need to count the number of integers between $-2$ and $5$, inclusive. There are $5$ positive integers, $2$ negative integers, and zero, so there are $\\boxed{8}$ integers that satisfy both $4n + 3 < 25$ and $-7n + 5 < 24$.",
+  "correct_ans": "8",
+  "problem_id": "10",
+  "response_with_ans": "Here's the code to solve the inequalities and find the intersection of their solutions:\n\n```python\nfrom sympy import symbols, Eq, solve\n\nn = symbols(\"n\")\ninequality_1 = 4 * n + 3 < 25\ninequality_2 = -7 * n + 5 < 24\n\nsolution_1 = solve(inequality_1, n)\nsolution_2 = solve(inequality_2, n)\n\nintersection = (max(solution_1[0], solution_2[0]), min(solution_1[1], solution_2[1]))\n\nprint(f\"Solution to inequality 1: {solution_1}\")\nprint(f\"Solution to inequality 2: {solution_2}\")\nprint(f\"Intersection of solutions: {intersection}\")\n```\n\nExecute this code, and let's see the solutions for both inequalities and their intersection.",
+  "round": 1,
+  "messages": [
+    {
+      "content": "How many integers $n$ satisfy both of the inequalities $4n + 3 < 25$ and $-7n + 5 < 24$?",
+      "role": "user"
+    },
+    {
+      "content": "To find the number of integers $n$ that satisfy both inequalities, we'll first solve each inequality individually, and then find the intersection of the solutions.\n\nStep 1: Solve the inequalities\n1. $4n + 3 < 25$\n2. $-7n + 5 < 24$\n\nStep 2: Find the intersection of the solutions\n\nStep 3: Count the number of integers in the intersection\n\nFirst, let's solve the inequalities using the python code.",
+      "role": "assistant"
+    },
+    {
+      "content": "",
+      "role": "user"
+    },
+    {
+      "content": "Here's the code to solve the inequalities and find the intersection of their solutions:\n\n```python\nfrom sympy import symbols, Eq, solve\n\nn = symbols(\"n\")\ninequality_1 = 4 * n + 3 < 25\ninequality_2 = -7 * n + 5 < 24\n\nsolution_1 = solve(inequality_1, n)\nsolution_2 = solve(inequality_2, n)\n\nintersection = (max(solution_1[0], solution_2[0]), min(solution_1[1], solution_2[1]))\n\nprint(f\"Solution to inequality 1: {solution_1}\")\nprint(f\"Solution to inequality 2: {solution_2}\")\nprint(f\"Intersection of solutions: {intersection}\")\n```\n\nExecute this code, and let's see the solutions for both inequalities and their intersection.",
+      "role": "assistant"
+    }
+  ],
+  "time": 19.949471950531006,
+  "check_result": "the answer is 8.\n\nthe answer is correct.",
+  "is_correct": true,
+  "trial": -1
+}

test/test_files/agenteval-in-out/agentchat_results/prealgebra/11.json

@@ -0,0 +1,24 @@
+{
+  "problem": "Bekah has exactly three brass house number digits: 2, 3 and 5. How many distinct numbers can she form using one or more of the digits?",
+  "level": "Level 5",
+  "type": "Prealgebra",
+  "solution": "If Bekah uses only one digit, she can form three numbers. If she uses two digits, she has three choices for the tens place and two for the units, so she can form six numbers. Finally, if Bekah uses all three digits, she has three choices for the hundreds place, two for the tens, and one for the units, so she can form six numbers. Thus, Bekah can form $3 + 6 + 6 = \\boxed{15}$ distinct numbers.",
+  "correct_ans": "15",
+  "problem_id": "11",
+  "response_with_ans": "To find the answer, we'll consider all possible combinations that Bekah can make using the three digits she has:\n1. Single-digit numbers: 2, 3, and 5\n2. Two-digits numbers using the digits: 23, 25, 32, 35, 52, and 53\n3. Three-digits number using all the digits: 235, 253, 325, 352, 523, and 532\n\nLet's calculate the number of distinct numbers Bekah can form using one or more of the digits with Python code:\n\n```python\nfrom itertools import permutations\n\ndigits = [2, 3, 5]\n\n# Single-digit numbers\none_digit_numbers = set(digits)\n\n# Two-digits numbers\ntwo_digit_numbers = set(int(''.join(map(str, p))) for p in permutations(digits, 2))\n\n# Three-digits numbers\nthree_digit_numbers = set(int(''.join(map(str, p))) for p in permutations(digits, 3))\n\ndistinct_numbers = one_digit_numbers | two_digit_numbers | three_digit_numbers\ntotal_distinct_numbers = len(distinct_numbers)\n\nprint(\"Distinct numbers:\", distinct_numbers)\nprint(\"Total distinct numbers:\", total_distinct_numbers)\n```\n\nPlease run the above Python code to calculate the total number of distinct numbers Bekah can form using one or more of the digits.",
+  "round": 0,
+  "messages": [
+    {
+      "content": "Bekah has exactly three brass house number digits: 2, 3 and 5. How many distinct numbers can she form using one or more of the digits?",
+      "role": "user"
+    },
+    {
+      "content": "To find the answer, we'll consider all possible combinations that Bekah can make using the three digits she has:\n1. Single-digit numbers: 2, 3, and 5\n2. Two-digits numbers using the digits: 23, 25, 32, 35, 52, and 53\n3. Three-digits number using all the digits: 235, 253, 325, 352, 523, and 532\n\nLet's calculate the number of distinct numbers Bekah can form using one or more of the digits with Python code:\n\n```python\nfrom itertools import permutations\n\ndigits = [2, 3, 5]\n\n# Single-digit numbers\none_digit_numbers = set(digits)\n\n# Two-digits numbers\ntwo_digit_numbers = set(int(''.join(map(str, p))) for p in permutations(digits, 2))\n\n# Three-digits numbers\nthree_digit_numbers = set(int(''.join(map(str, p))) for p in permutations(digits, 3))\n\ndistinct_numbers = one_digit_numbers | two_digit_numbers | three_digit_numbers\ntotal_distinct_numbers = len(distinct_numbers)\n\nprint(\"Distinct numbers:\", distinct_numbers)\nprint(\"Total distinct numbers:\", total_distinct_numbers)\n```\n\nPlease run the above Python code to calculate the total number of distinct numbers Bekah can form using one or more of the digits.",
+      "role": "assistant"
+    }
+  ],
+  "time": 15.620970249176025,
+  "check_result": "the answer is 15. please run the provided python code to verify the result:\n\n```python\nfrom itertools import permutations\n\ndigits = [2, 3, 5]\n\n# single-digit numbers\none_digit_numbers = set(digits)\n\n# two-digits numbers\ntwo_digit_numbers = set(int(''.join(map(str, p))) for p in permutations(digits, 2))\n\n# three-digits numbers\nthree_digit_numbers = set(int(''.join(map(str, p))) for p in permutations(digits, 3))\n\ndistinct_numbers = one_digit_numbers | two_digit_numbers | three_digit_numbers\ntotal_distinct_numbers = len(distinct_numbers)\n\nprint(\"distinct numbers:\", distinct_numbers)\nprint(\"total distinct numbers:\", total_distinct_numbers)\n```\n\nthe answer is correct.",
+  "is_correct": true,
+  "trial": -1
+}

test/test_files/agenteval-in-out/agentchat_results/prealgebra/12.json

@@ -0,0 +1,24 @@
+{
+  "problem": "Two numbers are said to be 'relatively prime' if their greatest common factor is 1. How many integers greater than 10 and less than 30 are relatively prime with 28?",
+  "level": "Level 5",
+  "type": "Prealgebra",
+  "solution": "Since $28=2^2\\cdot 7$, a positive integer is relatively prime with $28$ if and only if it contains neither $2$ nor $7$ in its prime factorization. In other words, we want to count the number of integers between $11$ and $29$ inclusive which are divisible by neither $2$ nor $7$.\n\nAll of the odd numbers are not divisible by 2; there are 10 such numbers.  The only one of these that is divisible by 7 is 21, so there are $10- 1 =\\boxed{9}$ numbers between 10 and 30 that are relatively prime with 28.",
+  "correct_ans": "9",
+  "problem_id": "12",
+  "response_with_ans": "I will find the relatively prime numbers with 28 in the range greater than 10 and less than 30 using Python code. I will use the math.gcd function and a loop to check each number in the specified range.\n\n```python\nimport math\n\ncount = 0\nrelatively_prime_numbers = []\nfor number in range(11, 30):\n    if math.gcd(28, number) == 1:\n        relatively_prime_numbers.append(number)\n        count += 1\n\nprint(f\"Relatively prime numbers with 28: {relatively_prime_numbers}\")\nprint(f\"Number of relatively prime numbers with 28: {count}\")\n```\nExecute the above code to find the integers relatively prime with 28 and their count.",
+  "round": 0,
+  "messages": [
+    {
+      "content": "Two numbers are said to be 'relatively prime' if their greatest common factor is 1. How many integers greater than 10 and less than 30 are relatively prime with 28?",
+      "role": "user"
+    },
+    {
+      "content": "I will find the relatively prime numbers with 28 in the range greater than 10 and less than 30 using Python code. I will use the math.gcd function and a loop to check each number in the specified range.\n\n```python\nimport math\n\ncount = 0\nrelatively_prime_numbers = []\nfor number in range(11, 30):\n    if math.gcd(28, number) == 1:\n        relatively_prime_numbers.append(number)\n        count += 1\n\nprint(f\"Relatively prime numbers with 28: {relatively_prime_numbers}\")\nprint(f\"Number of relatively prime numbers with 28: {count}\")\n```\nExecute the above code to find the integers relatively prime with 28 and their count.",
+      "role": "assistant"
+    }
+  ],
+  "time": 6.9820802211761475,
+  "check_result": "the answer is correct.",
+  "is_correct": true,
+  "trial": -1
+}