Updated ROADMAP.md

benchmarks/simple.py

@@ -0,0 +1,18 @@
+"""
+This is a sketch of a simple benchmark runner.
+"""
+
+tasks = [
+    {
+        "question": "",
+        "answer": "",
+    },
+    {"setup_script": "", "question": "", "answer": "", "evaluation_script": ""},
+]
+
+# For each task,
+# Start a thread that does the following:
+# Spin up a docker container
+# Run the setup script
+# Ask the question
+# Run the evaluation script or use an LLM to check the answer

docs/ROADMAP.md

@@ -9,6 +9,9 @@
 - [ ] Figure out how to get OI to answer to user input requests like python's `input()`. Do we somehow detect a delay in the output..? Is there some universal flag that TUIs emit when they expect user input? Should we do this semantically with embeddings, then ask OI to review it and respond..?
 - [ ] Placeholder text that gives a compelling example OI request. Probably use `textual`
 - [ ] Everything else `textual` offers, like could we make it easier to select text? Copy paste in and out? Code editing interface?
+- [ ] Let people turn off the active line highlighting
+- [ ] Add a --plain flag which doesn't use rich, just prints stuff in plain text
+- [ ] Use iPython stuff to track the active line, instead of inserting print statements, which makes debugging weird (From ChatGPT: For deeper insights into what's happening behind the scenes, including which line of code is being executed, you can increase the logging level of the IPython kernel. You can configure the kernel's logger to a more verbose setting, which logs each execution request. However, this requires modifying the kernel's startup settings, which might involve changing logging configurations in the IPython kernel source or when launching the kernel.)
 - [ ] Let people edit the code OI writes. Could just open it in the user's preferred editor. Simple. [Full description of how to implement this here.](https://github.com/KillianLucas/open-interpreter/pull/830#issuecomment-1854989795)
 - [ ] Display images in the terminal interface
 - [ ] There should be a function that just renders messages to the terminal, so we can revive conversation navigator, and let people look at their conversations

pyproject.toml

@@ -80,6 +80,7 @@ build-backend = "poetry.core.masonry.api"
 [tool.poetry.scripts]
 interpreter = "interpreter.terminal_interface.start_terminal_interface:main"
 i = "interpreter.terminal_interface.start_terminal_interface:main"
+ibench = "benchmarks.simple:main"
 
 [tool.black]
 target-version = ['py311']