NewVersion: Fixes & Improvements (e.g. Cost Assessment&Tracking, Azure, +more)

[drudilorenzo]

Since the project is no longer officially supported, I created a fork in which I inserted numerous fixes and improvements.

For more info see the repo

Feel free to contribute & Star the repo to ask for faster development :) -> repo

New Features:

• Easy configuration + Azure support
• Cost tracking using openai-cost-logger
• Set cost upperbound and stop the experiment when it is reached
• New models and OpenAI API support
• Cost assessment (link)
• Added the simulation skip-morning-s-14 that starts at ~8am and permits to see actions and interactions earlier.
• Added zoom-in/zoom-out using Z/X keys
• Powerful automated script for enhanced simulation performance (see repo readme).

Fixes:

• Resolves Klaus Muller persona.scratch.importance_trigger_curr:: 141 150 #93
• Resolves Azure Openai #61
• Resolves FileNotFoundError: [Errno 2] No such file or directory: '../../environment/frontend_server/storage/test1/movement/0.json' #40, Resolves movement folder issue #44, Resolves No such file or directory: '../../environment/frontend_server/storage/test-simulation/movement/0.json' #59
• Resolves Remove clustrmaps spyware/analytics or make it opt-in #50
• Resolves Gpt 3.5/4 - Cost Assessment #150, Resolves anyone has a estimate on the cost per hour? #56
• Resolves Start a day at 6am so something is happening sooner #26
• Resolves zoom in or out map  #122
• Resolves Whenever run xxx ends, appears IndexError: list index out of range #111

[dafrontman]

Issue #111 fix
I figured this one out. It's just a string parsing issue. The duration variable in line 378 gets messed up because the code earlier in the function is expecting the gpt_response to contain the string "duration in minutes:" but when the error happens the prompt has a string that reads "duration:" only. So the code fails after that because "in minutes" is not there. I just wrote a quick fix to get it working. Checked that "duration in minutes:" was there and if not I parsed it with just "duration:"

  if "duration in minutes" in i: 
    k = [j.strip() for j in i.split("(duration in minutes:")]
  else:
    k = [j.strip() for j in i.split("(duration:")]

Thanks, nice catch! I'll try that

There was yet another case where the gpt_response had an entry that did not have a "duration" AT ALL. It just contained the task. So I modified the code a bit more. My fix is SUPER ugly. Hopefully someone will write a cleaner code block than me but here's what I did. Not tested yet:

  if "duration in minutes" in i: 
    k = [j.strip() for j in i.split("(duration in minutes:")]
  elif "duration" in i:
    k = [j.strip() for j in i.split("(duration:")]
  else:
    task = i
    duration = 0 #arbitrary value, don't know if 0 minutes will hurt anything
  if "duration" in i:
    task = k[0]
    if task[-1] == ".": 
      task = task[:-1]
    duration = int(k[1].split(",")[0].strip())
  cr += [[task, duration]]

[drudilorenzo]

Issue #111 fix
I figured this one out. It's just a string parsing issue. The duration variable in line 378 gets messed up because the code earlier in the function is expecting the gpt_response to contain the string "duration in minutes:" but when the error happens the prompt has a string that reads "duration:" only. So the code fails after that because "in minutes" is not there. I just wrote a quick fix to get it working. Checked that "duration in minutes:" was there and if not I parsed it with just "duration:"

  if "duration in minutes" in i: 
    k = [j.strip() for j in i.split("(duration in minutes:")]
  else:
    k = [j.strip() for j in i.split("(duration:")]

Thanks, nice catch! I'll try that

There was yet another case where the gpt_response had an entry that did not have a "duration" AT ALL. It just contained the task. So I modified the code a bit more. My fix is SUPER ugly. Hopefully someone will write a cleaner code block than me but here's what I did. Not tested yet:

  if "duration in minutes" in i: 
    k = [j.strip() for j in i.split("(duration in minutes:")]
  elif "duration" in i:
    k = [j.strip() for j in i.split("(duration:")]
  else:
    task = i
    duration = 0 #arbitrary value, don't know if 0 minutes will hurt anything
  if "duration" in i:
    task = k[0]
    if task[-1] == ".": 
      task = task[:-1]
    duration = int(k[1].split(",")[0].strip())
  cr += [[task, duration]]

I am concerned that it relies entirely on the model answer. If this is the case, we can't be certain that the answer will always be in the requested format.

To address this, I am developing a script that can resume the simulation from the previous step in case of failures. You can find it in my repository under the name run_backend_automatic. I will write a short guide on how to use it in the next few days (it contains several features that can be useful for everyone, e.g. execution w/o UI - save checkpoints every n steps - resume run in case of failures).

[dafrontman]

Issue #156

Execution time slowwwwww for n25.

[drudilorenzo]

Issue #156

Execution time slowwwwww for n25.

Yeah, don't know how to solve it.
You are probably hitting the rate limit, right now is very slow also for me :(

I did a 15k steps run and it took ~41hrs if I remember well (but I have to run another long run in the next days, I'll keep you updated with the exact time).

[drudilorenzo]

Hey guys, I have finally added the automated script for execution. You can find a quick description in my repo readme.

That includes lots of interesting features:

• Automatic Saving: The simulation automatically saves progress every 200 steps, ensuring you never lose data.
• Error Recovery: In the event of an error, the simulation automatically resumes by stepping back and restarting from the last successful point. This is crucial as the model relies on formatted answers, which can sometimes cause exceptions.
• Automatic Tab Opening: A new browser tab will automatically open when necessary.
• Headless Mode: The scripts support running simulations in Chrome's headless mode, enabling execution on a server without a UI (it needs headless-chrome installed.
• Configurable Port Number: You can configure the port number as needed.

For more details look to the repo readme and feel free to ask questions here :)

p.s. You do need to run also the frontend server but you don't have to take care of opening a browser tab.

[austinmw]

Hi @drudilorenzo, thanks for your fork!

I run into the same error with your fork as I do with the original repository:
#97 (comment)

It seems others are experiencing this error as well. Any idea how to solve it? I followed your install steps and am running on an M2 MBP.

[dafrontman]

Hi @drudilorenzo, thanks for your fork!

I run into the same error with your fork as I do with the original repository: #97 (comment)

It seems others are experiencing this error as well. Any idea how to solve it? I followed your install steps and am running on an M2 MBP.

I suspect this is an error having to do with which directory you are executing reverie.py i.e. the backend server from. It has to match up with where the files are located for the simulation. So wherever base_the_ville_isabella_maria_klaus is located then its relative path must be reachable from wherever you ran reverie.py. At least that's what happened to me when I was messing around with using a batch file to run things. I ran the batch file in a different location from the source code and it couldn't fine the simulation files.

[austinmw]

@dafrontman Hmmm, thanks for your advice. I did the following:

terminal 1:

cd environment/frontend_server
python manage.py runserver

terminal 2:

cd reverie/backend_server
python reverie.py

Did you run them from different locations?

[dafrontman]

@dafrontman Hmmm, thanks for your advice. I did the following:

terminal 1:

cd environment/frontend_server
python manage.py runserver

terminal 2:

cd reverie/backend_server
python reverie.py

Did you run them from different locations?

No, those are the right locations, but what I did that caused the error message to come from the server is I tried to automate my run process by putting that stuff in a batch file. When I did it that way, because I ran the batch file from a different directory, I think python saw my execution path as different from "reverie/backend_server" so when it tried to grab the simulation data and it was using relative paths it couldn't find it because I launched from the wrong path.

[drudilorenzo]

Hi @drudilorenzo, thanks for your fork!

I run into the same error with your fork as I do with the original repository: #97 (comment)

It seems others are experiencing this error as well. Any idea how to solve it? I followed your install steps and am running on an M2 MBP.

Hi!

Unfortunately, I have never encountered the error you mentioned (I am using Ubuntu and never directly tried the project on MacOS).

Here are a few questions to better understand your setup:

1. Do you always see the message Please start the backend first or do you see it only when you refresh the page?
2. Are you using the conda env specified?
3. Which browser are you using?

The correct procedure should be:

Step 1:

Activate the conda env.
Create all the files required (utils.py in the case of the original repo, open_config.json in my fork).

Step 2:

cd environment/frontend_server
python manage.py runserver

Step 3:

In another terminal:

cd reverie/backend_server
python reverie.py

Step 4:

Open #http://127.0.0.1:8000/simulator_home

4. If you haven't done so yet, try the steps above both on the original repo and also on my fork.

Let me know if it works!

[austinmw]

Hey, thanks for your reply. Yeah, those are exactly the steps I've done. I also tried Ubuntu from an AWS EC2 instance and got the same result. I've tried Arc browser and standard chrome browser.

I'm able to see map page after starting the backend and first browsing tohttp://127.0.0.1:8000/simulator_home, but it doesn't show the current time and it doesn't update when I enter a run command. Also when I refresh the page it shows that error, which seems to come from this file:

generative_agents/environment/frontend_server/templates/home/error_start_backend.html

Line 4 in fe05a71

{% block content %}

Here's what it looks like:

[drudilorenzo]

@austinmw

The fact that it shows the error when you refresh the page is normal. I think that a possible reason is that, if you refresh the page, to be able to continue from the point you were before, there should be some mechanism able to read the previous state and reset everything as it was (just a guess, but don't focus on this. It is something that shouldn't affect the simulation).

What is more strange is that you are not seeing any update at all. Let's debug it using base_the_ville_isabella_maria_klaus. A run of 10 steps should take ~5 min (and keep in mind that in the first 2/3 minutes you won't see the time and any update because it will do the setup phase). But after 5/6 mins it should be finished.

Btw, which models are you using? Both chat completion and embeddings (All my experiments are done using gpt-35-turbo-0125 and text-embedding-3-small from Azure).

[austinmw]

Ah, didn't realize that that page when refreshing is normal!

After waiting a few minutes without touching anything I finally see it moving! I guess I was too impatient 😅

Thanks for your help!

[devpramod]

@drudilorenzo I am still facing an issue mentioned in #93
The simulation gets stuck at the Ville:Dorm for Oak Hill College:Klaus Mueller's room:bed
Do you have any suggestions? Thanks

[drudilorenzo]

@drudilorenzo I am still facing an issue mentioned in #93 The simulation gets stuck at the Ville:Dorm for Oak Hill College:Klaus Mueller's room:bed Do you have any suggestions? Thanks

1. How many steps are you running?
2. Which config file are you using?
3. Have you tried to wait as suggested above?

[zhangpeihaoks]

hey，when i try to run run_backend sh ，it have some error

Running backend server at: http://127.0.0.1:8000/simulator_home
Timestamp: 2024-09-12_17-10-08
Traceback (most recent call last):
  File "D:\codebase\generative_agents2\reverie\backend_server\reverie.py", line 37, in <module>
    from persona.persona import *
  File "D:\codebase\generative_agents2\reverie\backend_server\persona\persona.py", line 23, in <module>
    from persona.cognitive_modules.perceive import *
  File "D:\codebase\generative_agents2\reverie\backend_server\persona\cognitive_modules\perceive.py", line 12, in <module>
    from persona.prompt_template.gpt_structure import *
  File "D:\codebase\generative_agents2\reverie\backend_server\persona\prompt_template\gpt_structure.py", line 67, in <module>
    cost_logger = OpenAICostLogger_Singleton(
  File "D:\codebase\generative_agents2\reverie\backend_server\persona\prompt_template\openai_logger_singleton.py", line 16, in __call__
    instance = super().__call__(*args, **kwargs)
  File "D:\codebase\generative_agents2\reverie\backend_server\persona\prompt_template\openai_logger_singleton.py", line 33, in __init__
    self.__cost_logger = OpenAICostLogger(
  File "C:\Users\zhangpeihao\.conda\envs\generative_agents\lib\site-packages\openai_cost_logger\openai_cost_logger.py", line 43, in __init__
    self.__build_log_file()
  File "C:\Users\zhangpeihao\.conda\envs\generative_agents\lib\site-packages\openai_cost_logger\openai_cost_logger.py", line 145, in __build_log_file
    with open(self.filepath, 'w') as file:
OSError: [Errno 22] Invalid argument: 'cost-logs\\simulacra-test_2024-09-12_17:10:09.json'
imported openai-cost-logger

[zhangpeihaoks]

the openai_config.json need api_base environment,in the cn mainland must use proxy to use openai llm，please feature it

[chowington]

Thanks for your work, @drudilorenzo! My team and I have been iterating on your repo on our own fork, and have further improved this codebase based on your work.

Here is the PR which describes all the major updates we've made, in case anyone else interested: drudilorenzo#8