This model's maximum context length is 8191 tokens, however you requested 21485 tokens (21485 in your prompt; 0 for the completion). Please reduce your prompt; or completion length.

[7fromheaven]

Trying to get AutoGPT to read a .csv file within the auto_gpt_workspace path and receiving the following error:

openai.error.InvalidRequestError: This model's maximum context length is 8191 tokens, however you requested 21485 tokens (21485 in your prompt; 0 for the completion). Please reduce your prompt; or completion length.

Can I edit anything in the .env file to surpass the token limit? I have pinecone set up but my knowledge is limited. Apologies if this has already been solved. I'm new to coding and to Github.

[Boostrix]

You will want to split up the csv file into sections that the llm can handle - make that 5-8 parts to be on the safe side and still have space for the prompt itself.
Basically, split up the file so that it's treated like a handful of separate files (for now).

For starters, consider coming up with a challenge based on this, so that this can be added as a benchmark.
Get in touch with @merwanehamadi for details, or search the issue tracker for topics labelled "challenge":

• Help us build challenges! #3835

PS: And check out: #3031

[dereckmezquita]

I had a similar error but this is not from reading one file but from having it browse the internet. I want to search the internet and collect information for me into separate files and save those locally.

SYSTEM:  Command write_to_file returned: File written to successfully.
Traceback (most recent call last):
  File "/Users/DMezquita/.pyenv/versions/3.10.0/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/Users/DMezquita/.pyenv/versions/3.10.0/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/Users/DMezquita/Coding/Auto-GPT-0.3.1/autogpt/__main__.py", line 5, in <module>
    autogpt.cli.main()
  File "/Users/DMezquita/.pyenv/versions/3.10.0/lib/python3.10/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/Users/DMezquita/.pyenv/versions/3.10.0/lib/python3.10/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/Users/DMezquita/.pyenv/versions/3.10.0/lib/python3.10/site-packages/click/core.py", line 1635, in invoke
    rv = super().invoke(ctx)
  File "/Users/DMezquita/.pyenv/versions/3.10.0/lib/python3.10/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/DMezquita/.pyenv/versions/3.10.0/lib/python3.10/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/Users/DMezquita/.pyenv/versions/3.10.0/lib/python3.10/site-packages/click/decorators.py", line 26, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/Users/DMezquita/Coding/Auto-GPT-0.3.1/autogpt/cli.py", line 90, in main
    run_auto_gpt(
  File "/Users/DMezquita/Coding/Auto-GPT-0.3.1/autogpt/main.py", line 186, in run_auto_gpt
    agent.start_interaction_loop()
  File "/Users/DMezquita/Coding/Auto-GPT-0.3.1/autogpt/agent/agent.py", line 113, in start_interaction_loop
    assistant_reply = chat_with_ai(
  File "/Users/DMezquita/Coding/Auto-GPT-0.3.1/autogpt/llm/chat.py", line 165, in chat_with_ai
    agent.summary_memory = update_running_summary(
  File "/Users/DMezquita/Coding/Auto-GPT-0.3.1/autogpt/memory_management/summary_memory.py", line 128, in update_running_summary
    current_memory = create_chat_completion(messages, cfg.fast_llm_model)
  File "/Users/DMezquita/Coding/Auto-GPT-0.3.1/autogpt/llm/llm_utils.py", line 166, in create_chat_completion
    response = api_manager.create_chat_completion(
  File "/Users/DMezquita/Coding/Auto-GPT-0.3.1/autogpt/llm/api_manager.py", line 55, in create_chat_completion
    response = openai.ChatCompletion.create(
  File "/Users/DMezquita/.pyenv/versions/3.10.0/lib/python3.10/site-packages/openai/api_resources/chat_completion.py", line 25, in create
    return super().create(*args, **kwargs)
  File "/Users/DMezquita/.pyenv/versions/3.10.0/lib/python3.10/site-packages/openai/api_resources/abstract/engine_api_resource.py", line 153, in create
    response, _, api_key = requestor.request(
  File "/Users/DMezquita/.pyenv/versions/3.10.0/lib/python3.10/site-packages/openai/api_requestor.py", line 226, in request
    resp, got_stream = self._interpret_response(result, stream)
  File "/Users/DMezquita/.pyenv/versions/3.10.0/lib/python3.10/site-packages/openai/api_requestor.py", line 619, in _interpret_response
    self._interpret_response_line(
  File "/Users/DMezquita/.pyenv/versions/3.10.0/lib/python3.10/site-packages/openai/api_requestor.py", line 682, in _interpret_response_line
    raise self.handle_error_response(
openai.error.InvalidRequestError: This model's maximum context length is 8192 tokens. However, your messages resulted in 8695 tokens. Please reduce the length of the messages.
Press any key to continue...

[Boostrix]

marked as challenge and talked to @merwanehamadi about capturing this via corresponding challenge coverage.
closing for now

[waynehamadi]

#3835
we're going to ask for help for this one: we don't need the problem solved yet, but the problem to be written as a challenge first, so we solve it soon and forever.

[Boostrix]

we could gather some massive CSV data sets for this one to get people started with the data:

• https://www.stats.govt.nz/large-datasets/csv-files-for-download/
• https://data.world/datasets/csv

[campainer]

I am working on a chatbot with personal data.
what can be the best way to split the data
currently I am using UnstructuredMarkdownLoader() to load the data
but If my file is very big making the query request to llm can give size error
"This model's maximum context length is 8192 tokens. However, your messages resulted in 8427 tokens."
can any help me to find the best splitter