Conversational continuity

[jalsco]

Conversational continuity works great in this plugin. This question is more about understanding how you do it, and how the LLM does it, so I can adapt what I do....

It is a bit confusing for me, I have no idea how you actually make it so in your code. When I exercise th API myself each prompt starts afresh - absolutely no knowledge of what has gone before (no matter what it claims). I thought (clearly wrongly) that the API KEY was enough to have the LLM keep a memory of previous conversations. So I thought (wrongly) that I could manage a few different conversations concurrently simply by having a few different keys. Well that's unworkable... And, anyway OpenAI-sublime-text is my preferred way.

What I would like is a few different tabs open (not just one called AIChat) , and each one with different conversation or topic - that is being remembered. I am not sure how to do this using this plugin - or if it is even possible . Any thoughts on how I should adapt my USAGE to come closest to this. For example - each MODEL - to output to a different tab and independently keep its own conversational memory. Then I would have one tab as a discussion tab and another tab as code-only.

About "reset chat history" - I know it clears all the contents in the Tab. I assume it also resets all memory of the conversation with the LLM, so the LLM forgets everything from that conversation also ? Any more details on what that does would be appreciated. Again I am just trying to understand to adjust my usage. At first I just let it run (all in same tab) for a few thousand lines. IS that problematic ? Should it be reset from time to time ? What about letting it run indefinitely into the same tab ? It is just a file, right ?, so it is only the usual issues about Sublime handling it and memory usage, right ? OR is some of that file being sent each time (with the prompt) to the LLM ? Some google searches say programmer has to manage the sessions themselves - so I really don't know how to think through this...

[yaroslavyaroslav]

Wow that's a lot of questions you're arising here. While not of them I'm able to cover during the short response, but few things I definitely capable to elaborate.

1. The history stored local in kinda json file and each iteration it just gets appended by a new message. The whole this history is sent on each of your request to llm. So the longer you're chatting with llm the bigger request you're sending. It stores like [user, assistant, user, assistant] array after all.
2. You can have separate history per project setup. To not overwhelm the answer let me guide you to the main readme "history" section. Applying that on your side will lead to having isolated llm setup (picked assistant, history, token counts) for a particular project.
3. Llm has limitations on their side of how long request you can send to it, most of modern services are about to have it on 128k tokens overall. It's quite a lot so you unlikely will hit that bound in ordinary usage, but it's still worth to be aware of that limitation.

[jalsco]

I am sorry for asking so many questions, I will tone that back..
But thanks for your great answer 🙏🏻😅

I knew I was doing something wrong in my usage - now I know what/how/why.

I will go forward using Projects in Sublime-Text much more than i have been, with one AiChat tab per Project. I will also "reset history" whenever its a topic change, within each Project... rather than just continuing on for thousands of lines, across multiple topics.

That will help alot.

[yaroslavyaroslav]

Nah, feel free to asking all the kind of questions are arising on your side. It's just that I'd skip those of them that are far beyond of the scope of the plugin (like the in depth knowledge of how llm implanted).

However to highlight you and all the rest of the folks one approach to utilise this plugin at its best I'd like to add the following:

Personally I found that using the plugin with either llama3.370b or gpt-4o the most thing that is model is missing to help me during my session is the context of the project, and the results of weather its suggestion succeeded or not. So I personally use the following pipeline:

1. To add whole files that I'm working on rn to the model through the sheet
2. To ask to solve some task on them
3. To paste the suggested solution into the files
4. To build the solution (and pass build errors if occurred)
5. To run tests on the files (and pass failing tests output if occurred)
6. Repeat it in the loop.

All but 1 point happening within the phantom i.e. non-persistent mode because it's just doesn't improve the helpfulness of the model in some observable way. The other thing to mention is that in current develop branch there's code that makes the provided pipeline is a way more convenient, but it's far from being ready for release yet.

[jalsco]

Looks likely I will "adopt" 😁 a pre-loved mac mini M4 with 24gb ram (only 256gb of disk) in next week or so. My plan is to install one of the MLX models on it:

https://huggingface.co/models?library=mlx

Even using MLX the Llama 3.3 70 Billion is probably too big for that machine. But I will try to see if it runs: Probably this one: https://huggingface.co/mlx-community/Llama-3.3-70B-Instruct-8bit

My reading of that page is model was converted to use MLX and then shrunk to 8-bit, which reduces it to only 19.8B params - so maybe it will run for me. If not I will go back to a smaller 3.2 model or a coding specific model from that MLX page.

Note: XAi's Grok improved DRAMATICALLY a few days ago. Only since 12 Dec 2024, grok-2-1212 has been available via API. The coding skill level improved significantly. I fed it some quite basic PY challenges that grok-beta couldn't get right, and it had no problems. Elon keeps saying Grok3 will exceed everything else out there, and be released in Dec 2024. If that happens maybe I will use nothing else. Wait and see...

I have not yet plugin tried with sheets but will be doing that soon. Thank you so much for your guidance. I will start using similar pipeline as you suggested above ✅😊