feat: stream LLM responses

[jamadeo]

Only databricks provider to start, but should not be hard to do this for others that support streaming.

Desktop:

streaming-gui.mov

CLI:

cli.mov

[jamadeo]

This almost certainly isn't the best way to do this, but without it I could not get new messages to trigger a re-render. I could use a react expert's help here :)

[jamadeo]

@zanesq any ideas here? I want to re-render every time we add a chunk, and so far the only thing that made it work was adding this fake state to force it.

[zanesq]

@jamadeo I think the best way is to use an immutable approach instead something like this. Instead of mutating in place, always create a new array/object when updating state. React should detect the change and re-render automatically.

Instead of

const lastMessage = currentMessages[currentMessages.length - 1];
lastMessage.content = [...lastMessage.content, ...newMessage.content];
forceUpdate();

try

currentMessages = [
  ...currentMessages.slice(0, -1),
  {
    ...currentMessages[currentMessages.length - 1],
    content: [
      ...currentMessages[currentMessages.length - 1].content,
      ...newMessage.content,
    ],
  },
];

[michaelneale]

@jamadeo looks really great - but there is a bug where it seems to not call tools correctly
so may need to chase that down

[jamadeo]

Thanks @baxen for the review! I actually effectively reverted the subagent change because it doesn't really help anything to use streaming in that context. If we, in the future, want to show partial text responses from models to subagents in a streamed way we can think about it then, but no reason to fit it in now.

[baxen]

Amazing!

[michaelneale]

@jamadeo oops thought it would be an easier merge than that to update it, but looking good. Sorry have slightly broken it (looking at it) - feel free to yank that last merge if it is all broken and not fixed by your morning time.

[michaelneale]

I think some work still needed:

using databricks, when it finishes streaming never goes back to the prompt (even if I press enter)

[michaelneale]

have been testing this with CLI + databricks and seems ✅