Add support for streaming responses for Amazon Bedrock LLM by Benyuel · Pull Request #2367 · langchain-ai/langchainjs

Ben Pleasanton (Benyuel) · 2023-08-22T17:17:20Z

This PR adds support for using the Amazon Bedrock LLM with streaming. See the added test for an example use.

vercel · 2023-08-22T17:17:25Z

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Updated (UTC)
langchainjs-docs	✅ Ready (Inspect)	Visit Preview	Aug 24, 2023 11:16am

dosubot · 2023-08-22T17:19:32Z

@@ -590,9 +593,11 @@
    "@aws-sdk/client-sagemaker-runtime": "^3.310.0",


Great work on the PR! I've noticed that this diff introduces two new dependencies, "@aws-sdk/eventstream-marshaller" and "@aws-sdk/util-utf8-universal", which may be peer/dev/hard dependencies. I'm flagging this for maintainers to review.

dosubot · 2023-08-22T17:19:34Z

@@ -0,0 +1,267 @@
+import { SignatureV4 } from "@aws-sdk/signature-v4";


This comment is flagging the addition of a new HTTP request using the fetch function in the BedrockChat class, and it should be reviewed by maintainers.

dosubot · 2023-08-22T17:19:36Z

@@ -0,0 +1,267 @@
+import { SignatureV4 } from "@aws-sdk/signature-v4";


This PR adds a dependency on an environment variable AWS_DEFAULT_REGION via the getEnvironmentVariable function on line 97. Please review this change to ensure it is handled correctly.

Jacob Lee (jacoblee93) · 2023-08-23T02:48:53Z

This looks really fantastic, but I don't think it really fits as a chat model (there's no dialog turns/messages, for example). The OpenAIChat class was really just a one-off when chat models were new.

We can just add streaming to the LLM version I think?

Jacob Lee (jacoblee93) · 2023-08-23T02:56:24Z

+      );
+    }
+
+    if (this.streaming) {


We now have a new interface for .stream():

https://js.langchain.com/docs/guides/expression_language/interface

Also, it'd be more elegant to always use the streaming call and implement non-streaming requests as just concatenating the entire stream:

https://github.com/hwchase17/langchainjs/blob/main/langchain/src/llms/ollama.ts#L124

Jacob Lee (jacoblee93) · 2023-08-23T02:57:05Z

Let me know if you need help using the newer package! I'm happy to dig in tomorrow.

…ew, before removing deprecated module and looking into stream interface

…ing live

Jacob Lee (jacoblee93) · 2023-08-23T16:12:28Z

Thanks for the quick updates!

To get .stream() working you'll actually need to implement _streamResponseChunks as shown here:

https://github.com/hwchase17/langchainjs/blob/main/langchain/src/llms/openai.ts#L452

I can do this later today though - it should be very similar to what you have, and we can share the streaming logic.

…aws-bedrock-streaming

Jacob Lee (jacoblee93) · 2023-08-23T22:02:02Z

Ben Pleasanton (@Benyuel) Refactored a bit so we can support the .stream() method and added an integration test - can you confirm yarn test:single langchain/src/llms/tests/bedrock.int.test.ts works for you?

I also added retries and the ability to pass an abort signal to cancel in flight requests (hackily).

Also CC Vitaly Neyman (@vitaly-ps) for a second look?

Ben Pleasanton (Benyuel) · 2023-08-24T10:52:59Z

Thanks so much Jacob Lee (@jacoblee93)!

I made a minor change to get the tests working for me, and also added passing in temperature and maxTokens into the request.

I think this is ready from my perspective

I also added retries and the ability to pass an abort signal to cancel in flight requests

I tested this works as well 👍

Jacob Lee (jacoblee93) · 2023-08-24T20:56:18Z

    chunks.push(chunk);
  }
-  expect(chunks.length).toBeGreaterThan(1);
+  expect(chunks.length).toBeGreaterThanOrEqual(1);


Shouldn't this be greater than 1?

The mock I added does streaming, but just 1 chunk.
https://github.com/hwchase17/langchainjs/blob/7f2271769488152259791226d4c3767f60919ae9/langchain/src/llms/tests/bedrock.int.test.ts#L321-L337
That's why I swapped this.

Jacob Lee (jacoblee93) · 2023-08-24T20:57:23Z

    maxTokens: 20,
    region,
    model,
+    async fetchFn(


Ah I intended this one to test that this works with a live Bedrock instance, without a fetch function mock. Can you try that out?

oh sorry, you can revert the mock I added. I just assumed no connection to bedrock so mocked it.

To answer your question though, yes I tried this .stream against a live bedrock endpoint too.

Jacob Lee (jacoblee93) · 2023-08-24T20:58:02Z

Hey Ben Pleasanton (@Benyuel), I think you changed the test I added to use a mocked function - does this work with a live Bedrock instance?

Jacob Lee (jacoblee93) · 2023-08-24T20:58:43Z

      return responseBody.completion;
    } else if (provider === "ai21") {
-      return responseBody.completions[0].data.text;
+      return responseBody.data.text;


This was wrong before?

This was correct before for the non-streaming bedrock api. But when I change to the bedrock invoke-with-response-stream endpoint, these change to what's in this PR currently.

Jacob Lee (jacoblee93) · 2023-08-24T21:39:02Z

Looks great! Thanks again for your patience!

Uzay Macar (uzaymacar) · 2023-10-07T23:49:32Z

Hey both, I've found this PR from one of the linked issues -- I'm still confused about how to implement streaming with Bedrock similar to what is shown in the docs: https://js.langchain.com/docs/modules/model_io/models/chat/how_to/streaming. Simply changing ChatOpenAI with ChatBedrock does not work, but maybe I need to do something else?

Jacob Lee (jacoblee93) · 2023-10-08T00:00:03Z

Ah apologies, we really need to update that - instead use:

https://js.langchain.com/docs/expression_language/interface#stream

Uzay Macar (uzaymacar) · 2023-10-08T00:06:07Z

Awesome thanks Jacob Lee (@jacoblee93), I'm now able to stream with Bedrock too 👍

Ben Pleasanton (Benyuel) added 2 commits August 22, 2023 11:26

draft

93a9fcd

draft

6cce56e

dosubot bot added the auto:enhancement label Aug 22, 2023

dosubot bot reviewed Aug 22, 2023

View reviewed changes

vercel bot temporarily deployed to Preview August 22, 2023 17:26 Inactive

Jacob Lee (jacoblee93) reviewed Aug 23, 2023

View reviewed changes

Comment thread langchain/src/llms/bedrock-chat.ts Outdated

Jacob Lee (jacoblee93) reviewed Aug 23, 2023

View reviewed changes

Comment thread langchain/src/llms/bedrock-chat.ts Outdated

Jacob Lee (jacoblee93) reviewed Aug 23, 2023

View reviewed changes

Jacob Lee (jacoblee93) self-assigned this Aug 23, 2023

Jacob Lee (jacoblee93) added the in progress PRs that are in progress but not ready to merge label Aug 23, 2023

Ben Pleasanton (Benyuel) marked this pull request as draft August 23, 2023 10:24

Ben Pleasanton (Benyuel) added 2 commits August 23, 2023 07:16

stash draft changes migrating to bedrock.ts instead of bedrock-chat n…

b45f348

…ew, before removing deprecated module and looking into stream interface

remove extraneous example previously added

5ccfe2e

Ben Pleasanton (Benyuel) changed the title ~~Add Amazon Bedrock LLM chat model integration with streaming support~~ Add support for streaming responses for Amazon Bedrock LLM Aug 23, 2023

vercel bot temporarily deployed to Preview August 23, 2023 11:41 Inactive

draft new non-deprecated dependency before fixing up linting and test…

36181ce

…ing live

vercel bot temporarily deployed to Preview August 23, 2023 12:33 Inactive

fix linting in test

83d4223

vercel bot temporarily deployed to Preview August 23, 2023 13:04 Inactive

Jacob Lee (jacoblee93) added 3 commits August 23, 2023 14:28

Merge branch 'main' of https://github.com/hwchase17/langchainjs into …

3939d92

…aws-bedrock-streaming

Add .stream() runnable support, integration test

d1ecd28

Merge branch 'main' of https://github.com/hwchase17/langchainjs into …

8b6e39c

…aws-bedrock-streaming

Jacob Lee (jacoblee93) marked this pull request as ready for review August 23, 2023 22:00

Jacob Lee (jacoblee93) added question Further information is requested close PRs that need one or two touch-ups to be ready and removed in progress PRs that are in progress but not ready to merge labels Aug 23, 2023

vercel bot temporarily deployed to Preview August 23, 2023 22:26 Inactive

Pass in max tokens and temp to the api

7f22717

vercel bot temporarily deployed to Preview August 24, 2023 11:03 Inactive

Merge branch 'main' into aws-bedrock-streaming

8b49ebb

vercel bot temporarily deployed to Preview August 24, 2023 11:16 Inactive

Jacob Lee (jacoblee93) reviewed Aug 24, 2023

View reviewed changes

Jacob Lee (jacoblee93) merged commit f23711f into langchain-ai:main Aug 24, 2023

		@@ -590,9 +593,11 @@
		"@aws-sdk/client-sagemaker-runtime": "^3.310.0",

		@@ -0,0 +1,267 @@
		import { SignatureV4 } from "@aws-sdk/signature-v4";

Conversation

Ben Pleasanton (Benyuel) commented Aug 22, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

vercel bot commented Aug 22, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dosubot bot Aug 22, 2023

Choose a reason for hiding this comment

Uh oh!

dosubot bot Aug 22, 2023

Choose a reason for hiding this comment

Uh oh!

dosubot bot Aug 22, 2023

Choose a reason for hiding this comment

Uh oh!

Jacob Lee (jacoblee93) commented Aug 23, 2023

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Jacob Lee (jacoblee93) commented Aug 23, 2023

Uh oh!

Jacob Lee (jacoblee93) commented Aug 23, 2023

Uh oh!

Jacob Lee (jacoblee93) commented Aug 23, 2023

Uh oh!

Ben Pleasanton (Benyuel) commented Aug 24, 2023

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Jacob Lee (jacoblee93) commented Aug 24, 2023

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Jacob Lee (jacoblee93) commented Aug 24, 2023

Uh oh!

Uzay Macar (uzaymacar) commented Oct 7, 2023

Uh oh!

Jacob Lee (jacoblee93) commented Oct 8, 2023

Uh oh!

Uzay Macar (uzaymacar) commented Oct 8, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Ben Pleasanton (Benyuel) commented Aug 22, 2023 •

edited

Loading

vercel bot commented Aug 22, 2023 •

edited

Loading