Skip to content

feat(provider): Add support to pass fallback provider through stream … #880

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
Yaswanth-2874 wants to merge 1 commit into from
Closed

feat(provider): Add support to pass fallback provider through stream … #880

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 20 additions & 1 deletion src/lib/neurolink.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2853,13 +2853,13 @@
* @see {@link stream} for streaming generation
* @since 1.0.0
*/
async generate(

Check warning on line 2856 in src/lib/neurolink.ts

View workflow job for this annotation

GitHub Actions / test (20) 

Async method 'generate' has too many lines (517). Maximum allowed is 300

Check warning on line 2856 in src/lib/neurolink.ts

View workflow job for this annotation

GitHub Actions / 🛡️ Code Quality & Security Gate 

Async method 'generate' has too many lines (517). Maximum allowed is 300
optionsOrPrompt: GenerateOptions | string,
): Promise<GenerateResult> {
return tracers.sdk.startActiveSpan(
"neurolink.generate",
{ kind: SpanKind.INTERNAL },
async (generateSpan) => {

Check warning on line 2862 in src/lib/neurolink.ts

View workflow job for this annotation

GitHub Actions / test (20) 

Async arrow function has too many lines (509). Maximum allowed is 300

Check warning on line 2862 in src/lib/neurolink.ts

View workflow job for this annotation

GitHub Actions / 🛡️ Code Quality & Security Gate 

Async arrow function has too many lines (509). Maximum allowed is 300
// Set metrics trace context for parent-child span linking.
// The generation span will be the root (no parentSpanId).
// Tool spans will be children of the root span via rootSpanId.
Expand All @@ -2872,7 +2872,7 @@
// so concurrent generate/stream calls don't race.
return metricsTraceContextStorage.run(
{ traceId: metricsTraceId, parentSpanId: metricsRootSpanId },
async () => {

Check warning on line 2875 in src/lib/neurolink.ts

View workflow job for this annotation

GitHub Actions / test (20) 

Async arrow function has too many lines (494). Maximum allowed is 300

Check warning on line 2875 in src/lib/neurolink.ts

View workflow job for this annotation

GitHub Actions / 🛡️ Code Quality & Security Gate 

Async arrow function has too many lines (494). Maximum allowed is 300
try {
const originalPrompt =
this._extractOriginalPrompt(optionsOrPrompt);
Expand Down Expand Up @@ -2971,7 +2971,7 @@
// Set session and user IDs from context for Langfuse spans and execute with proper async scoping
return await this.setLangfuseContextFromOptions(
options,
async () => {

Check warning on line 2974 in src/lib/neurolink.ts

View workflow job for this annotation

GitHub Actions / test (20) 

Async arrow function has too many lines (362). Maximum allowed is 300

Check warning on line 2974 in src/lib/neurolink.ts

View workflow job for this annotation

GitHub Actions / 🛡️ Code Quality & Security Gate 

Async arrow function has too many lines (362). Maximum allowed is 300
if (
this.conversationMemoryConfig?.conversationMemory
?.mem0Enabled &&
Expand Down Expand Up @@ -3802,7 +3802,7 @@
* 4. Fall back to direct provider generation
* 5. Store conversation turn for future context
*/
private async generateTextInternal(

Check warning on line 3805 in src/lib/neurolink.ts

View workflow job for this annotation

GitHub Actions / test (20) 

Async method 'generateTextInternal' has too many lines (426). Maximum allowed is 300

Check warning on line 3805 in src/lib/neurolink.ts

View workflow job for this annotation

GitHub Actions / 🛡️ Code Quality & Security Gate 

Async method 'generateTextInternal' has too many lines (426). Maximum allowed is 300
options: TextGenerationOptions,
): Promise<TextGenerationResult> {
return tracers.sdk.startActiveSpan(
Expand Down Expand Up @@ -6161,7 +6161,12 @@
/* non-blocking */
}

const fallbackRoute = ModelRouter.getFallbackRoute(
const optFallbackProvider = enhancedOptions.fallbackProvider || undefined;
const optFallbackModel = enhancedOptions.fallbackModel || undefined;
const envFallbackProvider = process.env.FALLBACK_PROVIDER || undefined;
const envFallbackModel = process.env.FALLBACK_MODEL || undefined;

const modelConfigRoute = ModelRouter.getFallbackRoute(
originalPrompt || enhancedOptions.input.text || "",
{
provider: providerName,
Expand All @@ -6172,9 +6177,23 @@
{ fallbackStrategy: "auto" },
);

const fallbackRoute = {
...modelConfigRoute,
provider:
optFallbackProvider ?? envFallbackProvider ?? modelConfigRoute.provider,
model: optFallbackModel ?? envFallbackModel ?? modelConfigRoute.model,
};

logger.warn("Retrying with fallback provider", {
originalProvider: providerName,
fallbackProvider: fallbackRoute.provider,
fallbackModel: fallbackRoute.model,
fallbackSource:
optFallbackProvider || optFallbackModel
? "options"
: envFallbackProvider || envFallbackModel
? "env"
: "model_config",
reason: errorMsg,
});

Expand Down
2 changes: 2 additions & 0 deletions src/lib/types/streamTypes.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -473,6 +473,8 @@ export type StreamOptions = {
* @internal Set by NeuroLink SDK — not typically used directly by consumers.
*/
fileRegistry?: unknown;
fallbackProvider?: string;
fallbackModel?: string;
};

/**
Expand Down
Loading