Skip to content

fix(config): negatively cache deterministic hosted config rejections and stop reporting 4xx errors #3719

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
kojiwakayama merged 2 commits into from
Aug 14, 2026
Merged

fix(config): negatively cache deterministic hosted config rejections and stop reporting 4xx errors #3719

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
1500751
fix(config): negatively cache deterministic hosted config rejections …
kojiwakayama Aug 14, 2026
91e0eb2
Keep observability API reference aligned with capture changes
kojiwakayama Aug 14, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 6 additions & 6 deletions docs/api-reference/veryfront/observability.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -43,13 +43,13 @@ const result = await withSpan("load-data", async () => {
| Name | Description | Source |
| ---------------------------------------- | -------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------ |
| `addSpanEvent` | Event emitted for add span. | [source](https://github.com/veryfront/veryfront-code/blob/main/src/observability/tracing/index.ts#L70) |
| `captureApplicationError` | | [source](https://github.com/veryfront/veryfront-code/blob/main/src/observability/application-errors.ts#L223) |
| `captureApplicationError` | | [source](https://github.com/veryfront/veryfront-code/blob/main/src/observability/application-errors.ts#L224) |
| `createChildSpan` | Create child span. | [source](https://github.com/veryfront/veryfront-code/blob/main/src/observability/tracing/index.ts#L79) |
| `createFileLogSubscriber` | Create file log subscriber. | [source](https://github.com/veryfront/veryfront-code/blob/main/src/observability/file-log-subscriber.ts#L541) |
| `createOpenTelemetryServiceTracer` | Create open telemetry service tracer. | [source](https://github.com/veryfront/veryfront-code/blob/main/src/observability/tracing/service-tracer.ts#L364) |
| `endSpan` | End an active tracing span. | [source](https://github.com/veryfront/veryfront-code/blob/main/src/observability/tracing/index.ts#L57) |
| `extractContext` | Context for extract. | [source](https://github.com/veryfront/veryfront-code/blob/main/src/observability/tracing/index.ts#L88) |
| `flushApplicationErrors` | | [source](https://github.com/veryfront/veryfront-code/blob/main/src/observability/application-errors.ts#L241) |
| `flushApplicationErrors` | | [source](https://github.com/veryfront/veryfront-code/blob/main/src/observability/application-errors.ts#L242) |
| `getActiveContext` | Context for get active. | [source](https://github.com/veryfront/veryfront-code/blob/main/src/observability/tracing/index.ts#L98) |
| `getErrorCollector` | Return error collector. | [source](https://github.com/veryfront/veryfront-code/blob/main/src/observability/error-collector.ts#L406) |
| `getGlobalMetricsAPI` | | [source](https://github.com/veryfront/veryfront-code/blob/main/src/observability/tracing/api-shim.ts#L667) |
Expand All @@ -58,7 +58,7 @@ const result = await withSpan("load-data", async () => {
| `getMetricsState` | State for get metrics. | [source](https://github.com/veryfront/veryfront-code/blob/main/src/observability/metrics/index.ts#L38) |
| `getTraceContext` | Context for get trace. | [source](https://github.com/veryfront/veryfront-code/blob/main/src/observability/tracing/otlp-setup.ts#L500) |
| `initAutoInstrumentation` | Initialize automatic instrumentation wrappers. | [source](https://github.com/veryfront/veryfront-code/blob/main/src/observability/auto-instrument/orchestrator.ts#L15) |
| `initializeApplicationErrorReporter` | Activate an explicitly selected reporter initializer. | [source](https://github.com/veryfront/veryfront-code/blob/main/src/observability/application-errors.ts#L148) |
| `initializeApplicationErrorReporter` | Activate an explicitly selected reporter initializer. | [source](https://github.com/veryfront/veryfront-code/blob/main/src/observability/application-errors.ts#L149) |
| `initializeOTLP` | Initialize OTLP tracing export. | [source](https://github.com/veryfront/veryfront-code/blob/main/src/observability/tracing/otlp-setup.ts#L113) |
| `initMetrics` | Initialize metrics collection. | [source](https://github.com/veryfront/veryfront-code/blob/main/src/observability/metrics/index.ts#L20) |
| `initTracing` | Initialize tracing for the current runtime. | [source](https://github.com/veryfront/veryfront-code/blob/main/src/observability/tracing/index.ts#L18) |
Expand Down Expand Up @@ -134,7 +134,7 @@ const result = await withSpan("load-data", async () => {
| `ApplicationErrorReporter` | Provider-neutral application error capture and flush interface. | [source](https://github.com/veryfront/veryfront-code/blob/main/src/observability/application-error-contract.ts#L23) |
| `ApplicationErrorReporterInitializationContext` | Runtime context passed to an explicitly selected reporter initializer. | [source](https://github.com/veryfront/veryfront-code/blob/main/src/extensions/observability/application-error-reporter.ts#L9) |
| `ApplicationErrorReporterInitializer` | Application-composition contract for an error-reporting implementation. | [source](https://github.com/veryfront/veryfront-code/blob/main/src/extensions/observability/application-error-reporter.ts#L20) |
| `ApplicationErrorReporterLifecycle` | Active application-error reporter ownership. | [source](https://github.com/veryfront/veryfront-code/blob/main/src/observability/application-errors.ts#L27) |
| `ApplicationErrorReporterLifecycle` | Active application-error reporter ownership. | [source](https://github.com/veryfront/veryfront-code/blob/main/src/observability/application-errors.ts#L28) |
| `ApplicationErrorReporterSession` | Reporter and cleanup ownership returned by an application-selected initializer. | [source](https://github.com/veryfront/veryfront-code/blob/main/src/extensions/observability/application-error-reporter.ts#L14) |
| `AttributeValue` | | [source](https://github.com/veryfront/veryfront-code/blob/main/src/observability/tracing/api-shim.ts#L33) |
| `AutoInstrumentConfig` | Configuration used by auto instrument. | [source](https://github.com/veryfront/veryfront-code/blob/main/src/observability/auto-instrument/types.ts#L24) |
Expand Down Expand Up @@ -237,8 +237,8 @@ import {

| Name | Description | Source |
| ---------------------------- | ------------------------------------------------- | ------------------------------------------------------------------------------------------------------------ |
| `captureApplicationError` | | [source](https://github.com/veryfront/veryfront-code/blob/main/src/observability/application-errors.ts#L223) |
| `flushApplicationErrors` | | [source](https://github.com/veryfront/veryfront-code/blob/main/src/observability/application-errors.ts#L241) |
| `captureApplicationError` | | [source](https://github.com/veryfront/veryfront-code/blob/main/src/observability/application-errors.ts#L224) |
| `flushApplicationErrors` | | [source](https://github.com/veryfront/veryfront-code/blob/main/src/observability/application-errors.ts#L242) |
| `initializeSentry` | Initialize the process-wide Sentry reporter once. | [source](https://github.com/veryfront/veryfront-code/blob/main/src/observability/sentry.ts#L86) |
| `initializeSentryFromEnv` | | [source](https://github.com/veryfront/veryfront-code/blob/main/src/observability/sentry.ts#L70) |
| `isSentryEnabled` | Return whether Sentry is explicitly enabled. | [source](https://github.com/veryfront/veryfront-code/blob/main/src/observability/sentry.ts#L39) |
Expand Down
179 changes: 179 additions & 0 deletions src/config/loader.test.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2209,6 +2209,185 @@ export default config as const;
assertEquals(reads, 4);
});

describe("hosted config negative caching", () => {
const productionSourceContext = {
productionMode: true,
releaseId: "release-negative-cache",
environmentName: "Production",
} as const;
type PreparedContext = Awaited<
ReturnType<typeof prepareDeclarativeConfigContext>
>;
type TestAdapter = ReturnType<typeof setup>;

function createHostedAdapter(
readSource: () => string = () => 'export default { title: "source" };',
): TestAdapter {
const adapter = setup();
Object.assign(adapter.fs, {
getUnderlyingAdapter: () => adapter.fs,
isMultiProjectMode: () => true,
isVeryfrontAdapter: () => true,
exists: async (path: string) => path === "/veryfront.config.ts",
readFile: async (path: string) => {
if (path !== "/veryfront.config.ts") throw configCandidateNotFound(path);
return readSource();
},
});
return adapter;
}

function loadProductionHostedConfig(
adapter: TestAdapter,
preparedContext: PreparedContext,
) {
const projectId = "project-negative-cache";
return runWithRequestContext(
{
projectSlug: projectId,
projectId,
token: "token",
productionMode: true,
releaseId: productionSourceContext.releaseId,
environmentName: productionSourceContext.environmentName,
},
() =>
getHostedConfig(`/hosted/${projectId}`, adapter, {
cacheKey: projectId,
sourceContext: productionSourceContext,
preparedContext,
}),
);
}

function prepareProductionContext(): Promise<PreparedContext> {
return prepareDeclarativeConfigContext({
environmentName: "Production",
environment: { TENANT: "tenant" },
});
}

it("does not re-evaluate a deterministically rejected hosted config on later requests", async () => {
const adapter = createHostedAdapter();
const preparedContext = await prepareProductionContext();
let evaluations = 0;
__setHostedConfigEvaluatorForTests(async () => {
evaluations += 1;
throw new DeclarativeConfigEvaluationError({
code: "forbidden-capability",
phase: "validate",
reason: "unsupported-call",
});
});

const first = await assertRejects(
() => loadProductionHostedConfig(adapter, preparedContext),
VeryfrontError,
) as VeryfrontError;
const second = await assertRejects(
() => loadProductionHostedConfig(adapter, preparedContext),
VeryfrontError,
) as VeryfrontError;

assertEquals(first.slug, "config-parse-error");
assertEquals(second.slug, "config-parse-error");
assertStringIncludes(
first.detail ?? "",
"Hosted configuration rejected (forbidden-capability: unsupported-call)",
);
assertStringIncludes(
second.detail ?? "",
"Hosted configuration rejected (forbidden-capability: unsupported-call)",
);
assertEquals(
evaluations,
1,
"a deterministic rejection must be negatively cached, not re-evaluated per request",
);
});

it("re-evaluates a rejected hosted config after the source changes", async () => {
let source = "const forbidden = process.env;\nexport default { title: 'source' };";
const adapter = createHostedAdapter(() => source);
const preparedContext = await prepareProductionContext();
let evaluations = 0;
__setHostedConfigEvaluatorForTests(async () => {
evaluations += 1;
if (evaluations === 1) {
throw new DeclarativeConfigEvaluationError({
code: "forbidden-capability",
phase: "validate",
reason: "unsupported-call",
});
}
return { title: "corrected" };
});

await assertRejects(
() => loadProductionHostedConfig(adapter, preparedContext),
VeryfrontError,
);
source = 'export default { title: "corrected" };';
const corrected = await loadProductionHostedConfig(adapter, preparedContext);

assertEquals(corrected.title, "corrected");
assertEquals(evaluations, 2);
});

it("re-evaluates a rejected hosted config after clearConfigCache", async () => {
const adapter = createHostedAdapter();
const preparedContext = await prepareProductionContext();
let evaluations = 0;
__setHostedConfigEvaluatorForTests(async () => {
evaluations += 1;
throw new DeclarativeConfigEvaluationError({
code: "forbidden-capability",
phase: "validate",
reason: "unsupported-call",
});
});

await assertRejects(
() => loadProductionHostedConfig(adapter, preparedContext),
VeryfrontError,
);
clearConfigCache();
await assertRejects(
() => loadProductionHostedConfig(adapter, preparedContext),
VeryfrontError,
);

assertEquals(evaluations, 2);
});

it("never negatively caches retryable infrastructure failures", async () => {
const adapter = createHostedAdapter();
const preparedContext = await prepareProductionContext();
let evaluations = 0;
__setHostedConfigEvaluatorForTests(async () => {
evaluations += 1;
if (evaluations === 1) {
throw new DeclarativeConfigEvaluationError({
code: "evaluator-unavailable",
phase: "worker",
reason: "worker-timeout",
retryable: true,
});
}
return { title: "recovered" };
});

await assertRejects(
() => loadProductionHostedConfig(adapter, preparedContext),
VeryfrontError,
);
const recovered = await loadProductionHostedConfig(adapter, preparedContext);

assertEquals(recovered.title, "recovered");
assertEquals(evaluations, 2);
});
});

describe("hosted config single-flight", () => {
const productionSourceContext = {
productionMode: true,
Expand Down
53 changes: 52 additions & 1 deletion src/config/loader.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -567,6 +567,26 @@ const configCacheByProject = new LRUCache<string, ConfigCacheEntry>({
maxEntries: DEFAULT_CONFIG_CACHE_MAX_ENTRIES,
});

interface HostedConfigFailureCacheEntry {
readonly revision: number;
readonly error: DeclarativeConfigEvaluationError;
}

/**
* Negative cache for deterministic hosted config rejections.
*
* The hosted cache key already folds in the exact source digest, policy
* version and environment fingerprint, so a rejected source stays rejected
* until the tenant ships different content; re-sending it to the evaluator
* worker on every request only repeats the same failure.
*/
const hostedConfigFailureCacheByProject = new LRUCache<
string,
HostedConfigFailureCacheEntry
>({
maxEntries: DEFAULT_CONFIG_CACHE_MAX_ENTRIES,
});

type HostedConfigEvaluator = typeof evaluatePreparedDeclarativeConfigInWorker;

interface HostedConfigSourceSelection {
Expand Down Expand Up @@ -632,8 +652,9 @@ const trustedConfigFlights = new IntrinsicMap<string, TrustedConfigFlight>();
const trustedVirtualFilesystemIds = new IntrinsicWeakMap<object, number>();
let nextTrustedVirtualFilesystemId = 1;

// Register cache for monitoring
// Register caches for monitoring
registerLRUCache("config-cache", configCacheByProject);
registerLRUCache("config-failure-cache", hostedConfigFailureCacheByProject);

let cacheRevision = 0;

Expand Down Expand Up @@ -1069,6 +1090,19 @@ function buildHostedConfigFlightKey(hostedCacheKey: string, revision: number): s
return `${revision}:${hostedCacheKey}`;
}

/**
* Whether a hosted evaluation failure is guaranteed to repeat for the same
* cache key. Worker-phase and retryable failures are infrastructure
* conditions that can succeed on retry, so they must never be cached.
*/
function isDeterministicHostedConfigRejection(
error: unknown,
): error is DeclarativeConfigEvaluationError {
return error instanceof DeclarativeConfigEvaluationError &&
!error.retryable &&
error.phase !== "worker";
}

function createHostedConfigFlight(
flightKey: string,
hostedCacheKey: string,
Expand Down Expand Up @@ -1117,6 +1151,15 @@ function createHostedConfigFlight(
},
(error: unknown) => {
finish();
if (
usePersistentCache && cacheRevision === revisionAtStart &&
isDeterministicHostedConfigRejection(error)
) {
hostedConfigFailureCacheByProject.set(hostedCacheKey, {
revision: revisionAtStart,
error,
});
}
result.reject(error);
},
);
Expand Down Expand Up @@ -1739,6 +1782,13 @@ function loadHostedConfigFromSource(
return cached.config;
}

const cachedFailure = usePersistentCache
? hostedConfigFailureCacheByProject.get(hostedCacheKey)
: undefined;
if (cachedFailure?.revision === revisionAtStart) {
throw cachedFailure.error;
}

const flight = getOrCreateHostedConfigFlight(
hostedCacheKey,
payload,
Expand Down Expand Up @@ -2492,6 +2542,7 @@ export function __getTrustedConfigFlightStateForTests(): Readonly<{

export function clearConfigCache(): void {
configCacheByProject.clear();
hostedConfigFailureCacheByProject.clear();
cacheRevision++;
}

Expand Down
Loading