file/image embedding flow fixes

.github/workflows/release-pipeline.yml

@@ -305,6 +305,17 @@ jobs:
           # dl.google.com / packages.cloud.google.com / objects.githubusercontent.com
           # are the gcloud CLI install; oauth2.googleapis.com + accounts.google.com
           # are the service-account token exchange for the Vertex parity cells.
+          #
+          # www.berkshirehathaway.com hosts the PDF the document-input rows pass as
+          # document.source.type "url". Anthropic fetches such a URL server-side, but
+          # Bedrock/Vertex/Gemini/OpenAI/Azure have no url member on their document
+          # types, so Bifrost downloads it here and inlines the bytes - meaning the
+          # RUNNER needs egress to it. Omitting it fails 44 rows whose requests all
+          # go to 127.0.0.1, which reads as "PDF input is broken on five providers".
+          #
+          # discoveryengine.googleapis.com is the Vertex semantic-ranker backend for
+          # the rerank rows. Its host is built in Go rather than declared in
+          # config.json, so auditing the seeded provider config does not reveal it.
           allowed-endpoints: >
             127.0.0.1:8080
             7defe2860d5ee49a1e667e1eeea34b25.r2.cloudflarestorage.com:443
@@ -332,6 +343,7 @@ jobs:
             bedrock-runtime.us-east-1.amazonaws.com:443
             bedrock.us-east-1.amazonaws.com:443
             bifrost-batch-api-file-upload-testing.s3.us-east-1.amazonaws.com:443
+            discoveryengine.googleapis.com:443
             dl.google.com:443
             fal.media:443
             fonts.googleapis.com:443
@@ -357,6 +369,7 @@ jobs:
             storage.googleapis.com:443
             sum.golang.org:443
             us-central1-aiplatform.googleapis.com:443
+            www.berkshirehathaway.com:443
             www.getbifrost.ai:443
 
       - name: Checkout repository
@@ -524,6 +537,7 @@ jobs:
             bedrock-runtime.us-east-1.amazonaws.com:443
             bedrock.us-east-1.amazonaws.com:443
             bifrost-batch-api-file-upload-testing.s3.us-east-1.amazonaws.com:443
+            discoveryengine.googleapis.com:443
             dl.google.com:443
             fal.media:443
             fonts.googleapis.com:443
@@ -549,6 +563,7 @@ jobs:
             storage.googleapis.com:443
             sum.golang.org:443
             us-central1-aiplatform.googleapis.com:443
+            www.berkshirehathaway.com:443
             www.getbifrost.ai:443
 
       - name: Checkout repository

.github/workflows/run-core-tests.yml

@@ -106,6 +106,7 @@ jobs:
             bedrock-runtime.us-east-1.amazonaws.com:443
             bedrock.us-east-1.amazonaws.com:443
             bifrost-batch-api-file-upload-testing.s3.us-east-1.amazonaws.com:443
+            discoveryengine.googleapis.com:443
             dl.google.com:443
             fal.media:443
             fonts.googleapis.com:443
@@ -131,6 +132,7 @@ jobs:
             storage.googleapis.com:443
             sum.golang.org:443
             us-central1-aiplatform.googleapis.com:443
+            www.berkshirehathaway.com:443
             www.getbifrost.ai:443
 
       - name: Checkout requested branch

.github/workflows/scripts/check-egress-allowlist.sh

@@ -1,8 +1,13 @@
 #!/usr/bin/env bash
 set -euo pipefail
 
-# Assert that every workflow job which runs apt-get under a blocked egress
-# policy also allowlists the Ubuntu APT hosts it will need.
+# Two guards on harden-runner egress allowlists:
+#
+#   1. every job that runs apt-get under `egress-policy: block` allowlists the
+#      Ubuntu APT hosts it will need;
+#   2. every job that runs the provider harness allowlists each external host
+#      the harness will dial - see the second python block for why that is not
+#      the same set as "the hosts the collection sends requests to".
 #
 # harden-runner's `egress-policy: block` drops every destination absent from
 # `allowed-endpoints`, so a job that shells out to `apt-get update` fails at the
@@ -16,6 +21,12 @@ set -euo pipefail
 
 SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
 WORKFLOW_DIR="$(cd "${SCRIPT_DIR}/.." && pwd)"
+# Exported for the python blocks below. They run from stdin (`python3 - ...`),
+# where __file__ is "<stdin>" rather than a path - deriving the repo root there
+# silently resolves somewhere above the checkout, every input file reads as
+# absent, and the check passes having examined nothing.
+REPO_ROOT="$(cd "${WORKFLOW_DIR}/../.." && pwd)"
+export REPO_ROOT
 
 if [[ $# -gt 0 ]]; then
   FILES=("$@")
@@ -87,3 +98,194 @@ if failures:
 
 print(f"OK: all {checked} apt-using blocked-egress job(s) allowlist the Ubuntu APT hosts.")
 PY
+
+# ---------------------------------------------------------------------------
+# 2. Provider-harness hosts
+# ---------------------------------------------------------------------------
+#
+# The harness sends nearly every request to 127.0.0.1:8080, so its collection
+# looks like it needs no egress at all. It does: bifrost-http runs on the SAME
+# runner and makes the upstream provider calls, and it downloads any URL passed
+# as a document/image source for providers whose APIs cannot take a URL
+# directly. Those fetches leave the runner and are governed by the same
+# allowlist.
+#
+# That combination is why this check exists. A blocked host surfaces as a
+# connection error attributed to the provider, so it reads as a flaky provider
+# rather than a firewall - and release-pipeline.yml only runs on push to main,
+# so the first time anyone sees it is during a real release.
+#
+# The rule enforced here is deliberately blunt: every external host appearing in
+# the harness collection or its seeded provider config must be either
+# allowlisted or listed in NOT_DIALLED_HOSTS with a reason. Classifying
+# automatically would mean deciding which URLs Bifrost fetches and which the
+# provider fetches, which depends on per-provider API shape and would silently
+# get it wrong. Forcing a human to say which one it is cannot.
+
+python3 - "${FILES[@]}" <<'PY'
+import json
+import os
+import re
+import sys
+import yaml
+
+REPO_ROOT = os.environ["REPO_ROOT"]
+COLLECTION = os.path.join(REPO_ROOT, "tests/e2e/api/collections/provider-harness.json")
+PROVIDER_CONFIG = os.path.join(REPO_ROOT, "tests/integrations/python/config.json")
+
+# A missing input means this check verified nothing, which must be an error and
+# not a silent pass - that is the exact failure mode the REPO_ROOT export fixes.
+for required_input in (COLLECTION, PROVIDER_CONFIG):
+    if not os.path.exists(required_input):
+        print(f"::error::egress check input not found: {required_input}", file=sys.stderr)
+        sys.exit(1)
+
+# Scripts that mean "this job runs the provider harness".
+HARNESS_MARKERS = ("test-core.sh", "test-provider-harness.sh")
+
+# Hosts that appear in the sources scanned above but that the runner never
+# dials, each with the reason it is exempt. Anything not listed here and not
+# allowlisted fails this check, so a new host forces an explicit decision rather
+# than being quietly assumed harmless.
+NOT_DIALLED_HOSTS = {
+    # Fetched by the provider's own server-side tooling, never by Bifrost.
+    "example.com": "MCP server_url and web_fetch targets - dereferenced by the provider",
+    "www.youtube.com": "Gemini video input - Google fetches the URL server-side",
+    "en.wikipedia.org": "googleSearch excludeDomains filter value, not a URL anyone fetches",
+    "placeholder.search.windows.net": "Azure AI Search placeholder in a request body; never resolved",
+    # Deliberately unresolvable - negative tests assert the failure path.
+    "bifrost.invalid": "reserved .invalid TLD; negative-path tests assert it fails to resolve",
+    # Documentation links inside descriptions and comments.
+    "anthropic.com": "doc link in a folder description",
+    "www.anthropic.com": "doc link in a folder description",
+    "platform.claude.com": "doc link in a folder description",
+    "developers.openai.com": "doc link in a folder description",
+    "docs.aws.amazon.com": "doc link in a folder description",
+    "docs.x.ai": "doc link in a folder description",
+    "schema.getpostman.com": "Postman collection schema declaration, not fetched at run time",
+    # Identifiers that merely look like endpoints.
+    "s3.amazonaws.com": "XML namespace URI in a ListBucketResult document; namespaces are names, not fetches",
+    "www.googleapis.com": "the cloud-platform OAuth SCOPE string; the token exchange itself goes to oauth2.googleapis.com",
+}
+
+HOST_RE = re.compile(r"https?://([A-Za-z0-9._-]+\.[A-Za-z]{2,})")
+# {{var}} placeholders resolve at run time from secrets; the concrete hosts they
+# expand to (regional Bedrock/Vertex endpoints) cannot be known statically, so
+# they are out of scope here and stay a manual review item.
+TEMPLATED = re.compile(r"\{\{")
+
+
+def hosts_in(path):
+    if not os.path.exists(path):
+        return set()
+    with open(path, encoding="utf-8") as fh:
+        raw = fh.read()
+    found = set()
+    for line in raw.splitlines():
+        if TEMPLATED.search(line):
+            # Strip the templated spans, keep any fully-literal URLs on the line.
+            line = re.sub(r"https?://[^\s\"']*\{\{[^\s\"']*", " ", line)
+        for m in HOST_RE.finditer(line):
+            found.add(m.group(1))
+    return found
+
+
+# Provider endpoints Bifrost builds in Go rather than reading from config, which
+# is why scanning the seeded config alone is not enough: discoveryengine
+# (the Vertex reranker backend) is assembled in core/providers/vertex and appears
+# in no config file, so it was missing from two workflows' allowlists at once.
+# Restricted to the cloud-provider domains whose hostnames are constructed this
+# way; anything else in that source tree is a doc link, not a dial.
+GO_HOST_SUFFIXES = (".googleapis.com", ".amazonaws.com", ".api.aws")
+
+
+def hosts_in_go(root):
+    found = set()
+    providers_dir = os.path.join(root, "core", "providers")
+    for dirpath, _, filenames in os.walk(providers_dir):
+        for name in filenames:
+            if not name.endswith(".go") or name.endswith("_test.go"):
+                continue
+            with open(os.path.join(dirpath, name), encoding="utf-8", errors="ignore") as fh:
+                text = fh.read()
+            for m in HOST_RE.finditer(text):
+                host = m.group(1)
+                if host.endswith(GO_HOST_SUFFIXES):
+                    found.add(host)
+    return found
+
+
+required = hosts_in(COLLECTION) | hosts_in(PROVIDER_CONFIG) | hosts_in_go(REPO_ROOT)
+required -= set(NOT_DIALLED_HOSTS)
+required = {h for h in required if h not in ("localhost",)}
+
+failures = []
+checked = 0
+
+for path in sys.argv[1:]:
+    with open(path, encoding="utf-8") as fh:
+        doc = yaml.safe_load(fh) or {}
+
+    for job_name, job in (doc.get("jobs") or {}).items():
+        if not isinstance(job, dict):
+            continue
+        steps = job.get("steps") or []
+
+        runs_harness = any(
+            any(marker in (step.get("run") or "") for marker in HARNESS_MARKERS)
+            for step in steps
+            if isinstance(step, dict)
+        )
+        if not runs_harness:
+            continue
+
+        harden = next(
+            (
+                step
+                for step in steps
+                if isinstance(step, dict)
+                and "step-security/harden-runner" in (step.get("uses") or "")
+            ),
+            None,
+        )
+        if harden is None:
+            continue
+
+        with_block = harden.get("with") or {}
+        if str(with_block.get("egress-policy", "")).strip() != "block":
+            continue
+
+        checked += 1
+        allowed = {
+            entry.rsplit(":", 1)[0]
+            for entry in (with_block.get("allowed-endpoints") or "").split()
+        }
+        missing = sorted(h for h in required if h not in allowed)
+        if missing:
+            failures.append((path, job_name, missing))
+
+for path, job_name, missing in failures:
+    print(
+        f"FAIL {path} :: job '{job_name}' runs the provider harness under "
+        f"egress-policy: block but omits {', '.join(missing)}",
+        file=sys.stderr,
+    )
+
+if failures:
+    print(
+        "\nEach host above appears in the provider harness collection or its seeded\n"
+        "provider config. Either add it to that job's allowed-endpoints, or - if the\n"
+        "provider dereferences it server-side and the runner never dials it - add it to\n"
+        "NOT_DIALLED_HOSTS in this script with the reason.",
+        file=sys.stderr,
+    )
+    sys.exit(1)
+
+if checked == 0:
+    print("OK: no harness-running blocked-egress jobs found.")
+else:
+    print(
+        f"OK: all {checked} harness-running blocked-egress job(s) allowlist the "
+        f"{len(required)} host(s) the harness dials."
+    )
+PY

.github/workflows/workflow-lint.yml

@@ -14,7 +14,7 @@ permissions:
 
 jobs:
   check-egress-allowlist:
-    name: Egress allowlist covers apt usage
+    name: Egress allowlist covers apt + provider-harness hosts
     runs-on: ubuntu-latest
     timeout-minutes: 5
     steps:

Makefile

@@ -2332,6 +2332,10 @@ run-provider-harness-test: $(if $(HELP),,install-newman) ## Run the Bifrost prov
 		fi; \
 	fi; \
 	say "$(CYAN)Augmenting provider harness with generated streaming/thinking cases...$(NC)"; \
+	: "VERTEX_ACCESS_TOKEN_VAL is exported so the token-parity matrix can skip the Vertex"; \
+	: "direct legs when gcloud could not mint a token, instead of emitting cells that post an"; \
+	: "unresolved {{vertexAccessToken}} placeholder and 401 en masse."; \
+	export VERTEX_ACCESS_TOKEN_VAL; \
 	$(USE_NODE); run_quiet node tests/e2e/api/runners/augment-provider-harness.mjs \
 		--source tests/e2e/api/collections/provider-harness.json \
 		--out tmp/harness-augmented.json || { say "$(RED)Harness augmentation failed$(NC)"; exit 1; }; \

core/changelog.md

@@ -0,0 +1,15 @@
+- feat: support Gemini's server-side `toolCall`/`toolResponse` parts with `thoughtSignature` round-trip fidelity - server-side search rounds now surface as `web_search_call` items carrying their own call ID and queries, unmapped tool types are preserved on the native round-trip instead of being dropped, and each `thoughtSignature` appears exactly once across the reconstructed parts so Gemini accepts the replayed turn
+- feat: async 3D generation on Runware via `/videos` plus a raw `/runware_passthrough` route - `taskType` is now read from extra_params so any Runware async task can be driven through `/videos` (the 16:9 1080p width/height defaults now apply only to `videoInference`), `outputs.files[].url` is surfaced as `VideoOutput` URLs with the content type derived from the file extension, and the passthrough route forwards raw task arrays for capabilities with no first-class Bifrost surface such as upscaling and background removal
+- feat: surface Runware's provider-reported per-task `cost` across image, video/3D and passthrough so pricing uses the exact figure verbatim instead of a datasheet estimate - this matters for task types like 3D that have no datasheet rate; when no cost is reported the behavior is unchanged
+- feat: send `s3://` image and document references to Bedrock Converse as the `s3Location` source member instead of downloading the bytes and re-uploading them - Converse resolves the object itself, which skips a round trip and the 25 MiB inline cap entirely. Image format is derived from the object extension since nothing is fetched and there is no `Content-Type` to read, and an extension-less object is rejected up front rather than producing an opaque 400
+- feat: resolve Vertex URL sources per model family rather than inlining everything - a `gs://` URI is now forwarded to Gemini/Gemma as `fileData.fileUri` (the documented form, resolved under the caller's own project IAM, and the only thing that keeps multi-hundred-MB video inputs viable) and read from Cloud Storage with the request key's own Google credentials for Claude-on-Vertex, which accepts base64 sources only. `http(s)` is still always fetched: forwarding one was measured against the harness and Vertex rejected every endpoint shape with `URL_REJECTED-REJECTED_FC_TOO_MANY_PENDING`
+- fix: allow Vertex AI to send function declarations and a Google Search tool in the same request without `includeServerSideToolInvocations` - Vertex accepts the combination natively, so Google Search was being dropped for no reason, and search localization via `RetrievalConfig.LatLng` is now preserved when both tool types are present
+- fix: prefer function declarations over Google Search when tool combination is disabled on Gemini - Google's tool combination is Preview and Gemini 3 only (https://ai.google.dev/gemini-api/docs/generate-content/tool-combination), so on every other model one of the two tool types has to be dropped. Function declarations now win: they carry the caller's own tools, or the ones Bifrost's MCP gateway synthesized from their connected servers, and dropping those leaves the model unable to invoke them at all while it answers as though the capabilities never existed. Dropping Google Search only costs grounding, so the model still answers, just without citations. One is disabled, the other is degraded. Set `include_server_side_tool_invocations` to send both. A lone Google Search tool still converts back correctly, and `retrievalConfig` is only emitted when a search tool actually survived conversion
+  <Warning>
+  Breaking on the Gemini API surface: a request carrying both function declarations and Google Search without `include_server_side_tool_invocations` previously kept Google Search and dropped the function declarations. It now does the opposite. Set `include_server_side_tool_invocations` to `true` to send both, which is supported on Gemini 3 models. Vertex is unaffected, since it accepts the combination natively and drops neither.
+  </Warning>
+- fix: always emit a Gemini candidate carrying its finish reason on `generateContent`, even when nothing visible was generated - a thinking model that spends its whole output budget before emitting a token is a successful 200 with an empty answer, but `Candidates` is `omitempty`, so dropping that candidate produced a body with no `candidates` key at all and left a lone `usageMetadata` object that every Gemini-shaped client dereferences blind
+- fix: drop payload-free Gemini parts when assembling a candidate - every `Part` field is `omitempty`, so such a part marshals to exactly `{}`; the harness observed one on the wire when a transcription request for an unintelligible tone came back as `parts:[{}]`, where it is noise a client will try to read and it masks the contentless case by making the parts slice look non-empty
+- fix: accept a bare model identifier on Bedrock rerank by synthesizing the foundation-model ARN from the resolved region - Rerank is the one Bedrock surface that names its model by ARN rather than by bare ID, so all three rerank drop-ins in the provider harness 400'd on `amazon.rerank-v1:0`. The partition is derived from the region (`aws`, `aws-cn`, `aws-us-gov`) so GovCloud and China build a correct ARN, and an explicit ARN still passes through untouched
+- fix: stop stripping `file_url` from OpenAI-shaped chat file blocks on marshal - dropping it produced `{"type":"file","file":{}}` and an upstream complaint about a missing `file_id`, which hid the fact that a source had been discarded. Providers that cannot take a URL now say so by name, and any OpenAI-compatible endpoint that does accept one keeps working without a Bifrost change
+- fix: leave URL content sources Bifrost cannot download in place on the OpenAI and native-Anthropic paths instead of failing the request - only `http(s)` is fetched, and whether a `gs://`, `s3://` or scheme-less reference is usable is the provider's call, so the source now travels as `{"type":"url"}` and the platform answers for itself