diff --git a/docs/contributing/environment-variables.md b/docs/contributing/environment-variables.md index aa4006d843..e8343a3e5f 100644 --- a/docs/contributing/environment-variables.md +++ b/docs/contributing/environment-variables.md @@ -91,23 +91,21 @@ Optional Worker secret: dev uses offline search while `WRANGLER_IS_LOCAL_DEV` is set or the binding is missing. -## Cloudflare API (Worker + Browser Rendering + Email) +## Cloudflare API (Worker + Email) Optional Worker secrets/vars (see `packages/worker/src/env-schema.ts` and `packages/worker/src/mcp/cloudflare/cloudflare-rest-client.ts`): - `CLOUDFLARE_API_TOKEN` — Cloudflare API token used by the internal API client - (`Authorization: Bearer ...`) for `page_to_markdown` Browser Rendering and the - Cloudflare Email sender. User Cloudflare API calls from codemode use saved - secrets and secret-aware `fetch` (see - `docs/contributing/skill-patterns/cloudflare-api-v4.md`). Local `npm run dev` - sets this to the Cloudflare mock token unless `AI_MODE=remote` or - `SKIP_CLOUDFLARE_MOCK=1`; when unset and no mock is attached, the billed - `page_to_markdown` Browser Rendering fallback fails fast with a setup hint. -- `CLOUDFLARE_ACCOUNT_ID` — Cloudflare account id required by the - `page_to_markdown` capability when it falls back to Browser Rendering - `POST /client/v4/accounts/{account_id}/browser-rendering/markdown`. This is a - Worker var (not a secret) and should match the account behind + (`Authorization: Bearer ...`) for Worker-side Cloudflare REST calls such as + the Cloudflare Email sender. User + Cloudflare API calls from codemode use saved secrets and secret-aware `fetch` + (see `docs/contributing/skill-patterns/cloudflare-api-v4.md`). Local + `npm run dev` sets this to the Cloudflare mock token unless + `AI_MODE=remote` or `SKIP_CLOUDFLARE_MOCK=1`. +- `CLOUDFLARE_ACCOUNT_ID` — Cloudflare account id required by the Cloudflare + Email Service REST API fallback used by local mocks and preview deploys. This + is a Worker var (not a secret) and should match the account behind `CLOUDFLARE_API_TOKEN`. - `CLOUDFLARE_API_BASE_URL` — API base URL; defaults to `https://api.cloudflare.com` when unset, including for outbound email sending. diff --git a/docs/contributing/mock-api-servers.md b/docs/contributing/mock-api-servers.md index 3e4785c61c..472267f6bd 100644 --- a/docs/contributing/mock-api-servers.md +++ b/docs/contributing/mock-api-servers.md @@ -17,8 +17,8 @@ Worker package so it can also be deployed alongside the main app. and set `ACME_API_BASE_URL` to the mock Worker origin. See `packages/mock-servers/cloudflare/` for a small Cloudflare API v4 subset -mock used by tests and `page_to_markdown` / internal API client (started by -`npm run dev` unless `SKIP_CLOUDFLARE_MOCK=1`). +mock used by tests and the internal API client (started by `npm run dev` +unless `SKIP_CLOUDFLARE_MOCK=1`). ### Tips diff --git a/docs/contributing/setup-manifest.md b/docs/contributing/setup-manifest.md index 9d9e642dd4..95b0c903c4 100644 --- a/docs/contributing/setup-manifest.md +++ b/docs/contributing/setup-manifest.md @@ -72,9 +72,8 @@ automatically: Cloudflare AI Gateway) - `CLOUDFLARE_ACCOUNT_ID` (required for local development when `AI_MODE=remote` so Wrangler can authenticate Workers AI requests against the correct account; - also required when using the `page_to_markdown` capability's Cloudflare - Browser Rendering fallback against the live API and for the Cloudflare Email - Service REST API fallback used by local mocks and preview deploys) + also required for the Cloudflare Email Service REST API fallback used by + local mocks and preview deploys) - `CLOUDFLARE_API_TOKEN` (required for local development when `AI_MODE=remote` so Wrangler can authenticate Workers AI requests; also reused by the Cloudflare Email Service REST API fallback when local/preview email is routed @@ -106,7 +105,8 @@ secrets from `packages/worker/.env`. Configure these GitHub Actions secrets and variables for workflows: - `CLOUDFLARE_API_TOKEN` (Workers deploy + D1 edit access on the correct - account) + account; also reused for remote AI and Cloudflare API workflows that run with + account secrets + skills) - `COOKIE_SECRET` (same format as local) - `APP_BASE_URL` (optional GitHub Actions **variable**, used by the production deploy as the canonical public app origin and written into the generated @@ -136,7 +136,10 @@ How to get/set each value: - `CLOUDFLARE_API_TOKEN` - In Cloudflare Dashboard, create an API Token with permissions to deploy - Workers and edit D1 on the target account. + Workers and edit D1 on the target account. This is the same token to reuse + for remote AI and Cloudflare API workflows that run with account secrets + + skills; when you do, also include the product permissions needed for those + APIs. - In GitHub: `Settings` → `Secrets and variables` → `Actions` → `New repository secret`. - `COOKIE_SECRET` @@ -174,13 +177,6 @@ How to get/set each value: - In GitHub: **Settings → Secrets and variables → Actions → Variables**, add `SENTRY_ORG` and `SENTRY_PROJECT` with your Sentry slugs (for example from `npx @sentry/wizard@latest -i sourcemaps`). -- `CLOUDFLARE_API_TOKEN` (required for remote AI; used by the Worker for - `page_to_markdown` Browser Rendering; user API calls use account secrets + - skills) - - Create a Cloudflare API token with the account permissions needed for the - product APIs you want to call. This same secret already powers production - deploys and the `page_to_markdown` Browser Rendering path. For Browser - Rendering fallback, include the **Browser Rendering - Edit** permission. - `CAPABILITY_REINDEX_SECRET` (optional) - Generate a long random secret (for example `openssl rand -hex 32`), store it as the repository secret `CAPABILITY_REINDEX_SECRET`, and let the deploy diff --git a/docs/contributing/setup.md b/docs/contributing/setup.md index 26706bf4f6..d0a05b41f0 100644 --- a/docs/contributing/setup.md +++ b/docs/contributing/setup.md @@ -35,8 +35,8 @@ Quick notes for getting a local kody environment running. - `npm run dev` (starts mock API servers automatically, the main worker, and the local home connector; it sets `AI_MODE=mock`, `AI_MOCK_BASE_URL`, and `CLOUDFLARE_API_BASE_URL` + `CLOUDFLARE_API_TOKEN` + `CLOUDFLARE_ACCOUNT_ID` - to the local Cloudflare API mock Worker for `page_to_markdown`, the internal - Cloudflare API client, and local email sending. Unless you already set + to the local Cloudflare API mock Worker for the internal Cloudflare API client + and local email sending. Unless you already set `CLOUDFLARE_EMAIL_FROM`, the launcher also defaults it to `reset@kody.dev`. Set `SKIP_CLOUDFLARE_MOCK=1` to skip the local Cloudflare mock entirely. The home connector receives the resolved worker origin via `WORKER_BASE_URL`. When diff --git a/docs/contributing/skill-patterns/cloudflare-api-v4.md b/docs/contributing/skill-patterns/cloudflare-api-v4.md index b6ebed3313..ea086b123e 100644 --- a/docs/contributing/skill-patterns/cloudflare-api-v4.md +++ b/docs/contributing/skill-patterns/cloudflare-api-v4.md @@ -25,8 +25,6 @@ Capability **allowlists** on secrets apply to **capability inputs** that use All API paths must be under **`/client/v4/`** (see [Cloudflare API docs](https://developers.cloudflare.com/fundamentals/api/how-to/make-api-calls/)). -The internal `CloudflareRestClient` used for Browser Rendering enforces the same -shape. ## Example skill (save via `meta_save_skill`) @@ -61,7 +59,5 @@ For **POST / PUT / PATCH / DELETE**, set `Content-Type: application/json` and ## Related -- Browser Rendering billing path still uses the Worker’s internal client for - `page_to_markdown` — that is unchanged. - Developer docs (read-only) pattern: [cloudflare-developer-docs.md](./cloudflare-developer-docs.md). diff --git a/docs/contributing/skill-patterns/cloudflare-developer-docs.md b/docs/contributing/skill-patterns/cloudflare-developer-docs.md index 5e404600b0..a51e49a58c 100644 --- a/docs/contributing/skill-patterns/cloudflare-developer-docs.md +++ b/docs/contributing/skill-patterns/cloudflare-developer-docs.md @@ -1,22 +1,15 @@ # Skill pattern: Cloudflare Developer Docs (`developers.cloudflare.com`) -The **`cloudflare_api_docs`** builtin capability was removed in favor of **saved -skills** (or one-off **`execute`** code) so Cloudflare-specific doc reading can -evolve without shipping Worker changes. - Use this pattern when you need API or product documentation from [developers.cloudflare.com](https://developers.cloudflare.com). -## Prefer cheaper steps first +## Preferred approach -1. **Markdown-preferred `fetch`** against a path under the docs site (see - allowlist below). Matches what the old builtin did before any billed - fallback. -2. If the response is still hard to use (heavy HTML, etc.), call - **`codemode.page_to_markdown`** with the same URL so Browser Rendering runs - only when needed (billed). +Use **markdown-preferred `fetch`** against a path under the docs site (see +allowlist below). Keep the skill focused on direct docs retrieval instead of +adding site-specific fallback machinery. -## Path allowlist (same policy as the former builtin) +## Path allowlist Paths must: @@ -77,26 +70,23 @@ Adjust `name`, `title`, `description`, and trust flags when saving. const url = new URL(path, ORIGIN).toString() const res = await fetch(url, { headers: { Accept: MARKDOWN_ACCEPT } }) const body = await res.text() - const contentType = res.headers.get('content-type') - const needsFallback = - contentType?.includes('text/html') && body.trimStart().startsWith('<') - if (needsFallback) { - return await codemode.page_to_markdown({ url }) - } return { status: res.status, - contentType, + contentType: res.headers.get('content-type'), markdownTokenEstimate: res.headers.get('x-markdown-tokens'), body: body.slice(0, 500_000), } } ``` +Callers should inspect `contentType` before treating `body` as Markdown. This +helper returns the raw sliced response body plus `markdownTokenEstimate`; HTML +responses are not auto-converted. + For parameterized skills, use **`meta_save_skill`** **`parameters`** (e.g. a required `path` string) and read **`params.path`** instead of a hard-coded path. ## Related -- Billed fallback: **`page_to_markdown`** (`packages/worker` coding domain). - Tracking: [issue #120](https://github.com/kentcdodds/kody/issues/120) (broader official patterns folder). diff --git a/docs/contributing/skill-patterns/index.md b/docs/contributing/skill-patterns/index.md index a1626a2235..6cf0e9bdec 100644 --- a/docs/contributing/skill-patterns/index.md +++ b/docs/contributing/skill-patterns/index.md @@ -7,4 +7,4 @@ Worker-only primitives (billing bindings, secret storage, etc.). | Pattern | Summary | | -------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------- | | [cloudflare-api-v4.md](./cloudflare-api-v4.md) | Call Cloudflare API v4 with secret-aware `fetch` to `api.cloudflare.com` (replaces removed `cloudflare_rest`). | -| [cloudflare-developer-docs.md](./cloudflare-developer-docs.md) | Fetch markdown from `developers.cloudflare.com` with an allowlisted path; optional `page_to_markdown` fallback. | +| [cloudflare-developer-docs.md](./cloudflare-developer-docs.md) | Fetch markdown from `developers.cloudflare.com` with an allowlisted path and return a bounded preview. | diff --git a/docs/use/search.md b/docs/use/search.md index fb004cc02c..4f1a2855a8 100644 --- a/docs/use/search.md +++ b/docs/use/search.md @@ -31,7 +31,7 @@ To get **full markdown and schemas for one hit** (for example a capability’s Examples: -- `page_to_markdown:capability` +- `kody_official_guide:capability` - `user:preferred_org:value` - `github:connector` - `my-skill-name:skill` diff --git a/packages/mock-servers/cloudflare/src/worker.ts b/packages/mock-servers/cloudflare/src/worker.ts index e92f2b6c82..9270930cc0 100644 --- a/packages/mock-servers/cloudflare/src/worker.ts +++ b/packages/mock-servers/cloudflare/src/worker.ts @@ -30,14 +30,6 @@ type DnsRecord = { ttl: number } -type BrowserRenderingMarkdownBody = { - url?: unknown - html?: unknown - userAgent?: unknown - rejectRequestPattern?: unknown - gotoOptions?: unknown -} - type DashboardEndpoint = { method: string path: string @@ -98,12 +90,6 @@ const dashboardEndpoints: Array = [ description: 'Create DNS record', requiresAuth: true, }, - { - method: 'POST', - path: `/client/v4/accounts/${fixtureAccount.id}/browser-rendering/markdown`, - description: 'Convert a page or HTML snippet to markdown', - requiresAuth: true, - }, { method: 'POST', path: `/client/v4/accounts/${fixtureAccount.id}/email/sending/send`, @@ -532,24 +518,6 @@ async function handleEmailSend( } async function routeApi(request: Request, env: MockCloudflareEnv, url: URL) { - if (request.method === 'GET' && url.pathname === '/__mocks/markdown') { - return new Response('# Mock markdown\n\nServed as markdown.\n', { - headers: { - 'content-type': 'text/markdown; charset=utf-8', - 'x-markdown-tokens': '8', - }, - }) - } - if (request.method === 'GET' && url.pathname === '/__mocks/markdown-error') { - return new Response('# Mock markdown error\n\nServer error page.\n', { - status: 500, - headers: { - 'content-type': 'text/markdown; charset=utf-8', - 'x-markdown-tokens': '9', - }, - }) - } - if (!isAuthorized(request, env, url)) { return errorEnvelope(401, 10000, 'Authentication error') } @@ -596,43 +564,6 @@ async function routeApi(request: Request, env: MockCloudflareEnv, url: URL) { ) } - const markdownMatch = url.pathname.match( - /^\/client\/v4\/accounts\/([^/]+)\/browser-rendering\/markdown\/?$/, - ) - if (markdownMatch && request.method === 'POST') { - const accountId = markdownMatch[1]! - const payload = (await readJsonBody( - request, - )) as BrowserRenderingMarkdownBody | null - if (payload === null) { - return errorEnvelope(400, 1001, 'invalid JSON body') - } - if (accountId !== fixtureAccount.id) { - return errorEnvelope(404, 1002, 'account not found') - } - const hasUrl = - typeof payload.url === 'string' && payload.url.trim().length > 0 - const hasHtml = - typeof payload.html === 'string' && payload.html.trim().length > 0 - if (!hasUrl && !hasHtml) { - return errorEnvelope(400, 1003, 'Either url or html is required.') - } - const mode = hasHtml ? 'html' : 'url' - const sourceValue = hasHtml - ? String(payload.html).trim() - : String(payload.url).trim() - const markdown = [ - '# Mock Browser Rendering', - '', - `mode: ${mode}`, - `source: ${sourceValue}`, - ...(typeof payload.userAgent === 'string' && payload.userAgent.length > 0 - ? [`userAgent: ${payload.userAgent}`] - : []), - ].join('\n') - return envelope(markdown, { status: 200 }) - } - const emailMatch = url.pathname.match( /^\/client\/v4\/accounts\/([^/]+)\/email\/sending\/send\/?$/, ) diff --git a/packages/worker/.env.example b/packages/worker/.env.example index a4166f1491..56bf7e0923 100644 --- a/packages/worker/.env.example +++ b/packages/worker/.env.example @@ -29,10 +29,9 @@ CLOUDFLARE_EMAIL_FROM=reset@kody.dev # SENTRY_ORG=your-org-slug # SENTRY_PROJECT=your-project-slug -# Cloudflare API / Browser Rendering / Email (optional; defaults to the public Cloudflare API, and `npm run dev` sets a mock URL + token for `page_to_markdown`, local email, and the internal API client) +# Cloudflare API / Email (optional; defaults to the public Cloudflare API, and `npm run dev` sets a mock URL + token for local email and the internal API client) # Uses Authorization: Bearer against /client/v4/* endpoints. # Reuse `CLOUDFLARE_API_TOKEN` for the live API. In mock mode `npm run dev` injects a local token. -# page_to_markdown also needs CLOUDFLARE_ACCOUNT_ID when it falls back to Cloudflare Browser Rendering `/markdown`. # CLOUDFLARE_API_BASE_URL=https://api.cloudflare.com # Skip starting the Cloudflare mock during dev (use live API + token in `packages/worker/.env` instead): # SKIP_CLOUDFLARE_MOCK=1 diff --git a/packages/worker/src/mcp/capabilities/coding/coding-capabilities.node.test.ts b/packages/worker/src/mcp/capabilities/coding/coding-capabilities.node.test.ts index edc4488482..59f435e2ba 100644 --- a/packages/worker/src/mcp/capabilities/coding/coding-capabilities.node.test.ts +++ b/packages/worker/src/mcp/capabilities/coding/coding-capabilities.node.test.ts @@ -8,7 +8,6 @@ import { wranglerBin, } from '#mcp/test-process.ts' import { createCloudflareRestClient } from '#mcp/cloudflare/cloudflare-rest-client.ts' -import { pageToMarkdownCapability } from './page-to-markdown.ts' const cloudflareWorkerConfig = 'packages/mock-servers/cloudflare/wrangler.jsonc' const projectRoot = process.cwd() @@ -127,130 +126,3 @@ test( ).rejects.toThrow('path must start with `/client/v4/`') }, ) - -test( - 'page_to_markdown returns negotiated markdown without Browser Rendering', - wranglerMockTimeout, - async () => { - const token = 'coding-cloudflare-page-negotiated-token' - await using mock = await startCloudflareMock(token) - const ctx = mockCloudflareContext(mock.origin, mock.token) - const result = await pageToMarkdownCapability.handler( - { - url: `${mock.origin}/__mocks/markdown`, - }, - ctx, - ) - expect(result.source).toBe('negotiated') - expect(result.browserRendering).toBeNull() - expect(result.negotiated?.contentType).toContain('text/markdown') - expect(result.markdown).toContain('# Mock markdown') - }, -) - -test( - 'page_to_markdown does not treat markdown error responses as negotiated success', - wranglerMockTimeout, - async () => { - const token = 'coding-cloudflare-page-error-token' - await using mock = await startCloudflareMock(token) - const ctx = mockCloudflareContext(mock.origin, mock.token) - const result = await pageToMarkdownCapability.handler( - { - url: `${mock.origin}/__mocks/markdown-error`, - }, - ctx, - ) - expect(result.source).toBe('browser_rendering') - expect(result.negotiated?.status).toBe(500) - expect(result.negotiated?.contentType).toContain('text/markdown') - expect(result.browserRendering).toEqual({ - apiStatus: 200, - mode: 'url', - }) - expect(result.markdown).toContain('# Mock Browser Rendering') - expect(result.markdown).toContain( - `source: ${mock.origin}/__mocks/markdown-error`, - ) - }, -) - -test( - 'page_to_markdown falls back to Browser Rendering for html pages', - wranglerMockTimeout, - async () => { - const token = 'coding-cloudflare-page-markdown-token' - await using mock = await startCloudflareMock(token) - const ctx = mockCloudflareContext(mock.origin, mock.token) - const result = await pageToMarkdownCapability.handler( - { - url: `${mock.origin}/__mocks`, - userAgent: 'kody-test-agent', - }, - ctx, - ) - expect(result.source).toBe('browser_rendering') - expect(result.negotiated?.contentType).toContain('text/html') - expect(result.browserRendering).toEqual({ - apiStatus: 200, - mode: 'url', - }) - expect(result.markdown).toContain('# Mock Browser Rendering') - expect(result.markdown).toContain(`source: ${mock.origin}/__mocks`) - expect(result.markdown).toContain('userAgent: kody-test-agent') - }, -) - -test( - 'page_to_markdown converts inline html with Browser Rendering', - wranglerMockTimeout, - async () => { - const token = 'coding-cloudflare-page-inline-token' - await using mock = await startCloudflareMock(token) - const ctx = mockCloudflareContext(mock.origin, mock.token) - const result = await pageToMarkdownCapability.handler( - { - html: '

Hello

From HTML

', - }, - ctx, - ) - expect(result.source).toBe('browser_rendering') - expect(result.url).toBeNull() - expect(result.negotiated).toBeNull() - expect(result.browserRendering).toEqual({ - apiStatus: 200, - mode: 'html', - }) - expect(result.markdown).toContain('mode: html') - expect(result.markdown).toContain( - 'source:

Hello

From HTML

', - ) - }, -) - -test( - 'page_to_markdown fails clearly when Browser Rendering account id is missing', - wranglerMockTimeout, - async () => { - const token = 'coding-cloudflare-page-no-account-token' - await using mock = await startCloudflareMock(token) - const ctx = { - env: { - CLOUDFLARE_API_TOKEN: token, - CLOUDFLARE_API_BASE_URL: mock.origin, - } as Env, - callerContext: { - baseUrl: 'http://localhost:3742', - user: null, - }, - } - await expect( - pageToMarkdownCapability.handler( - { - url: `${mock.origin}/__mocks`, - }, - ctx, - ), - ).rejects.toThrow('CLOUDFLARE_ACCOUNT_ID is not set') - }, -) diff --git a/packages/worker/src/mcp/capabilities/coding/domain.ts b/packages/worker/src/mcp/capabilities/coding/domain.ts index ed6e04e647..50e30a353c 100644 --- a/packages/worker/src/mcp/capabilities/coding/domain.ts +++ b/packages/worker/src/mcp/capabilities/coding/domain.ts @@ -1,11 +1,10 @@ import { defineDomain } from '../define-domain.ts' import { capabilityDomainNames } from '../domain-metadata.ts' import { kodyOfficialGuideCapability } from './kody-official-guide.ts' -import { pageToMarkdownCapability } from './page-to-markdown.ts' export const codingDomain = defineDomain({ name: capabilityDomainNames.coding, description: - 'Software work such as billed page-to-markdown fallback for hard-to-read web pages, official Kody guides from the repository, and coding-agent workflows. Prefer normal fetch, browser tools, or saved skills before the billed fallback. For Cloudflare API v4 and developers.cloudflare.com doc patterns, see docs/contributing/skill-patterns/.', - capabilities: [kodyOfficialGuideCapability, pageToMarkdownCapability], + 'Software work such as official Kody guides from the repository and coding-agent workflows. For Cloudflare API v4 and developers.cloudflare.com doc patterns, see docs/contributing/skill-patterns/.', + capabilities: [kodyOfficialGuideCapability], }) diff --git a/packages/worker/src/mcp/capabilities/coding/fetch-page-markdown.ts b/packages/worker/src/mcp/capabilities/coding/fetch-page-markdown.ts deleted file mode 100644 index 4e5ab75c10..0000000000 --- a/packages/worker/src/mcp/capabilities/coding/fetch-page-markdown.ts +++ /dev/null @@ -1,215 +0,0 @@ -import { createCloudflareRestClient } from '#mcp/cloudflare/cloudflare-rest-client.ts' -import { fetchMarkdownPreferredDoc } from './fetch-markdown-doc.ts' - -const maxInlineHtmlChars = 2_000_000 - -export type PageToMarkdownInput = { - url?: string - html?: string - userAgent?: string - rejectRequestPattern?: Array - gotoOptions?: { - waitUntil?: 'load' | 'domcontentloaded' | 'networkidle0' | 'networkidle2' - } -} - -export type PageToMarkdownResult = { - source: 'negotiated' | 'browser_rendering' - markdown: string - url: string | null - negotiated: { - status: number - contentType: string | null - markdownTokenEstimate: string | null - } | null - browserRendering: { - apiStatus: number - mode: 'url' | 'html' - } | null -} - -function readContentTypeMediaType(contentType: string | null) { - return contentType?.split(';', 1)[0]?.trim().toLowerCase() ?? null -} - -function shouldUseNegotiatedContent(input: { - status: number - contentType: string | null - markdownTokenEstimate: string | null -}) { - if (input.status < 200 || input.status >= 300) return false - const mediaType = readContentTypeMediaType(input.contentType) - if (mediaType === 'text/markdown' || mediaType === 'text/plain') return true - return input.markdownTokenEstimate != null -} - -function readConfiguredCloudflareAccountId( - env: Pick, -) { - const accountId = env.CLOUDFLARE_ACCOUNT_ID?.trim() - if (!accountId) { - throw new Error( - 'CLOUDFLARE_ACCOUNT_ID is not set. page_to_markdown uses Cloudflare Browser Rendering as a billed fallback when negotiation does not return markdown or plain text.', - ) - } - return accountId -} - -type BrowserRenderingResponseBody = { - success?: boolean - result?: unknown - errors?: Array<{ message?: string }> -} - -function readBrowserRenderingErrorMessage(body: BrowserRenderingResponseBody) { - const messages = Array.isArray(body.errors) - ? body.errors - .map((error) => error?.message?.trim()) - .filter((message): message is string => Boolean(message)) - : [] - return messages.join('; ') -} - -async function convertWithBrowserRendering( - env: Pick< - Env, - 'CLOUDFLARE_API_TOKEN' | 'CLOUDFLARE_API_BASE_URL' | 'CLOUDFLARE_ACCOUNT_ID' - >, - input: { - url?: string - html?: string - userAgent?: string - rejectRequestPattern?: Array - gotoOptions?: { - waitUntil?: 'load' | 'domcontentloaded' | 'networkidle0' | 'networkidle2' - } - }, -) { - const accountId = readConfiguredCloudflareAccountId(env) - const client = createCloudflareRestClient(env) - const response = await client.rawRequest({ - method: 'POST', - path: `/client/v4/accounts/${accountId}/browser-rendering/markdown`, - body: { - ...(input.url ? { url: input.url } : {}), - ...(input.html ? { html: input.html } : {}), - ...(input.userAgent ? { userAgent: input.userAgent } : {}), - ...(input.rejectRequestPattern - ? { rejectRequestPattern: input.rejectRequestPattern } - : {}), - ...(input.gotoOptions ? { gotoOptions: input.gotoOptions } : {}), - }, - }) - const body = (response.body ?? {}) as BrowserRenderingResponseBody - if (body.success !== true || typeof body.result !== 'string') { - const details = readBrowserRenderingErrorMessage(body) - throw new Error( - details.length > 0 - ? `Cloudflare Browser Rendering markdown failed (${response.status}): ${details}` - : `Cloudflare Browser Rendering markdown failed (${response.status}).`, - ) - } - return { - apiStatus: response.status, - markdown: body.result, - mode: input.html ? ('html' as const) : ('url' as const), - } -} - -export function assertSafePageToMarkdownUrl(url: string) { - const trimmed = url.trim() - if (!trimmed) { - throw new Error('url cannot be empty.') - } - if (/[\s]/.test(trimmed)) { - throw new Error('url must not contain whitespace.') - } - if (trimmed.length > 2048) { - throw new Error('url exceeds maximum length.') - } - const parsed = new URL(trimmed) - if (parsed.username || parsed.password) { - throw new Error('url must not include embedded credentials.') - } - if (parsed.protocol !== 'http:' && parsed.protocol !== 'https:') { - throw new Error('url must use http or https.') - } - return parsed.toString() -} - -export function assertSafePageToMarkdownHtml(html: string) { - const trimmed = html.trim() - if (!trimmed) { - throw new Error('html cannot be empty.') - } - if (trimmed.length > maxInlineHtmlChars) { - throw new Error( - `html exceeds maximum length of ${maxInlineHtmlChars} characters.`, - ) - } - return trimmed -} - -export async function fetchNegotiatedThenMaybeBrowserRender( - env: Pick< - Env, - 'CLOUDFLARE_API_TOKEN' | 'CLOUDFLARE_API_BASE_URL' | 'CLOUDFLARE_ACCOUNT_ID' - >, - input: PageToMarkdownInput, -): Promise { - if (input.html) { - const browserRendering = await convertWithBrowserRendering(env, { - html: assertSafePageToMarkdownHtml(input.html), - userAgent: input.userAgent, - rejectRequestPattern: input.rejectRequestPattern, - gotoOptions: input.gotoOptions, - }) - return { - source: 'browser_rendering', - markdown: browserRendering.markdown, - url: null, - negotiated: null, - browserRendering: { - apiStatus: browserRendering.apiStatus, - mode: browserRendering.mode, - }, - } - } - - const safeUrl = assertSafePageToMarkdownUrl(input.url ?? '') - const negotiated = await fetchMarkdownPreferredDoc(safeUrl) - if (shouldUseNegotiatedContent(negotiated)) { - return { - source: 'negotiated', - markdown: negotiated.body, - url: safeUrl, - negotiated: { - status: negotiated.status, - contentType: negotiated.contentType, - markdownTokenEstimate: negotiated.markdownTokenEstimate, - }, - browserRendering: null, - } - } - - const browserRendering = await convertWithBrowserRendering(env, { - url: safeUrl, - userAgent: input.userAgent, - rejectRequestPattern: input.rejectRequestPattern, - gotoOptions: input.gotoOptions, - }) - return { - source: 'browser_rendering', - markdown: browserRendering.markdown, - url: safeUrl, - negotiated: { - status: negotiated.status, - contentType: negotiated.contentType, - markdownTokenEstimate: negotiated.markdownTokenEstimate, - }, - browserRendering: { - apiStatus: browserRendering.apiStatus, - mode: browserRendering.mode, - }, - } -} diff --git a/packages/worker/src/mcp/capabilities/coding/index.ts b/packages/worker/src/mcp/capabilities/coding/index.ts index d76ad28c86..0049b0657f 100644 --- a/packages/worker/src/mcp/capabilities/coding/index.ts +++ b/packages/worker/src/mcp/capabilities/coding/index.ts @@ -1,6 +1,5 @@ import { codingDomain } from './domain.ts' export { codingDomain } from './domain.ts' -export { pageToMarkdownCapability } from './page-to-markdown.ts' export const codingCapabilities = codingDomain.capabilities diff --git a/packages/worker/src/mcp/capabilities/coding/page-to-markdown.ts b/packages/worker/src/mcp/capabilities/coding/page-to-markdown.ts deleted file mode 100644 index 739a52203f..0000000000 --- a/packages/worker/src/mcp/capabilities/coding/page-to-markdown.ts +++ /dev/null @@ -1,112 +0,0 @@ -import { z } from 'zod' -import { defineDomainCapability } from '../define-domain-capability.ts' -import { capabilityDomainNames } from '../domain-metadata.ts' -import { type CapabilityContext } from '../types.ts' -import { fetchNegotiatedThenMaybeBrowserRender } from './fetch-page-markdown.ts' - -const gotoWaitUntilSchema = z.enum([ - 'load', - 'domcontentloaded', - 'networkidle0', - 'networkidle2', -]) - -const inputSchema = z - .object({ - url: z - .string() - .optional() - .describe( - 'Page URL to read as markdown. Prefer your existing web-reading tools first (for example normal fetch with `Accept: text/markdown`, browser tools, or a saved docs-fetch skill) and use this only as a fallback when they return unhelpful HTML or cannot load the page.', - ), - html: z - .string() - .optional() - .describe( - 'Optional raw HTML to convert to markdown. Use this only when you already have HTML and cheaper tools cannot give you markdown directly.', - ), - userAgent: z - .string() - .optional() - .describe('Optional Browser Rendering user agent override.'), - rejectRequestPattern: z - .array(z.string()) - .optional() - .describe( - 'Optional Browser Rendering request-block regex patterns, for example to skip CSS.', - ), - gotoOptions: z - .object({ - waitUntil: gotoWaitUntilSchema - .optional() - .describe( - 'Optional Browser Rendering waitUntil strategy for JS-heavy pages.', - ), - }) - .optional() - .describe( - 'Optional Browser Rendering navigation controls. Only used when the billed fallback runs.', - ), - }) - .refine((value) => Boolean(value.url) !== Boolean(value.html), { - message: 'Provide exactly one of `url` or `html`.', - path: ['url'], - }) - -const outputSchema = z.object({ - source: z - .enum(['negotiated', 'browser_rendering']) - .describe( - 'Whether the result came from the cheap markdown-preferred fetch or the billed Browser Rendering fallback.', - ), - markdown: z.string().describe('Final markdown or plain-text result.'), - url: z - .string() - .nullable() - .describe('Final normalized URL, or null when converting inline HTML.'), - negotiated: z - .object({ - status: z.number(), - contentType: z.string().nullable(), - markdownTokenEstimate: z.string().nullable(), - }) - .nullable() - .describe( - 'Negotiated fetch metadata. Present when a URL was fetched before deciding whether fallback was needed.', - ), - browserRendering: z - .object({ - apiStatus: z.number(), - mode: z.enum(['url', 'html']), - }) - .nullable() - .describe( - 'Browser Rendering metadata. Present only when the billed fallback was used.', - ), -}) - -export const pageToMarkdownCapability = defineDomainCapability( - capabilityDomainNames.coding, - { - name: 'page_to_markdown', - description: - 'Generic page-to-markdown helper. Try cheaper web-reading mechanisms first (normal fetch with `Accept: text/markdown`, browser/IDE tools, or a saved docs-fetch skill) and use this only as a fallback when they return useless HTML or cannot load a page. This capability first does a normal markdown-preferred fetch; only if that still yields HTML does it call billed Cloudflare Browser Rendering `/markdown`.', - keywords: [ - 'markdown', - 'html to markdown', - 'web page', - 'browser rendering', - 'fallback', - 'generic', - 'content extraction', - ], - readOnly: true, - idempotent: true, - destructive: false, - inputSchema, - outputSchema, - async handler(args, ctx: CapabilityContext) { - return fetchNegotiatedThenMaybeBrowserRender(ctx.env, args) - }, - }, -) diff --git a/packages/worker/src/mcp/cloudflare/cloudflare-rest-client.ts b/packages/worker/src/mcp/cloudflare/cloudflare-rest-client.ts index 81efc9cd22..5e4c9be6f2 100644 --- a/packages/worker/src/mcp/cloudflare/cloudflare-rest-client.ts +++ b/packages/worker/src/mcp/cloudflare/cloudflare-rest-client.ts @@ -1,7 +1,7 @@ /** - * Minimal Cloudflare API v4 HTTP client (Bearer token). Used by - * `page_to_markdown` Browser Rendering and tests. User-facing Cloudflare API - * access is via saved skills and secret-aware `fetch`; see + * Minimal Cloudflare API v4 HTTP client (Bearer token) for internal Worker-side + * Cloudflare API calls and tests. User-facing Cloudflare API access is via + * saved skills and secret-aware `fetch`; see * `docs/contributing/skill-patterns/cloudflare-api-v4.md`. * * @see https://developers.cloudflare.com/fundamentals/api/how-to/make-api-calls/ diff --git a/packages/worker/src/mcp/server-instructions.ts b/packages/worker/src/mcp/server-instructions.ts index 7347d1a551..eaee0e6841 100644 --- a/packages/worker/src/mcp/server-instructions.ts +++ b/packages/worker/src/mcp/server-instructions.ts @@ -42,7 +42,7 @@ search - Examples: - search({ query: 'saved dashboard app or generated UI runtime' }) - search({ query: 'Cloudflare API zones dns workers d1' }) - - search({ entity: 'page_to_markdown:capability' }) + - search({ entity: 'kody_official_guide:capability' }) execute - Async arrow function; \`codemode\` + OAuth helpers \`refreshAccessToken\` / \`createAuthenticatedFetch\`. Prefer one \`execute\` when the plan is clear. Full rules for \`fetch\`, placeholders, \`secret_list\` / \`value_get\`, and \`x-kody-secret\`: see the \`execute\` tool description. diff --git a/packages/worker/src/mcp/skills/infer-codemode-capabilities.node.test.ts b/packages/worker/src/mcp/skills/infer-codemode-capabilities.node.test.ts index ec08e711b3..d834a918bb 100644 --- a/packages/worker/src/mcp/skills/infer-codemode-capabilities.node.test.ts +++ b/packages/worker/src/mcp/skills/infer-codemode-capabilities.node.test.ts @@ -19,9 +19,9 @@ test('infers static codemode member names', () => { }) test('infers bracket string literal access', () => { - const src = `async () => await codemode['page_to_markdown']({ url: 'https://example.com/doc' })` + const src = `async () => await codemode['kody_official_guide']({ guide: 'oauth' })` const { staticNames } = inferCodemodeCapabilities(src) - expect(staticNames).toContain('page_to_markdown') + expect(staticNames).toContain('kody_official_guide') }) test('infers secret_set member access', () => { diff --git a/packages/worker/src/mcp/tools/execute.node.test.ts b/packages/worker/src/mcp/tools/execute.node.test.ts index 4e15976712..81ad9c6738 100644 --- a/packages/worker/src/mcp/tools/execute.node.test.ts +++ b/packages/worker/src/mcp/tools/execute.node.test.ts @@ -5,7 +5,7 @@ const mockModule = vi.hoisted(() => ({ runCodemodeWithRegistry: vi.fn(), getCapabilityRegistryForContext: vi.fn(async () => ({ capabilityHandlers: { - page_to_markdown: true, + kody_official_guide: true, }, })), })) diff --git a/packages/worker/src/mcp/tools/execute.ts b/packages/worker/src/mcp/tools/execute.ts index 5f4e4683ea..afd00be324 100644 --- a/packages/worker/src/mcp/tools/execute.ts +++ b/packages/worker/src/mcp/tools/execute.ts @@ -62,10 +62,9 @@ Prefer one \`execute\` when the workflow is clear; split calls when you need new Example: \`async () => { - const page = await codemode.page_to_markdown({ - url: 'https://developers.cloudflare.com/api/resources/accounts/', + return await codemode.kody_official_guide({ + guide: 'integration_bootstrap', }); - return { source: page.source, preview: page.markdown.slice(0, 120) }; }\` To return non-text MCP content blocks (e.g. images), see: https://github.com/kentcdodds/kody/blob/main/docs/use/raw-content-blocks.md diff --git a/packages/worker/src/mcp/tools/search-format.ts b/packages/worker/src/mcp/tools/search-format.ts index 874d470425..1bf89592c6 100644 --- a/packages/worker/src/mcp/tools/search-format.ts +++ b/packages/worker/src/mcp/tools/search-format.ts @@ -318,7 +318,7 @@ export function formatSearchMarkdown(input: { const lines: Array = ['# Search results', ''] if (input.includePreamble ?? true) { lines.push( - 'For full detail on one hit, call `search` with `entity: "{id}:{type}"` (example: `page_to_markdown:capability`).', + 'For full detail on one hit, call `search` with `entity: "{id}:{type}"` (example: `kody_official_guide:capability`).', '', '**How to run matches:**', '', diff --git a/packages/worker/src/mcp/tools/search.ts b/packages/worker/src/mcp/tools/search.ts index 5df849af69..b2a425e276 100644 --- a/packages/worker/src/mcp/tools/search.ts +++ b/packages/worker/src/mcp/tools/search.ts @@ -159,7 +159,7 @@ Example arguments: - \`{ "query": "saved interactive dashboard app", "limit": 10 }\` - \`{ "query": "preferred org value or saved connector", "limit": 10 }\` - \`{ "query": "github automation", "skill_collection": "release-engineering" }\` -- \`{ "entity": "page_to_markdown:capability" }\` +- \`{ "entity": "kody_official_guide:capability" }\` - \`{ "entity": "user:preferred_org:value" }\` - \`{ "entity": "github:connector" }\` - To run a skill: \`meta_run_skill({ "name": "github-pr-summary", "params": { "owner": "kentcdodds" } })\`