Add skills for FTR & Scout testing

.agent/skills/ftr-testing/SKILL.md

@@ -0,0 +1,75 @@
+---
+name: FTR Testing
+description: Use when creating, updating, debugging, or reviewing Kibana Functional Test Runner (FTR) tests, including test structure, services/page objects, loadTestFile patterns, tags, and how to run FTR locally.
+---
+
+# FTR Testing
+
+## Overview
+FTR (FunctionalTestRunner) runs Kibana UI functional tests written in mocha with `@kbn/expect`. Core principle: use FTR services/page objects for interactions, keep tests organized by config, and understand loadTestFile-driven suites.
+
+## Core workflow
+
+1. Identify the FTR config and test file location.
+   - FTR suites live under `test/**` or `x-pack/**/test/**` with config files.
+2. Understand the test structure.
+   - Tests export a provider function that defines a mocha suite.
+   - Use `describe/it/before/beforeEach/after/afterEach`.
+3. Use services and page objects.
+   - Services provide shared capabilities (browser, testSubjects, retry, esArchiver).
+   - Services are named singletons created from `FtrService` subclasses.
+   - Page objects wrap UI interactions.
+4. Watch for `loadTestFile` usage.
+   - Index files can load multiple suites with shared setup.
+5. Use tags in `describe()` to control CI grouping and skips.
+6. If unfamiliar with a page, run the existing FTR tests to learn the flow before migrating.
+
+## Quick reference
+
+- Run all-in-one: `node scripts/functional_tests`
+- Run server + tests:
+  - `node scripts/functional_tests_server`
+  - `node scripts/functional_test_runner --config <path>`
+- Common services: `browser`, `testSubjects`, `retry`, `esArchiver`, `kibanaServer`.
+- Page objects and services are fetched via `getPageObjects()` / `getService()`.
+
+## Common patterns
+
+### loadTestFile
+
+```ts
+export default ({ loadTestFile }: FtrProviderContext) => {
+  describe('suite', () => {
+    loadTestFile(require.resolve('./pages/rules_page'));
+  });
+};
+```
+
+Notes:
+- Index files often include shared setup/teardown; it applies to every loaded suite.
+- When migrating, each `loadTestFile` target becomes its own Scout spec and shared setup
+  must be duplicated or refactored into fixtures/helpers.
+
+### Services and page objects
+
+```ts
+export default ({ getService, getPageObjects }: FtrProviderContext) => {
+  const testSubjects = getService('testSubjects');
+  const browser = getService('browser');
+  const pageObjects = getPageObjects(['header', 'common']);
+
+  describe('My suite', () => {
+    it('does something', async () => {
+      await pageObjects.common.navigateToApp('home');
+      await testSubjects.existOrFail('homeApp');
+      expect(await browser.getCurrentUrl()).toContain('home');
+    });
+  });
+};
+```
+
+## Common mistakes
+
+- Adding UI logic directly in tests instead of using services/page objects.
+- Ignoring `loadTestFile` shared setup in index files.
+- Running with the wrong config file (stateful vs serverless).

.agent/skills/scout-api-testing/SKILL.md

@@ -0,0 +1,107 @@
+---
+name: Scout API Testing
+description: Use when creating, updating, debugging, or reviewing Scout API tests in Kibana (apiTest/apiClient/requestAuth/samlAuth/apiServices), including auth choices, response assertions, and API service patterns.
+---
+
+# Scout API Testing
+
+## Core rules (API)
+
+- API specs live in `<module-root>/test/scout*/api/{tests,parallel_tests}/**/*.spec.ts` (examples: `test/scout/api/...`, `test/scout_uiam_local/api/...`).
+- Use the Scout package that matches the module root:
+- `src/platform/**` or `x-pack/platform/**` -> `@kbn/scout`
+- `x-pack/solutions/observability/**` -> `@kbn/scout-oblt`
+- `x-pack/solutions/search/**` -> `@kbn/scout-search`
+- `x-pack/solutions/security/**` -> `@kbn/scout-security`
+- Prefer a single top-level `apiTest.describe(...)` per file and avoid nested `describe` blocks; multiple top-level `describe`s are supported, but files get hard to read quickly.
+- Tags: add `{ tag: ... }` on the suite (or individual tests) so CI/discovery can select the right deployment target (for example `tags.DEPLOYMENT_AGNOSTIC` or `['@ess']`). Unlike UI tests, API tests don’t currently validate tags at runtime.
+- If the module provides Scout fixtures, import `apiTest` from `<module-root>/test/scout*/api/fixtures` to get module-specific extensions. Importing directly from the module’s Scout package is also fine when you don’t need extensions.
+- Browser fixtures are disabled for `apiTest` (no `page`, `browserAuth`, `pageObjects`).
+
+## Imports
+
+- Test framework + tags: `import { apiTest, tags } from '@kbn/scout';` (or the module's Scout package, e.g. `@kbn/scout-oblt`)
+- Assertions: `import { expect } from '@kbn/scout/api';` (or `@kbn/scout-oblt/api`, etc.) — **not** from the main entry
+- Types: `import type { RoleApiCredentials } from '@kbn/scout';`
+- `expect` is **not** exported from the main `@kbn/scout` entry. Use the `/api` subpath for API tests.
+
+## Auth: pick based on endpoint
+
+- `api/*` endpoints: use **API keys** via `requestAuth` (`getApiKey`, `getApiKeyForCustomRole`).
+- `internal/*` endpoints: use **cookies** via `samlAuth.asInteractiveUser(...)`.
+
+## Recommended test shape
+
+1. **Prepare** environment (optional): `apiServices`/`kbnClient`/`esArchiver` in `beforeAll`.
+2. **Authenticate** (least privilege): generate credentials in `beforeAll` and reuse.
+3. **Request**: call the endpoint with `apiClient` and the right headers.
+4. **Assert**: verify `statusCode` and response body; verify side effects via `apiServices`/`kbnClient` when needed.
+
+Important: `apiServices`/`kbnClient` run with elevated privileges. Don’t use them to validate the endpoint under test (use `apiClient` + scoped auth).
+
+Header reminders:
+- State-changing requests usually need `kbn-xsrf`.
+- Prefer sending `x-elastic-internal-origin: kibana` for Kibana APIs.
+- Include `elastic-api-version` for versioned public APIs (e.g. `'2023-10-31'`) or internal APIs (e.g. `'1'`).
+
+## Assertions
+
+- `apiClient` methods (`get`, `post`, `put`, `delete`, `patch`, `head`) return `{ statusCode, body, headers }`.
+- Use the custom matchers from `@kbn/scout/api`:
+  - `expect(response).toHaveStatusCode(200)`
+  - `expect(response).toHaveStatusText('OK')`
+  - `expect(response).toHaveHeaders({ 'content-type': 'application/json' })`
+- Standard matchers (`toBe`, `toStrictEqual`, `toMatchObject`, etc.) and asymmetric matchers (`expect.objectContaining(...)`, `expect.any(String)`) are also available.
+
+## API services
+
+- Put reusable server-side helpers behind `apiServices` (no UI interactions). Use it for setup/teardown and verifying side effects, not for RBAC validation.
+- **Module-local service**: create it under `<module-root>/test/scout*/api/services/<service>_api_service.ts` (or similar). Register it by extending the module's `apiServices` fixture in `<module-root>/test/scout*/api/fixtures/index.ts` (prefer `{ scope: 'worker' }` when the helper doesn't need per-test state).
+- **Shared service** (reused across modules): consider contributing it to the Scout packages under `src/platform/packages/shared/kbn-scout/src/playwright/fixtures/scope/worker/apis/`.
+
+## Extending fixtures
+
+When tests need custom auth helpers or API services, extend `apiTest` in the module's `fixtures/index.ts`:
+
+```ts
+import { apiTest as base } from '@kbn/scout'; // or the module's Scout package
+import type { RequestAuthFixture } from '@kbn/scout';
+
+interface MyApiFixtures {
+  requestAuth: RequestAuthFixture & { getMyPluginApiKey: () => Promise<RoleApiCredentials> };
+}
+
+export const apiTest = base.extend<MyApiFixtures>({
+  requestAuth: async ({ requestAuth }, use) => {
+    const getMyPluginApiKey = async () =>
+      requestAuth.getApiKeyForCustomRole({
+        kibana: [{ base: [], feature: { myPlugin: ['all'] }, spaces: ['*'] }],
+      });
+    await use({ ...requestAuth, getMyPluginApiKey });
+  },
+});
+```
+
+Tests then import `apiTest` from the local fixtures: `import { apiTest } from '../fixtures';`
+
+## Parallelism
+
+- Treat Scout API tests as sequential by default. Parallel API runs require manual isolation (spaces, indices, saved objects) and are uncommon.
+
+## Run / debug quickly
+
+- Use either `--config` or `--testFiles` (they are mutually exclusive).
+- Run by config: `node scripts/scout.js run-tests --stateful --config <module-root>/test/scout*/api/playwright.config.ts` (or `.../api/parallel.playwright.config.ts` for parallel API runs)
+- Run by file/dir (Scout derives the right `playwright.config.ts` vs `parallel.playwright.config.ts`): `node scripts/scout.js run-tests --stateful --testFiles <module-root>/test/scout*/api/tests/my.spec.ts`
+- For faster iteration, start servers once in another terminal: `node scripts/scout.js start-server --stateful [--config-dir <configSet>]`, then run Playwright directly: `npx playwright test --config <...> --project local --grep <tag>`.
+- `--config-dir` notes:
+- `run-tests` auto-detects the custom config dir from `.../test/scout_<name>/...` paths (override with `--config-dir <name>` if needed).
+- `start-server` has no Playwright config to inspect, so pass `--config-dir <name>` when your tests require a custom server config.
+- Debug: `SCOUT_LOG_LEVEL=debug`
+
+## References
+
+Open only what you need:
+
+- requestAuth vs samlAuth, headers, and least-privilege auth tips: `references/scout-api-auth.md`
+- Creating and registering `apiServices` helpers (kbnClient + retries + logging): `references/scout-api-services.md`

.agent/skills/scout-api-testing/references/scout-api-auth.md

@@ -0,0 +1,87 @@
+# Scout API Test Authentication (requestAuth vs samlAuth)
+
+Use this when writing API tests with `apiTest`/`apiClient`, especially when validating RBAC.
+
+## Pick an auth method
+
+- `api/*` endpoints: use API keys via `requestAuth`.
+- `internal/*` endpoints: use cookies via `samlAuth.asInteractiveUser(...)`.
+
+Both methods return headers you spread into `apiClient` requests.
+
+## API key auth (requestAuth)
+
+- `requestAuth.getApiKey(roleName)`
+- `requestAuth.getApiKeyForCustomRole(roleDescriptor)`
+
+```ts
+import type { RoleApiCredentials } from '@kbn/scout'; // or the module's Scout package (e.g. @kbn/scout-oblt)
+import { apiTest, tags } from '@kbn/scout'; // or the module's Scout package
+import { expect } from '@kbn/scout/api'; // or '@kbn/scout-oblt/api', etc.
+
+const COMMON_HEADERS = {
+  'kbn-xsrf': 'scout',
+  'x-elastic-internal-origin': 'kibana',
+  'elastic-api-version': '2023-10-31', // include for versioned public APIs
+};
+
+apiTest.describe('GET /api/my_plugin/foo', { tag: tags.DEPLOYMENT_AGNOSTIC }, () => {
+  let viewer: RoleApiCredentials;
+
+  apiTest.beforeAll(async ({ requestAuth }) => {
+    viewer = await requestAuth.getApiKey('viewer');
+  });
+
+  apiTest('works', async ({ apiClient }) => {
+    const response = await apiClient.get('api/my_plugin/foo', {
+      headers: { ...COMMON_HEADERS, ...viewer.apiKeyHeader },
+      responseType: 'json',
+    });
+
+    expect(response).toHaveStatusCode(200);
+    expect(response.body).toStrictEqual(expect.objectContaining({ id: expect.any(String) }));
+  });
+});
+```
+
+## Cookie auth (samlAuth) for internal endpoints
+
+```ts
+import { apiTest, tags } from '@kbn/scout'; // or the module's Scout package
+import { expect } from '@kbn/scout/api'; // or '@kbn/scout-oblt/api', etc.
+
+const INTERNAL_HEADERS = {
+  'kbn-xsrf': 'scout',
+  'x-elastic-internal-origin': 'kibana',
+};
+
+apiTest.describe('GET /internal/my_plugin/foo', { tag: tags.DEPLOYMENT_AGNOSTIC }, () => {
+  apiTest('calls internal endpoint', async ({ apiClient, samlAuth }) => {
+    const { cookieHeader } = await samlAuth.asInteractiveUser('viewer');
+
+    const response = await apiClient.get('internal/my_plugin/foo', {
+      headers: { ...INTERNAL_HEADERS, ...cookieHeader },
+      responseType: 'json',
+    });
+
+    expect(response).toHaveStatusCode(200);
+  });
+});
+```
+
+## API assertions (`@kbn/scout/api`)
+
+Import `expect` from `@kbn/scout/api` (or `@kbn/scout-<solution>/api`). It provides custom matchers on top of standard ones:
+
+- `expect(response).toHaveStatusCode(200)` — assert HTTP status code.
+- `expect(response).toHaveStatusText('OK')` — assert HTTP status text.
+- `expect(response).toHaveHeaders({ 'content-type': 'application/json' })` — assert response headers.
+- Standard matchers like `toBe`, `toStrictEqual`, `toBeDefined`, `toMatchObject` are also available.
+- Asymmetric matchers: `expect.objectContaining(...)`, `expect.any(String)`, `expect.toBeGreaterThan(0)`, etc.
+
+`apiClient` methods (`get`, `post`, `put`, `delete`, `patch`, `head`) return `{ statusCode, body, headers }`.
+
+## Tips
+
+- Generate credentials in `beforeAll` if reused across tests.
+- Prefer custom roles for permission-boundary tests instead of `admin`.

.agent/skills/scout-api-testing/references/scout-api-services.md

@@ -0,0 +1,66 @@
+# Scout API Services
+
+API services provide server-side helpers through the `apiServices` fixture.
+Keep API services strictly server-side (no UI interactions).
+Import helper utilities (like `measurePerformanceAsync`) from the Scout package used by the module (`@kbn/scout` or the relevant solution package).
+
+## Create a new API service (summary)
+
+1. Add a new service file under the API fixtures directory.
+2. Add a `types.ts` file for request/response types when the API is non-trivial.
+3. Export a helper function that accepts `log` and `kbnClient` and uses
+   `kbnClient.request` with retries (and `ignoreErrors` when needed).
+4. Wrap calls with `measurePerformanceAsync` for consistent logging.
+5. Register the service in the API fixtures index so it appears under
+   `apiServices.<name>`.
+
+## Minimal sketch
+
+```ts
+import type { KbnClient, ScoutLogger } from '@kbn/scout'; // or the module's Scout package (e.g. @kbn/scout-security)
+import { measurePerformanceAsync } from '@kbn/scout'; // or the module's Scout package
+
+export interface MyApiService {
+  enable: () => Promise<void>;
+}
+
+export const getMyApiService = ({
+  log,
+  kbnClient,
+}: {
+  log: ScoutLogger;
+  kbnClient: KbnClient;
+}): MyApiService => {
+  return {
+    enable: async () => {
+      await measurePerformanceAsync(log, 'myService.enable', async () => {
+        await kbnClient.request({
+          method: 'POST',
+          path: '/api/my/endpoint',
+          retries: 3,
+        });
+      });
+    },
+  };
+};
+```
+
+Register it in the fixture so tests can call:
+
+```ts
+await apiServices.myService.enable();
+```
+
+When adding a module-local API service, extend the `apiServices` fixture (prefer worker scope) and merge in the new
+service:
+
+```ts
+apiServices: async ({ apiServices, kbnClient, log }, use) => {
+  const extended = {
+    ...apiServices,
+    myService: getMyApiService({ kbnClient, log }),
+  };
+
+  await use(extended);
+}
+```

.agent/skills/scout-best-practices-reviewer/SKILL.md

@@ -0,0 +1,18 @@
+---
+name: Scout Best Practices Reviewer
+description: Use when writing and reviewing Scout UI and API test files.
+---
+
+# Scout Best Practices Reviewer
+
+## Overview
+
+- Identify whether the spec is **UI** (`test`/`spaceTest`) or **API** (`apiTest`) and review against the relevant best practices.
+- Keep feedback concise and actionable: focus on correctness, flake risk, and CI/runtime impact.
+- Report findings ordered by severity and include the matching best-practice heading (from the reference) next to each finding.
+
+## References
+
+Open only what you need:
+
+- Review checklist (tagging, structure, auth, flake control): `references/scout-best-practices.md`