Repository navigation
test(integration): measure authenticated delivery status p95 #260
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Draft
seonghobae
wants to merge
58
commits into
feat/plugin-delivery-attempt-status-http-v1
Choose a base branch
from
perf/plugin-delivery-status-k6-v1
base: feat/plugin-delivery-attempt-status-http-v1
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Draft
Changes from 23 commits
Commits
Show all changes
58 commits
Select commit
Hold shift + click to select a range
710f54d
test(integration): add status load authority generator
seonghobae ab1c5ba
test(integration): add real status k6 contract
seonghobae ff2c56d
test(integration): seed durable status performance fixture
seonghobae e252d94
test(integration): verify delivery status p95 on PostgreSQL
seonghobae f7802a8
test(integration): verify status with current k6 stable
seonghobae 0f6562b
fix(integration): honor verified TLS in status benchmark
seonghobae 388d78b
perf(integration): make replay consume one round trip
seonghobae 295332d
perf(integration): collapse replay consume round trip
seonghobae e05637a
test(integration): cover single-call replay consume
seonghobae 19489da
test(integration): gate replay hot-path repair
seonghobae 6adfd90
chore(integration): expose canonical formatter delta
seonghobae fd494cd
style(integration): apply canonical replay formatting
seonghobae 547ea77
style(integration): format status authority generator
seonghobae 850d007
style(integration): format delivery status k6
seonghobae 1053da2
test(integration): prove replay consume concurrency in PostgreSQL
seonghobae a194204
test(integration): verify replay repair and status p95
seonghobae b15b0dc
style(integration): format replay PostgreSQL acceptance
seonghobae 5c0a8e2
test(integration): expose canonical replay acceptance format
seonghobae cfb3f5a
style(integration): apply canonical replay acceptance format
seonghobae 9452931
test(integration): restore final status p95 verifier
seonghobae f4ddeee
test(integration): use dedicated replay acceptance database
seonghobae a24862e
test(integration): separate runtime and harness database authority
seonghobae a81c8f2
test(integration): retire completed status p95 verifier
seonghobae e1e22e3
test(integration): reject mutable replay SQL evidence
seonghobae b9c4594
ci(integration): verify replay SQL evidence snapshot
seonghobae b563f70
fix(integration): snapshot replay SQL evidence
seonghobae 8b19326
test(integration): reject hostile replay SQL access
seonghobae c8282a9
fix(integration): bound hostile replay SQL evidence
seonghobae 8f540c5
test(integration): reject hostile replay request evidence
seonghobae a200c24
fix(integration): bound replay request evidence access
seonghobae 003dfd0
test(integration): reject expired replay evidence
seonghobae 214e273
fix(integration): reject expired replay evidence in database
seonghobae f15c094
fix(integration): exclusively create k6 authority bundle
seonghobae 4fc85bf
test(integration): require exact status evidence keys
seonghobae bab0e71
test(integration): require TLS for signed k6 authority
seonghobae c0eee74
test(integration): restore TLS k6 current-head acceptance
seonghobae 8020d0f
test(ci): avoid unsupported SQL prettier check
seonghobae 3806f4a
fix(ci): focus replay verifier and build sdk dependency
seonghobae 5c6e051
fix(perf): keep scenario controls out of reserved k6 env
seonghobae e3da325
fix(perf): align authority fixture with custom controls
seonghobae ad75adc
fix(perf): remove k6 option env collision
seonghobae bb1c864
test(perf): profile delivery status latency stages
seonghobae 50ed920
test(perf): retain post-failure latency profile
seonghobae dc35367
test(perf): isolate server latency profile
seonghobae 448f35b
test(perf): align direct profile with k6 request timing
seonghobae a149cc8
style(perf): restore canonical profiler formatting
seonghobae 4794e34
style(perf): finish canonical profiler formatting
seonghobae 4fd46c8
style(perf): wrap profiler context secret
seonghobae 26f6cad
ci(perf): expose exact profiler formatting delta
seonghobae 86ace33
style(perf): apply profiler formatting
seonghobae fc44d0d
ci(perf): retire profiler formatter diagnostic
seonghobae cf8361b
ci(perf): retire replay verifier after exact green
seonghobae 840c9c2
ci(perf): retire k6 verifier after exact green
seonghobae a12888b
ci(perf): restore k6 verifier after failed exact head
seonghobae 9ce9f89
chore(perf): retire exact-green k6 verifier
seonghobae a5de82b
ci(perf): require replicated exact-head p95 evidence
seonghobae 24d8079
ci(perf): retain replicated p95 evidence
seonghobae 7967976
ci(perf): retire proven delivery-status verifier
seonghobae File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
47 changes: 47 additions & 0 deletions
47
apps/integration-service/migrations/0012_plugin_operator_context_replay_consume.sql
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,47 @@ | ||
| CREATE OR REPLACE FUNCTION plugin_integration.consume_plugin_operator_context_replay( | ||
| p_evidence_id uuid, | ||
| p_consumed_at timestamptz, | ||
| p_expires_at timestamptz | ||
| ) | ||
| RETURNS boolean | ||
| LANGUAGE plpgsql | ||
| AS $$ | ||
| DECLARE | ||
| consumed_evidence_id uuid; | ||
| BEGIN | ||
| INSERT INTO plugin_integration.plugin_operator_context_replay_record ( | ||
| evidence_id, | ||
| consumed_at, | ||
| expires_at | ||
| ) VALUES ( | ||
| p_evidence_id, | ||
| p_consumed_at, | ||
| p_expires_at | ||
| ) | ||
| ON CONFLICT (evidence_id) DO UPDATE | ||
| SET consumed_at = EXCLUDED.consumed_at, | ||
| expires_at = EXCLUDED.expires_at | ||
| WHERE plugin_integration.plugin_operator_context_replay_record.expires_at < now() | ||
| RETURNING evidence_id INTO consumed_evidence_id; | ||
|
|
||
| IF consumed_evidence_id IS NULL THEN | ||
| RETURN false; | ||
| END IF; | ||
|
|
||
| DELETE FROM plugin_integration.plugin_operator_context_replay_record AS replay_record | ||
| WHERE replay_record.evidence_id IN ( | ||
| SELECT expired_record.evidence_id | ||
| FROM plugin_integration.plugin_operator_context_replay_record AS expired_record | ||
| WHERE expired_record.expires_at < now() | ||
| AND expired_record.evidence_id <> p_evidence_id | ||
| ORDER BY expired_record.expires_at | ||
| FOR UPDATE SKIP LOCKED | ||
| LIMIT 32 | ||
| ); | ||
|
|
||
| RETURN true; | ||
| END; | ||
| $$; | ||
|
|
||
| COMMENT ON FUNCTION plugin_integration.consume_plugin_operator_context_replay(uuid, timestamptz, timestamptz) IS | ||
| 'Atomically consumes one Plugin operator evidence identity and performs bounded post-consume expiry cleanup in the same PostgreSQL round trip.'; | ||
65 changes: 65 additions & 0 deletions
65
apps/integration-service/perf/generate-plugin-delivery-status-authority.mjs
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,65 @@ | ||
| import { createHmac, randomUUID } from 'node:crypto'; | ||
| import { writeFile } from 'node:fs/promises'; | ||
|
|
||
| const DEFAULT_ITERATIONS = 1000; | ||
| const MAX_ITERATIONS = 10000; | ||
| const WORKSPACE_ID = '11111111-1111-4111-8111-111111111111'; | ||
| const USER_ID = '22222222-2222-4222-8222-222222222222'; | ||
| const DELIVERY_ID = '33333333-3333-4333-8333-333333333333'; | ||
| const PATH = `/v1/plugins/delivery-attempts/${DELIVERY_ID}`; | ||
|
|
||
| function fail(message) { | ||
| throw new Error(message); | ||
| } | ||
|
|
||
| function iterations(value) { | ||
| const parsed = value === undefined ? DEFAULT_ITERATIONS : Number(value); | ||
| if (!Number.isSafeInteger(parsed) || parsed < 1 || parsed > MAX_ITERATIONS) { | ||
| return fail('K6_ITERATIONS must be an integer between 1 and 10000'); | ||
| } | ||
| return parsed; | ||
| } | ||
|
|
||
| function secret(value) { | ||
| if (typeof value !== 'string' || Buffer.byteLength(value, 'utf8') < 32) { | ||
| return fail( | ||
| 'INTEGRATION_OPERATOR_CONTEXT_SECRET must contain at least 32 UTF-8 bytes', | ||
| ); | ||
| } | ||
| return value; | ||
| } | ||
|
|
||
| const outputPath = process.argv[2]; | ||
| if (!outputPath) { | ||
| fail('authority output path is required'); | ||
| } | ||
|
|
||
| const count = iterations(process.env.K6_ITERATIONS); | ||
| const contextSecret = secret(process.env.INTEGRATION_OPERATOR_CONTEXT_SECRET); | ||
| const issuedAt = String(Math.floor(Date.now() / 1000)); | ||
| const authorities = Array.from({ length: count }, () => { | ||
| const evidenceId = randomUUID(); | ||
| const signature = createHmac('sha256', contextSecret) | ||
| .update( | ||
| `life-os.integration-operator-context.v1\n${WORKSPACE_ID}\n${USER_ID}\n${evidenceId}\n${issuedAt}\nGET\n${PATH}`, | ||
| 'utf8', | ||
| ) | ||
| .digest('base64url'); | ||
|
|
||
| return Object.freeze({ | ||
| path: PATH, | ||
| deliveryId: DELIVERY_ID, | ||
| headers: Object.freeze({ | ||
| 'x-life-os-workspace-id': WORKSPACE_ID, | ||
| 'x-life-os-user-id': USER_ID, | ||
| 'x-life-os-context-evidence-id': evidenceId, | ||
| 'x-life-os-context-issued-at': issuedAt, | ||
| 'x-life-os-context-signature': signature, | ||
| }), | ||
| }); | ||
| }); | ||
|
|
||
| await writeFile(outputPath, `${JSON.stringify(authorities)}\n`, { | ||
| encoding: 'utf8', | ||
| mode: 0o600, | ||
| }); | ||
|
coderabbitai[bot] marked this conversation as resolved.
Resolved
|
||
83 changes: 83 additions & 0 deletions
83
apps/integration-service/perf/plugin-delivery-attempt-status.k6.js
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,83 @@ | ||
| import http from 'k6/http'; | ||
| import { check } from 'k6'; | ||
| import { SharedArray } from 'k6/data'; | ||
| import exec from 'k6/execution'; | ||
|
|
||
| function boundedInteger(name, fallback, minimum, maximum) { | ||
| const raw = __ENV[name]; | ||
| const value = raw === undefined || raw === '' ? fallback : Number(raw); | ||
| if (!Number.isInteger(value) || value < minimum || value > maximum) { | ||
| throw new Error( | ||
| `${name} must be an integer between ${minimum} and ${maximum}`, | ||
| ); | ||
| } | ||
| return value; | ||
| } | ||
|
|
||
| const iterations = boundedInteger('K6_ITERATIONS', 1000, 1, 10000); | ||
| const vus = boundedInteger('K6_VUS', 10, 1, 100); | ||
| const authorityFile = __ENV.K6_AUTHORITY_FILE; | ||
| const baseUrl = __ENV.K6_BASE_URL; | ||
|
|
||
| if (!authorityFile) { | ||
| throw new Error('K6_AUTHORITY_FILE is required'); | ||
| } | ||
| if (!baseUrl || !/^http:\/\/127\.0\.0\.1:\d+$/u.test(baseUrl)) { | ||
| throw new Error('K6_BASE_URL must be an explicit loopback HTTP endpoint'); | ||
|
coderabbitai[bot] marked this conversation as resolved.
Resolved
|
||
| } | ||
|
|
||
| const authorities = new SharedArray('plugin-delivery-status-authority', () => { | ||
| const parsed = JSON.parse(open(authorityFile)); | ||
| if (!Array.isArray(parsed) || parsed.length !== iterations) { | ||
| throw new Error( | ||
| 'authority bundle must contain exactly K6_ITERATIONS entries', | ||
| ); | ||
| } | ||
| return parsed; | ||
| }); | ||
|
|
||
| export const options = { | ||
| scenarios: { | ||
| plugin_delivery_status: { | ||
| executor: 'shared-iterations', | ||
| vus, | ||
| iterations, | ||
| maxDuration: '45s', | ||
| }, | ||
| }, | ||
| thresholds: { | ||
| 'http_req_duration{endpoint:plugin_delivery_status}': ['p(95)<20'], | ||
| 'http_req_failed{endpoint:plugin_delivery_status}': ['rate==0'], | ||
| checks: ['rate==1'], | ||
| }, | ||
| summaryTrendStats: ['avg', 'med', 'p(90)', 'p(95)', 'p(99)', 'max'], | ||
| }; | ||
|
coderabbitai[bot] marked this conversation as resolved.
Resolved
|
||
|
|
||
| export default function () { | ||
| const authority = authorities[exec.scenario.iterationInTest]; | ||
| if (!authority) { | ||
| throw new Error('missing unique authority for iteration'); | ||
| } | ||
|
|
||
| const response = http.get(`${baseUrl}${authority.path}`, { | ||
| headers: authority.headers, | ||
| redirects: 0, | ||
| timeout: '2s', | ||
| tags: { endpoint: 'plugin_delivery_status' }, | ||
| }); | ||
|
|
||
| check(response, { | ||
| 'status is 200': (result) => result.status === 200, | ||
| 'status evidence matches delivery': (result) => | ||
| result.status === 200 && | ||
| result.json('deliveryId') === authority.deliveryId, | ||
| 'status evidence remains credential-free': (result) => | ||
| result.status === 200 && | ||
| !Object.prototype.hasOwnProperty.call(result.json(), 'claimToken') && | ||
| !Object.prototype.hasOwnProperty.call( | ||
| result.json(), | ||
| 'claimTokenDigest', | ||
| ) && | ||
| !Object.prototype.hasOwnProperty.call(result.json(), 'credential'), | ||
|
seonghobae marked this conversation as resolved.
Resolved
|
||
| }); | ||
| } | ||
81 changes: 81 additions & 0 deletions
81
apps/integration-service/perf/seed-plugin-delivery-attempt-status.sql
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,81 @@ | ||
| BEGIN; | ||
|
|
||
| INSERT INTO plugin_integration.plugin_installation_record ( | ||
| installation_id, | ||
| workspace_id, | ||
| installed_by_user_id, | ||
| plugin_id, | ||
| plugin_contract_version, | ||
| manifest_sha256, | ||
| granted_capabilities, | ||
| installation_status, | ||
| installed_at, | ||
| revoked_at | ||
| ) VALUES ( | ||
| '77777777-7777-4777-8777-777777777777', | ||
| '11111111-1111-4111-8111-111111111111', | ||
| '22222222-2222-4222-8222-222222222222', | ||
| 'life-os.performance.status', | ||
| 'life-os.plugin.v1', | ||
| repeat('0', 64), | ||
| ARRAY[]::text[], | ||
| 'active', | ||
| clock_timestamp() - INTERVAL '10 minutes', | ||
| NULL | ||
| ); | ||
|
|
||
| INSERT INTO plugin_integration.plugin_delivery_origin_grant_record ( | ||
| authority_version, | ||
| grant_id, | ||
| installation_id, | ||
| workspace_id, | ||
| granted_by_user_id, | ||
| origin_uri, | ||
| grant_status, | ||
| granted_at, | ||
| revoked_at | ||
| ) VALUES ( | ||
| 'life-os.plugin-delivery-origin.v1', | ||
| '66666666-6666-4666-8666-666666666666', | ||
| '77777777-7777-4777-8777-777777777777', | ||
| '11111111-1111-4111-8111-111111111111', | ||
| '22222222-2222-4222-8222-222222222222', | ||
| 'https://performance.example.test', | ||
| 'active', | ||
| clock_timestamp() - INTERVAL '9 minutes', | ||
| NULL | ||
| ); | ||
|
|
||
| INSERT INTO plugin_integration.plugin_delivery_attempt_record ( | ||
| authority_version, | ||
| delivery_id, | ||
| grant_id, | ||
| installation_id, | ||
| workspace_id, | ||
| requested_by_user_id, | ||
| delivery_status, | ||
| attempt_count, | ||
| max_attempts, | ||
| requested_at, | ||
| updated_at, | ||
| next_attempt_at, | ||
| terminal_at, | ||
| last_outcome_code | ||
| ) VALUES ( | ||
| 'life-os.plugin-delivery-attempt.v1', | ||
| '33333333-3333-4333-8333-333333333333', | ||
| '66666666-6666-4666-8666-666666666666', | ||
| '77777777-7777-4777-8777-777777777777', | ||
| '11111111-1111-4111-8111-111111111111', | ||
| '22222222-2222-4222-8222-222222222222', | ||
| 'pending', | ||
| 0, | ||
| 3, | ||
| clock_timestamp() - INTERVAL '1 minute', | ||
| clock_timestamp() - INTERVAL '1 minute', | ||
| clock_timestamp() + INTERVAL '5 minutes', | ||
| NULL, | ||
| NULL | ||
| ); | ||
|
|
||
| COMMIT; |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.