-
Notifications
You must be signed in to change notification settings - Fork 0
⚡ Bolt: [성능 개선] computeTaskMetrics 최적화 #508
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
seonghobae
wants to merge
31
commits into
develop
Choose a base branch
from
bolt-optimize-compute-metrics-500206433809831171
base: develop
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
Show all changes
31 commits
Select commit
Hold shift + click to select a range
a6e271a
perf(computeTaskMetrics): replace map/reduce with int32array/loops
seonghobae 7d07992
test(perf): lock task metric cache semantics
seonghobae e394998
test(perf): run task metric regression in coverage
seonghobae d19d560
test(perf): add exact-head task metric benchmark evidence
seonghobae 3f3b6e6
perf(computeTaskMetrics): replace map/reduce with int32array/loops
seonghobae a118883
test(perf): benchmark metric computation against protected base
seonghobae e1ca5b7
ci(perf): run exact-base metrics benchmark
seonghobae 0094fc0
test(perf): instrument metric benchmark without production globals
seonghobae 765583d
Merge branch 'develop' into bolt-optimize-compute-metrics-50020643380…
opencode-agent[bot] cd52c06
merge(perf): reconcile protected XML import baseline
seonghobae 038b8e3
Merge protected develop into metrics performance branch
seonghobae 514b086
test(perf): require immutable benchmark base on push
seonghobae 5a17020
fix(perf): resolve benchmark base for protected pushes
seonghobae 6bb681d
fix(perf): fail closed on missing benchmark baselines
seonghobae e27f57d
test(perf): run benchmark base contract in unit CI
seonghobae 24a4e5c
test(perf): always close benchmark browser contexts
seonghobae 3a48fcf
merge(develop): reconcile metrics performance with OpenCode config
seonghobae 1699198
merge(develop): reconcile metrics performance with current protected …
seonghobae 0ae5a54
fix(stack): preserve protected orchestrator while reconciling metrics…
seonghobae 77daa1d
fix(stack): inherit protected Hono runtime baseline
seonghobae e77846d
fix(stack): inherit protected Playwright 1.62.1 in metrics benchmark
seonghobae da9b54e
test(perf): expose metrics benchmark order bias
seonghobae 9b6d98f
fix(perf): counterbalance metrics benchmark timings
seonghobae f119ac3
fix(perf): counterbalance metrics browser benchmark
seonghobae b36361e
test(perf): require immutable benchmark candidate
seonghobae 32bc31b
fix(perf): bind metrics benchmark candidate identity
seonghobae 59a8f03
fix(perf): benchmark exact contributor revision
seonghobae 1cc8b59
test(perf): expose stale protected-base benchmark
seonghobae f6fafe1
fix(perf): verify live protected benchmark base
seonghobae 388406f
fix(perf): reject stale protected-base benchmark
seonghobae 642034f
test(perf): isolate benchmark from cloud e2e
seonghobae File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,281 @@ | ||
| import { execFileSync } from 'node:child_process'; | ||
| import { readFileSync } from 'node:fs'; | ||
|
|
||
| import { test, expect } from '@playwright/test'; | ||
|
|
||
| import { | ||
| counterbalancedBenchmarkRounds, | ||
| resolveBenchmarkCandidateSha, | ||
| resolveVerifiedBenchmarkBaseSha, | ||
| summarizeCounterbalancedSamples, | ||
| } from '../helpers/benchmark-base.mjs'; | ||
|
|
||
| test.describe.configure({ retries: process.env.CI ? 2 : 0 }); | ||
|
|
||
| const TASK_COUNT = 10_000; | ||
| const SAMPLE_COUNT = 7; | ||
| const WARMUP_COUNT = 3; | ||
| const TARGET_IMPROVEMENT_PERCENT = 15; | ||
| const BASE_DATE = '2026-02-15'; | ||
|
|
||
| const DATE_WINDOWS = Object.freeze([ | ||
| ['2026-01-01', '2026-01-02'], | ||
| ['2026-01-02', '2026-01-12'], | ||
| ['2026-02-01', '2026-03-01'], | ||
| ['2026-02-15', '2026-02-15'], | ||
| ]); | ||
|
|
||
| function githubEvent() { | ||
| const eventPath = process.env.GITHUB_EVENT_PATH; | ||
| return eventPath ? JSON.parse(readFileSync(eventPath, 'utf8')) : {}; | ||
| } | ||
|
|
||
| function readOriginBranchTip(baseRef) { | ||
| const branch = String(baseRef || '').trim(); | ||
| if (!branch || branch.length > 255 || /[\u0000-\u001f\u007f]/u.test(branch)) { | ||
| throw new Error(`Invalid benchmark base ref: ${branch || '<missing>'}`); | ||
| } | ||
| const fullRef = `refs/heads/${branch}`; | ||
| let output; | ||
| try { | ||
| output = execFileSync('git', ['ls-remote', '--heads', 'origin', fullRef], { | ||
| encoding: 'utf8', | ||
| stdio: ['ignore', 'pipe', 'pipe'], | ||
| }); | ||
| } catch { | ||
| throw new Error(`Unable to resolve live benchmark base ${fullRef}`); | ||
| } | ||
|
|
||
| const matches = output | ||
| .split(/\r?\n/u) | ||
| .filter(Boolean) | ||
| .map((line) => line.split(/\s+/u)) | ||
| .filter(([, remoteRef]) => remoteRef === fullRef); | ||
| if (matches.length !== 1) { | ||
| throw new Error(`Expected exactly one live benchmark base for ${fullRef}, found ${matches.length}`); | ||
| } | ||
| return matches[0][0]; | ||
| } | ||
|
|
||
| function readGitFile(commitSha, path) { | ||
| const normalizedCommitSha = String(commitSha || ''); | ||
| if (!/^[a-f0-9]{40}$/.test(normalizedCommitSha)) { | ||
| throw new Error(`Invalid benchmark commit SHA: ${normalizedCommitSha || '<missing>'}`); | ||
| } | ||
|
|
||
| const spec = `${normalizedCommitSha}:${path}`; | ||
| try { | ||
| return execFileSync('git', ['show', spec], { encoding: 'utf8' }); | ||
| } catch { | ||
| execFileSync('git', ['fetch', '--depth=1', 'origin', normalizedCommitSha], { | ||
| encoding: 'utf8', | ||
| stdio: ['ignore', 'pipe', 'pipe'], | ||
| }); | ||
| return execFileSync('git', ['show', spec], { encoding: 'utf8' }); | ||
| } | ||
| } | ||
|
coderabbitai[bot] marked this conversation as resolved.
|
||
|
|
||
| function instrumentMetricsSource(source) { | ||
| const bootstrapCall = '\nbootstrap();'; | ||
| const bootstrapIndex = source.lastIndexOf(bootstrapCall); | ||
| if (bootstrapIndex === -1) { | ||
| throw new Error('Benchmark app source is missing the expected bootstrap call'); | ||
| } | ||
|
|
||
| const withoutBootstrap = `${source.slice(0, bootstrapIndex)}${source.slice(bootstrapIndex + bootstrapCall.length)}`; | ||
| return `${withoutBootstrap}\n\nwindow.__scopeweaveMetricsBenchmark = Object.freeze({\n seed(tasks, baseDate) {\n state.tasks = tasks;\n state.baseDate = baseDate;\n },\n compute() {\n return computeTaskMetrics();\n },\n});\n`; | ||
| } | ||
|
|
||
| function createTask(index) { | ||
| const [plannedStartDate, plannedEndDate] = DATE_WINDOWS[index % DATE_WINDOWS.length]; | ||
| return { | ||
| id: `metrics-performance-${index}`, | ||
| parentId: null, | ||
| depth: 1, | ||
| expanded: true, | ||
| pendingDelete: false, | ||
| isSynthetic: false, | ||
| phase: `Phase ${index}`, | ||
| activity: '', | ||
| task: '', | ||
| categoryLarge: '', | ||
| categoryMedium: '', | ||
| documentName: '', | ||
| owner: `owner-${index % 17}`, | ||
| supportTeam: '', | ||
| plannedStartDate, | ||
| plannedEndDate, | ||
| actualProgressStatus: '미착수(0%)', | ||
| actualStartDate: '', | ||
| actualEndDate: '', | ||
| predecessors: '', | ||
| budget: '', | ||
| actualCost: '', | ||
| sprint: '', | ||
| storyPoints: '', | ||
| }; | ||
| } | ||
|
|
||
| async function measureMetrics(browser, { appSource, label }) { | ||
| const context = await browser.newContext(); | ||
| try { | ||
| const page = await context.newPage(); | ||
| const instrumentedSource = instrumentMetricsSource(appSource); | ||
|
|
||
| await page.route('**/app.js', async (route) => { | ||
| await route.fulfill({ | ||
| status: 200, | ||
| contentType: 'application/javascript; charset=utf-8', | ||
| body: instrumentedSource, | ||
| }); | ||
| }); | ||
|
|
||
| await page.goto('/'); | ||
| const tasks = Array.from({ length: TASK_COUNT }, (_, index) => createTask(index)); | ||
|
|
||
| const result = await page.evaluate(async ({ seededTasks, baseDate, sampleCount, warmupCount }) => { | ||
| const benchmark = window.__scopeweaveMetricsBenchmark; | ||
| if (!benchmark) throw new Error('metrics benchmark bridge did not initialize'); | ||
| benchmark.seed(seededTasks, baseDate); | ||
|
|
||
| for (let warmup = 0; warmup < warmupCount; warmup += 1) { | ||
| benchmark.compute(); | ||
| } | ||
|
|
||
| const samples = []; | ||
| for (let sample = 0; sample < sampleCount; sample += 1) { | ||
| const startedAt = performance.now(); | ||
| benchmark.compute(); | ||
| samples.push(performance.now() - startedAt); | ||
| } | ||
|
seonghobae marked this conversation as resolved.
|
||
|
|
||
| const metrics = benchmark.compute(); | ||
| const entries = Array.from(metrics.byTask, ([taskId, taskMetrics]) => [ | ||
| taskId, | ||
| taskMetrics.durationDays, | ||
| taskMetrics.weightRatio, | ||
| taskMetrics.plannedProgressRatio, | ||
| taskMetrics.actualProgressRatio, | ||
| taskMetrics.weightedPlannedRatio, | ||
| taskMetrics.weightedActualRatio, | ||
| taskMetrics.progressState.label, | ||
| taskMetrics.progressState.className, | ||
| taskMetrics.plannedDateWarning, | ||
| taskMetrics.actualDateWarning, | ||
| ]); | ||
| const snapshot = JSON.stringify({ | ||
| totalDays: metrics.totalDays, | ||
| totalWeightedPlannedRatio: metrics.totalWeightedPlannedRatio, | ||
| totalWeightedActualRatio: metrics.totalWeightedActualRatio, | ||
| entries, | ||
| }); | ||
| const digestBytes = new Uint8Array(await crypto.subtle.digest( | ||
| 'SHA-256', | ||
| new TextEncoder().encode(snapshot), | ||
| )); | ||
| const digest = Array.from(digestBytes, (byte) => byte.toString(16).padStart(2, '0')).join(''); | ||
|
|
||
| return { | ||
| samples, | ||
| digest, | ||
| totalDays: metrics.totalDays, | ||
| byTaskSize: metrics.byTask.size, | ||
| totalWeightedPlannedRatio: metrics.totalWeightedPlannedRatio, | ||
| totalWeightedActualRatio: metrics.totalWeightedActualRatio, | ||
| }; | ||
| }, { | ||
| seededTasks: tasks, | ||
| baseDate: BASE_DATE, | ||
| sampleCount: SAMPLE_COUNT, | ||
| warmupCount: WARMUP_COUNT, | ||
| }); | ||
|
|
||
| return { label, ...result }; | ||
| } finally { | ||
| await context.close(); | ||
| } | ||
| } | ||
|
|
||
| test('10,000-task metric computation preserves exact semantics without median regression', async ({ browser }) => { | ||
| test.setTimeout(240_000); | ||
|
|
||
| const event = githubEvent(); | ||
| const resolveCurrentBaseSha = () => resolveVerifiedBenchmarkBaseSha({ | ||
| override: process.env.SCOPEWEAVE_BENCHMARK_BASE_SHA, | ||
| event, | ||
| readLiveBaseSha: readOriginBranchTip, | ||
| }); | ||
| const baseSha = resolveCurrentBaseSha(); | ||
| const candidateSha = resolveBenchmarkCandidateSha({ | ||
| override: process.env.SCOPEWEAVE_BENCHMARK_HEAD_SHA, | ||
| event, | ||
| }); | ||
| const sourceByLabel = new Map([ | ||
| ['protected-base', readGitFile(baseSha, 'app.js')], | ||
| ['candidate', readGitFile(candidateSha, 'app.js')], | ||
| ]); | ||
| const measurementOrder = counterbalancedBenchmarkRounds(); | ||
| const measurements = []; | ||
|
|
||
| for (const round of measurementOrder) { | ||
| for (const label of round) { | ||
| measurements.push(await measureMetrics(browser, { | ||
| appSource: sourceByLabel.get(label), | ||
| label, | ||
| })); | ||
| } | ||
| } | ||
|
|
||
| const semanticReference = measurements[0]; | ||
| for (const measurement of measurements) { | ||
| expect(measurement.byTaskSize).toBe(TASK_COUNT); | ||
| expect(measurement.samples).toHaveLength(SAMPLE_COUNT); | ||
| expect(measurement.samples.every((duration) => duration > 0)).toBe(true); | ||
| expect(measurement.digest).toBe(semanticReference.digest); | ||
| expect(measurement.totalDays).toBe(semanticReference.totalDays); | ||
| expect(measurement.totalWeightedPlannedRatio).toBe(semanticReference.totalWeightedPlannedRatio); | ||
| expect(measurement.totalWeightedActualRatio).toBe(semanticReference.totalWeightedActualRatio); | ||
| } | ||
|
|
||
| const summary = summarizeCounterbalancedSamples(measurements); | ||
| expect(summary.baselineMedianDurationMs).toBeGreaterThan(0); | ||
| expect(summary.candidateMedianDurationMs).toBeGreaterThan(0); | ||
| // Wall-clock performance is runner-dependent; keep the 15% target as reported | ||
| // evidence while failing only on a measured median regression. | ||
| expect( | ||
| summary.improvementPercent, | ||
| `expected no counterbalanced median computeTaskMetrics regression for exact head ${candidateSha} over ${baseSha}, got ${summary.improvementPercent.toFixed(2)}%`, | ||
| ).toBeGreaterThanOrEqual(0); | ||
|
|
||
| const completionBaseSha = resolveCurrentBaseSha(); | ||
| expect(completionBaseSha).toBe(baseSha); | ||
|
|
||
| const sharedSemanticEvidence = { | ||
| digest: semanticReference.digest, | ||
| totalDays: semanticReference.totalDays, | ||
| byTaskSize: semanticReference.byTaskSize, | ||
| totalWeightedPlannedRatio: semanticReference.totalWeightedPlannedRatio, | ||
| totalWeightedActualRatio: semanticReference.totalWeightedActualRatio, | ||
| }; | ||
| console.log(`SCOPEWEAVE_METRICS_BENCHMARK ${JSON.stringify({ | ||
| taskCount: TASK_COUNT, | ||
| sampleCountPerMeasurement: SAMPLE_COUNT, | ||
| warmupCountPerMeasurement: WARMUP_COUNT, | ||
| measurementOrder, | ||
| protectedBaseSha: baseSha, | ||
| exactContributorHeadSha: candidateSha, | ||
| protectedBaselineAvailable: true, | ||
| targetImprovementPercent: TARGET_IMPROVEMENT_PERCENT, | ||
| optimizationDeltaPercent: summary.improvementPercent, | ||
| baseline: { | ||
| samples: summary.baselineSamples, | ||
| medianDurationMs: summary.baselineMedianDurationMs, | ||
| ...sharedSemanticEvidence, | ||
| }, | ||
| optimized: { | ||
| samples: summary.candidateSamples, | ||
| medianDurationMs: summary.candidateMedianDurationMs, | ||
| ...sharedSemanticEvidence, | ||
| }, | ||
| })}`); | ||
| }); | ||
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.