Skip to content

Consolidate benchmark config into a single generated registry #33

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
ford442 merged 1 commit into from
Aug 12, 2026
Merged

Consolidate benchmark config into a single generated registry #33

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 17 additions & 0 deletions .github/workflows/config-check.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
name: Benchmark Config Drift Check

on:
push:
branches: [main]
pull_request:
branches: [main]

jobs:
config-check:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4

@coderabbitai coderabbitai Bot Aug 12, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔒 Security & Privacy | 🟡 Minor | ⚡ Quick win

🧩 Analysis chain

🌐 Web query:

For actions/checkout@v4, does persist-credentials default to true, and does setting persist-credentials: false prevent the GitHub token from being stored in local Git configuration?

💡 Result:

For actions/checkout@v4, the persist-credentials input defaults to true [1][2][3]. Setting persist-credentials to false does prevent the GitHub token (or SSH key) from being configured in the local Git configuration [1][4][5]. When persist-credentials is set to true (the default), the action automatically configures the local Git settings with the provided token, which allows subsequent steps in the job to run authenticated Git commands [1][4]. Setting this to false opts out of this behavior, ensuring the token is not stored in the local Git configuration [1][5].

Citations:

🏁 Script executed:

#!/bin/bash
set -eu

printf '%s\n' '--- workflow ---'
cat -n .github/workflows/config-check.yml

printf '%s\n' '--- package scripts ---'
node - <<'JS'
const fs = require('fs');
const pkg = JSON.parse(fs.readFileSync('package.json', 'utf8'));
console.log(JSON.stringify({ scripts: pkg.scripts }, null, 2));
JS

printf '%s\n' '--- Git-related references in the config-check path ---'
rg -n -i 'git|github|token|credential|checkout|config:check' package.json .github/workflows . --glob '!node_modules' --glob '!dist' --glob '!build' | head -200

Repository: ford442/benching_machine

Length of output: 50381

🏁 Script executed:

#!/bin/bash
set -eu

printf '%s\n' '--- workflow ---'
cat -n .github/workflows/config-check.yml

printf '%s\n' '--- config:check script ---'
node - <<'JS'
const fs = require('fs');
const pkg = JSON.parse(fs.readFileSync('package.json', 'utf8'));
console.log(pkg.scripts && pkg.scripts['config:check']);
JS

printf '%s\n' '--- files referenced by config:check ---'
node - <<'JS'
const fs = require('fs');
const pkg = JSON.parse(fs.readFileSync('package.json', 'utf8'));
const script = pkg.scripts && pkg.scripts['config:check'];
console.log(script || '(missing)');
JS
rg -n --glob 'package.json' --glob '.github/workflows/config-check.yml' --glob '*.js' --glob '*.mjs' --glob '*.cjs' --glob '*.ts' --glob '*.yml' --glob '*.yaml' \
  'config:check|git (clone|fetch|pull|push)|actions/checkout|GITHUB_TOKEN|github\.token|persist-credentials' .

Repository: ford442/benching_machine

Length of output: 870

🏁 Script executed:

#!/bin/bash
set -eu

printf '%s\n' '--- generator outline and size ---'
wc -l scripts/generate-benchmark-config.js
ast-grep outline scripts/generate-benchmark-config.js

printf '%s\n' '--- Git, token, and credential references in generator ---'
rg -n -i 'git|github|token|credential|exec|spawn|https?://|fetch\(' scripts/generate-benchmark-config.js || true

printf '%s\n' '--- generator entry points and file operations ---'
rg -n 'process\.argv|readFile|writeFile|existsSync|readdir|path\.|require\(' scripts/generate-benchmark-config.js

Repository: ford442/benching_machine

Length of output: 1811

Disable persisted Git credentials.

Set persist-credentials: false because this job does not require Git authentication. This prevents actions/checkout@v4 from storing the GITHUB_TOKEN in local Git configuration.

Proposed fix 
-      - uses: actions/checkout@v4
+      - uses: actions/checkout@v4
+        with:
+          persist-credentials: false
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
- uses: actions/checkout@v4
- uses: actions/checkout@v4
with:
persist-credentials: false
🧰 Tools
🪛 zizmor (1.29.0)

[warning] 13-13: credential persistence through GitHub Actions artifacts (artipacked): does not set persist-credentials: false

(artipacked)

🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In @.github/workflows/config-check.yml at line 13, Update the
actions/checkout@v4 step to set persist-credentials to false, preventing the job
from storing the GITHUB_TOKEN in local Git configuration.

Source: Linters/SAST tools

- uses: actions/setup-node@v4
with:
node-version: 20
- run: npm run config:check
59 changes: 38 additions & 21 deletions CLAUDE.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,11 +16,16 @@ The app has two modes:

```
benching_machine/
├── shared/
│ └── benchmark-registry.json # CANONICAL config source (hand-edited — see below)
├── scripts/
│ └── generate-benchmark-config.js # Generates configs.generated.js + benchmarkRegistry.js
├── backend/
│ ├── cli.js # CLI entry point (Commander.js)
│ ├── server.js # Optional Express API (port 4000)
│ └── benchmarks/
│ ├── configs.js # MASTER config list + mock runner
│ ├── configs.js # Mock runner logic; imports generated config data
│ ├── configs.generated.js # GENERATED — do not hand-edit
│ ├── compilation.js # CLI compilation benchmarks (Benchmark.js)
│ ├── cpu.js # CPU benchmarks (Fibonacci, Prime, Matrix)
│ ├── memory.js # Memory benchmarks
Expand All @@ -33,14 +38,17 @@ benching_machine/
├── src/
│ ├── App.js # React root, state management
│ ├── index.js # React entry point
│ ├── generated/
│ │ └── benchmarkRegistry.js # GENERATED — do not hand-edit
│ ├── components/
│ │ ├── BenchmarkRunner.js # Control panel + config runner (mirrors configs.js)
│ │ ├── BenchmarkRunner.js # Control panel + config runner; imports generated config data
│ │ ├── HallwayVisualization.js # View mode switcher (hallway / charts)
│ │ ├── Hallway3DView.js # Immersive server rack view
│ │ ├── RackUnitDetail.js # Zoomed per-rack unit detail
│ │ └── CompilerComparisonView.js # Analytical compiler charts
│ └── utils/
│ ├── wasmLoader.js # Dynamic WASM module loader
│ ├── machineProfiles.js # Re-exports generated machineProfiles + helpers
│ └── snapshotManager.js # Save/load/diff benchmark snapshots
├── public/
│ ├── webgl-benchmarks.js # WebGL fragment shader compute
Expand Down Expand Up @@ -114,33 +122,42 @@ npm run build:cheerp

## How to Add a New Compilation Config

1. **`backend/benchmarks/configs.js`** — add an entry to `configurations[]`:
```js
Benchmark configuration has a **single hand-edited source file**: `shared/benchmark-registry.json`. Backend and frontend both import *generated* modules derived from it — never add configs directly to `backend/benchmarks/configs.js` or `src/components/BenchmarkRunner.js`.

1. **`shared/benchmark-registry.json`** — add an entry to `configurations[]`:
```json
{
id: 'js_esbuild',
name: 'esbuild',
desc: 'Go-based bundler/minifier',
color: '#ffcc00',
compilation: {
family: 'js',
toolchain: 'esbuild',
backend: 'Go',
language: 'JavaScript',
optLevel: 'minify',
flags: ['--minify', '--bundle'],
postProcess: [],
status: 'simulated', // change to 'real' when artifacts exist
}
"id": "js_esbuild",
"name": "esbuild",
"desc": "Go-based bundler/minifier",
"color": "#ffcc00",
"compilation": {
"family": "js",
"toolchain": "esbuild",
"backend": "Go",
"language": "JavaScript",
"optLevel": "minify",
"flags": ["--minify", "--bundle"],
"postProcess": [],
"status": "simulated"
},
"multiplier": 1.08
}
```
Add a case to `getMultiplier()` with a realistic relative value.
`status` is one of `'real' | 'simulated' | 'experimental'`. `multiplier` is the simulation factor used by both the CLI mock runner and the frontend fallback simulation.

2. **Regenerate**: run `npm run config:generate`. This regenerates:
- `backend/benchmarks/configs.generated.js` (CommonJS, imported by `backend/benchmarks/configs.js`)
- `src/generated/benchmarkRegistry.js` (ESM, imported by `src/components/BenchmarkRunner.js` and `src/utils/machineProfiles.js`)

2. **`src/components/BenchmarkRunner.js`** — mirror the same entry in its local `configurations[]` array (this file is the frontend's copy).
Commit the regenerated files alongside the registry change. `npm run config:check` (run in CI) fails the build if the generated files are stale relative to the registry.

3. **`src/utils/wasmLoader.js`** — if the config loads a WASM artifact, add a `case` for its `id` pointing to the artifact path.

4. **`BENCHMARKS_STATUS.md`** — add a row documenting real vs simulated status.

Hand-editing the config data in `backend/benchmarks/configs.js`, `src/components/BenchmarkRunner.js`, `src/utils/machineProfiles.js`, or either `*.generated.js` file will be overwritten (or caught by `npm run config:check`) the next time the registry is regenerated.

---

## Data Model
Expand Down Expand Up @@ -246,7 +263,7 @@ These are not yet in the benchmark suite but are worth adding:

## Common Pitfalls

- **BenchmarkRunner.js duplicates configs.js** — both must be kept in sync when adding new configs.
- **Editing generated files directly** — `backend/benchmarks/configs.generated.js` and `src/generated/benchmarkRegistry.js` are overwritten by `npm run config:generate`. Edit `shared/benchmark-registry.json` instead and regenerate.
- **WASM 404s in browser**: `wasmLoader.js` pre-checks file existence; if artifacts aren't built you'll get a silent fallback to simulation.
- **SharedArrayBuffer (for threads)**: requires COOP/COEP headers — the dev server in `package.json` scripts sets these via `REACT_APP_*` env or a custom server wrapper.
- **WebGPU availability**: only Chrome/Edge 113+. The app checks `navigator.gpu` and disables the GPU button if unavailable.
Expand Down
Loading
Loading