Skip to content

chore(web): #363 experiment, observe run creation for the baseline bot push - #748

Closed
thomasluizon wants to merge 2 commits into
mainfrom
chore/ticket-363-trigger-experiment
Closed

chore(web): #363 experiment, observe run creation for the baseline bot push#748
thomasluizon wants to merge 2 commits into
mainfrom
chore/ticket-363-trigger-experiment

Conversation

@thomasluizon

Copy link
Copy Markdown
Owner

Throwaway. Do not merge. It is closed and its branch deleted once the observation is recorded on #363.

What this is measuring

#363 asks whether GitHub creates pull_request runs for the baseline commit that visual.yml pushes with the workflow's GITHUB_TOKEN.

The comment at .github/workflows/visual.yml:86 says GitHub never starts runs for events that token causes, so the operator's next real push is what puts the 21 required contexts on the bot head. Pullfrog's review of #744 says GitHub documents an exception: a GITHUB_TOKEN update to an existing pull request creates synchronize runs held in an approval-required state.

History could not settle it. The only bot baseline push that exists, 309967d4 on #744, carried [skip ci], which suppresses run creation on its own. Nothing in that sample separates token suppression from marker suppression.

Why it can be settled now

The current workflow no longer carries [skip ci]. #744 removed it. So a re-seed run against main as it stands is exactly the clean experiment, with no workflow edit needed.

The design

  • One baseline is deleted, so the re-seed produces a non-empty staged diff and the bot push actually happens instead of hitting the "Baselines already up to date" branch.
  • The visual:update label is applied at creation, so HAS_UPDATE_LABEL is true and the first run seeds.
  • The control is this PR's own opening. That push is mine, from a real user account. If its runs appear, GitHub Actions is healthy, which rules out the platform incident seen on 2026-08-23 around 19:20Z where new heads got no required checks at all for about an hour.
  • The measurement reads the runs API for the bot head SHA itself, not the branch, per #363's recorded constraint. Reading the branch would not separate token suppression from [skip ci] suppression.

My head, the control: acb3ebc0da51f9955057161159610ff2ecfe7f27.

What is deliberately not tested

A checks re-run. GitHub re-runs retain the original run's GITHUB_SHA and GITHUB_REF, so the contexts land on the pre-baseline SHA. #363 records that constraint explicitly, and a re-run is never the correct operator step.

Throwaway. This branch exists only to run #363's experiment: trigger a
labeled re-seed and observe whether GitHub creates pull_request runs for
the baseline commit the workflow's GITHUB_TOKEN pushes.

Deleting one baseline guarantees the re-seed produces a non-empty staged
diff, so the bot push actually happens rather than hitting the workflow's
"Baselines already up to date" branch.

This branch is deleted once the observation is recorded.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@thomasluizon thomasluizon added the visual:update Re-seed hermetic visual baselines on the visual.yml gate label Aug 24, 2026
@vercel

vercel Bot commented Aug 24, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment
Project Deployment Actions Updated (UTC)
orbit-ui-mobile-web Ignored Ignored Aug 24, 2026 1:38am

Request Review

pullfrog[bot]
pullfrog Bot previously approved these changes Aug 24, 2026

@pullfrog pullfrog Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ No new issues found.

Reviewed changes The review covered the deliberate visual-baseline deletion and the seed workflow behavior it triggers.

  • Forced baseline seed — Removing the LFS-tracked login snapshot guarantees the labeled visual run recreates and stages that exact baseline, producing the bot push needed for the experiment.
  • Throwaway scope — The PR clearly records that the deletion is experimental only and that the PR and branch are to be removed after the observation, not merged.

Pullfrog  | View workflow run | Using GPT Sol𝕏

@thomasluizon

Copy link
Copy Markdown
Owner Author

Result recorded. Closing.

The experiment answered the question. Full output is on #363, and the fix is in #749.

Runs ARE created for the bot head, and held. GET /actions/runs?head_sha=353d1cb5... returned 9: eight pull_request runs, every one completed with conclusion=action_required, plus the dynamic run for this PR. The control, my own push acb3ebc0, produced 14 runs, so Actions was healthy and the platform incident from earlier that day is ruled out.

So visual.yml's comment was wrong on the mechanism. GitHub does not refuse to start runs for a GITHUB_TOKEN push; it gates them behind approval. A held run publishes no contexts, which is why the head looked unchecked and why the wrong explanation survived.

The operator step is to approve, verified live: POST .../actions/runs/32680420333/approve flipped the held Guards run to in_progress. Never a re-run, which retains the original GITHUB_SHA.

Closing without merging, as stated when this was opened. The deleted baseline was only ever a device to force the bot push.

@thomasluizon
thomasluizon deleted the chore/ticket-363-trigger-experiment branch August 24, 2026 01:47
@sonarqubecloud

Copy link
Copy Markdown

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

visual:update Re-seed hermetic visual baselines on the visual.yml gate

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant