Human factors: add research evidence and propose fire drill experiment - #24
Merged
Conversation
twaugh
commented
Mar 16, 2026
twaugh
commented
Contributor
- Add research evidence from established automation and AI-specific studies to human-factors.md
- Add a new section questioning whether the vision's two-point model (strategic intent + guarded paths) is sufficient, given that the oversight quality it depends on degrades without ongoing hands-on involvement
- Connect human factors back into the autonomy spectrum as open questions, so autonomy decisions can account for human effectiveness
- Propose experiment 002: an agent outage fire drill (chaos engineering for human capability) to test whether skill atrophy is real in this context
ralphbean
reviewed
Mar 16, 2026
ralphbean
left a comment
ralphbean
left a comment
Member
There was a problem hiding this comment.
Looks good, but I want to move the experiments to experiments/ out of docs/experiments/.
Can you move it there?
Ground the human factors concerns in established research (Bainbridge, Endsley & Kiris, Parasuraman & Manzey) and recent AI-specific studies (Anthropic skill formation, METR productivity trial). Add a new section questioning whether the vision's two-point model of human participation (strategic intent + guarded paths) is sufficient, given evidence that the oversight quality it depends on degrades without ongoing hands-on involvement. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add a note to graduation criteria acknowledging that autonomy changes the human role, not just the agent's. Add open questions about whether human engagement should be a graduation criterion, whether intermediate automation levels preserve better oversight, and whether engagement metrics should trigger autonomy revocation. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
twaugh
force-pushed
the
human-factors-evidence
branch
from
March 17, 2026 12:08
25fa5d9 to
0d16647
Compare
Contributor
Author
Moved. |
Chaos engineering for human capability — disable agents for two weeks after months of autonomy and measure whether the team can still operate at pre-autonomy baselines. Tests skill atrophy directly and surfaces agent dependency as an organisational risk. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
twaugh
force-pushed
the
human-factors-evidence
branch
from
March 17, 2026 12:20
0d16647 to
2ab52ad
Compare
ralphbean
approved these changes
Mar 17, 2026
ktdreyer
reviewed
Mar 19, 2026
|
|
||
| ### The ironies of automation | ||
|
|
||
| Bainbridge's foundational ["Ironies of Automation" (1983)](https://doi.org/10.1016/0005-1098(83)90046-8) identified the central paradox: automating a task removes the practice that keeps operators skilled enough to intervene when automation fails. The more reliable the automation, the worse the problem — because operators have fewer opportunities to exercise their skills, and failures are rarer but harder to catch. This remains one of the most-cited papers in human factors research and its core argument is directly applicable: if agents handle all routine development, the domain experts who approve guarded-path changes lose the practice that makes their approval meaningful. |
There was a problem hiding this comment.
Thanks for these citations, particularly these first two, "Irony of Automation" and "The Out-of-the-Loop Performance Problem".
I've seen these dynamics play out a couple times. I didn't know there were formal terms or papers for this.
2 tasks
Merged
13 tasks
waynesun09
added a commit
that referenced
this pull request
Aug 22, 2026
…ndbox image pi's google-vertex provider is Gemini-only and the upstream anthropic-vertex provider (earendil-works/pi#5262) is still open, so the interim provider from the runtime plan is adopted: twoGiants/pi-anthropic-vertex v0.1.13 (commit d3c9d10d, MIT), a thin shim that registers "anthropic-vertex" and delegates streaming to pi's built-in Anthropic provider through an AnthropicVertex client. - pinned by git tag + SHA256 of the GitHub tag tarball; dependencies from its vendored lockfile via npm ci --omit=dev --omit=peer --ignore-scripts (29 packages, no install scripts). --omit=peer mirrors pi's own package installer: the peer deps are pi itself, which the extension loader aliases to the running pi, so a local copy would be the wrong version. Verified with a local layer build. - installed under /opt/pi-extensions (sandbox.SandboxPiExtensionsDir), root-owned, read-only and outside PI_CODING_AGENT_DIR, so pi never auto-loads it; Run will pass runtime.PiVertexExtensionPath with -e. - imports checked against pi 0.84.2's pi-ai/compat exports; the mirrored option mapping was last synced upstream to pi 0.81.1 (sync issue #24 open) — documented as a re-check item on PI_VERSION bumps. - renovate tracks the tag (github-tags, no automerge); README pin table, docs/runtimes.md constraints and the image test updated. Assisted-by: Claude (fix) Signed-off-by: Wayne Sun <gsun@redhat.com>
This was referenced Sep 2, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.