Repository navigation
feat: define rights-safe CEFR language assessment profile #5
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Draft
seonghobae
wants to merge
37
commits into
agent/bootstrap-learning-contracts
Choose a base branch
from
feat/cefr-language-assessment-contracts
base: agent/bootstrap-learning-contracts
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Draft
Changes from 1 commit
Commits
Show all changes
37 commits
Select commit
Hold shift + click to select a range
5ee09bc
feat: define rights-safe CEFR language assessment profile
seonghobae 3efff35
test: add overall-reporting blueprint fixture
seonghobae 073bb53
test: reproduce CEFR schema and blueprint authority gaps
seonghobae e013b5b
test: run CEFR authority regressions before validator
seonghobae 7d22777
build: pin Draft 2020-12 validator dependencies
seonghobae bde27a7
ci: run hash-locked Draft 2020-12 validation
seonghobae 7de84e4
fix: pin exact language-profile revision in CEFR blueprints
seonghobae ef4370e
fix: pin English language-profile registry snapshot
seonghobae 61b0a70
fix: pin English overall-blueprint language profile
seonghobae da33604
fix: keep high-stakes fixture invalid for one reason
seonghobae b84aa90
fix: bind CEFR claims to blueprint and certification authority
seonghobae ae9f69c
fix: bind linked overall fixture to authorized blueprint
seonghobae 0c4764d
test: isolate incomplete-domain overall failure
seonghobae 56e96c1
test: reject overall result without blueprint authorization
seonghobae 341240a
fix: enforce Draft 2020-12 and blueprint authority
seonghobae 01054ba
docs: clarify allowed CEFR result-envelope evidence
seonghobae 5556033
docs: align CEFR claims and language-profile revisions
seonghobae 63f6be0
docs: replace moving CEFR references with exact revisions
seonghobae 6614536
docs: record Draft validation and blueprint authorization
seonghobae 4ee2772
docs: align CEFR design with executable validation
seonghobae c5a8f71
docs: record full CEFR schema and authority gates
seonghobae ad2be22
docs: record CEFR review-hardening implementation
seonghobae ec9a2aa
docs: align CEFR profile claims and validator
seonghobae 439f721
merge: synchronize CEFR stack and repair ADR collision
seonghobae 8725481
merge: synchronize proposed bootstrap authority
seonghobae 503e73f
fix: reject mutable CEFR revision aliases
seonghobae 4eac808
fix: require evidence-bound RLD revisions
seonghobae 96ed45d
merge: synchronize anchored authority gate
seonghobae 57c8368
fix: validate CEFR calendar timestamps
seonghobae fb3c169
merge: synchronize cross-runtime timestamp evidence
seonghobae ac31799
merge: synchronize generated-artifact gate
seonghobae fa52c3b
merge: synchronize documentation landing gate
seonghobae 1bddab8
Merge canonical bootstrap text-hygiene gate into CEFR profile
seonghobae 150bbf6
fix(cefr): bind reported levels to blueprint
seonghobae e98e803
merge: integrate bootstrap PR-range quality gate
seonghobae 2659e18
fix(cefr): replace mutable schema identities
seonghobae 8d7bd65
merge: integrate non-empty envelope references
seonghobae File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,56 @@ | ||
| # ADR 0002: Rights-safe CEFR language-assessment profile | ||
|
|
||
| ## Status | ||
|
|
||
| Accepted for the active stacked PR; not protected-`develop` truth until merged. | ||
|
|
||
| Approved by: ContextualWisdomLab repository owner | ||
| Approval date: 2026-08-27 | ||
|
|
||
| ## Context | ||
|
|
||
| CWL can compose learning management, content authoring, assessment hosting, psychometric estimation, model orchestration and longitudinal analysis. It lacks a shared contract for reporting a domain-level language-proficiency profile in relation to the Common European Framework of Reference for Languages (CEFR). | ||
|
|
||
| A naive contract would create material risks: | ||
|
|
||
| - copying official descriptor prose or translations into a public repository; | ||
| - treating CEFR as a single equally spaced numerical score; | ||
| - averaging incomplete skill results into one overall label; | ||
| - confusing alignment with empirical linking or certification; | ||
| - moving scoring arithmetic or result authority into an interoperability repository; | ||
| - leaking raw responses, audio, task content, provider payloads or PII. | ||
|
|
||
| ## Decision | ||
|
|
||
| Create `profiles/cwl_cefr_language_assessment/v1` as a metadata-only, versioned interoperability profile. | ||
|
|
||
| The profile: | ||
|
|
||
| 1. references the CEFR Companion Volume, language-specific Reference Level Description, descriptors, task/rubric releases, scoring profile, cut-score revision, standard-setting study and validation evidence by immutable opaque identity; | ||
| 2. supports Pre-A1, A1, A2, A2+, B1, B1+, B2, B2+, C1 and C2 while keeping domain results explicit; | ||
| 3. models reception, production, interaction and mediation through typed activity-domain codes; | ||
| 4. requires level probabilities, uncertainty, credible-level sets and descriptor-coverage references for every measured domain; | ||
| 5. prohibits a reported overall level unless required domains are complete and a versioned overall-reporting policy is pinned; | ||
| 6. requires standard-setting and linking-validation references before `cefr_linked` or certification-decision claims; | ||
| 7. requires standard-setting evidence for high-stakes or certification blueprints; | ||
| 8. rejects copied descriptor/task/response payload fields in fixtures and the quality gate; | ||
| 9. leaves numerical scoring in fast-mlsirm, instrument/result authority in Psychometrics Commons, content authority in Learning Content Studio, and learner actions in Learning Management Platform. | ||
|
|
||
| ## Consequences | ||
|
|
||
| ### Positive | ||
|
|
||
| - Consumers can exchange an auditable proficiency profile without cross-service SQL or payload duplication. | ||
| - Domain-level uncertainty and incomplete evidence remain visible. | ||
| - Rights and certification claims fail closed. | ||
| - Future target languages can use different RLD authorities without changing the common contract. | ||
|
|
||
| ### Negative | ||
|
|
||
| - The profile cannot prove that an assessment is linked to the CEFR. | ||
| - Consumers must resolve referenced artifacts through their authorized owning systems. | ||
| - Full JSON Schema conformance tooling and generated SDKs remain a later release slice. | ||
|
|
||
| ## Reversal conditions | ||
|
|
||
| A new major profile version is required if the CEFR framework representation, level system, domain taxonomy, claim semantics or result envelope changes incompatibly. New optional evidence references may be added compatibly only with conformance fixtures and consumer contract tests. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,61 @@ | ||
| # CEFR language-assessment research and standards basis | ||
|
|
||
| ## Product interpretation | ||
|
|
||
| The CEFR is a non-prescriptive reference framework and common meta-language for curriculum, materials and assessment. It describes six common levels and three plus levels, and the Companion Volume adds Pre-A1 and expands descriptors for mediation, online interaction, plurilingual/pluricultural competence, phonology and signing. | ||
|
|
||
| This repository does not reproduce the descriptor corpus. A contract stores only an immutable `descriptor_reference` and the exact framework/RLD authority needed by the consumer. | ||
|
|
||
| ## Assessment-development boundary | ||
|
|
||
| The 2026 revised *Manual for Language Test Development and Examining—For use with the CEFR* is the current Council of Europe test-development baseline. The separate Manual for Relating Examinations to the CEFR defines transparent, cumulative procedures for supporting a linking claim. The Council of Europe does not verify or validate an examination provider's claimed link. | ||
|
|
||
| The product vocabulary therefore distinguishes: | ||
|
|
||
| ```text | ||
| experimental | ||
| → CEFR-aligned | ||
| → CEFR-linked | ||
| → governed certification decision | ||
| ``` | ||
|
|
||
| A label cannot advance by editing narrative copy. It requires exact standard-setting, empirical validation and publication evidence. | ||
|
seonghobae marked this conversation as resolved.
Outdated
|
||
|
|
||
| ## Language-specific content | ||
|
|
||
| Reference Level Descriptions are language-specific inventories of linguistic forms and communicative content. Each target-language blueprint must pin one authorized RLD reference or explicitly record that no adequate RLD exists and route the instrument to research-only status. | ||
|
seonghobae marked this conversation as resolved.
Outdated
|
||
|
|
||
| ## Measurement governance | ||
|
|
||
| The 2014 *Standards for Educational and Psychological Testing* governs intended interpretation and use, validity evidence, reliability/precision, fairness, administration, reporting and the rights of test takers. CEFR alignment alone does not establish these properties. | ||
|
|
||
| The downstream scientific owner must evaluate, as applicable: | ||
|
|
||
| - multidimensional structure and local dependence; | ||
| - rater, task, criterion, occasion and scoring-engine facets; | ||
| - standard-setting and cut-score uncertainty; | ||
| - classification consistency and decision error; | ||
| - form linking and anchor stability; | ||
| - language, population, mode and accommodation DIF/invariance; | ||
| - human/AI rater drift and adjudication; | ||
| - true-parameter and classification recovery; | ||
| - longitudinal comparability before change interpretation. | ||
|
|
||
| ## APA 7th references | ||
|
|
||
| American Educational Research Association, American Psychological Association, & National Council on Measurement in Education. (2014). *Standards for educational and psychological testing*. American Educational Research Association. | ||
|
|
||
| Association of Language Testers in Europe. (2026). *New revised manual for language test development and examining: For use with the CEFR*. Council of Europe. | ||
|
|
||
| Council of Europe. (2009). *Relating language examinations to the Common European Framework of Reference for Languages: Learning, teaching, assessment (CEFR): A manual*. Council of Europe. | ||
|
|
||
| Council of Europe. (2020). *Common European Framework of Reference for Languages: Learning, teaching, assessment—Companion volume*. Council of Europe Publishing. | ||
|
|
||
| ## Official sources | ||
|
|
||
| - https://www.coe.int/en/web/common-european-framework-reference-languages/cefr-companion-volume-and-its-language-versions | ||
| - https://www.coe.int/en/web/common-european-framework-reference-languages/introduction-and-context | ||
| - https://www.coe.int/en/web/common-european-framework-reference-languages/cefr-descriptors | ||
| - https://www.coe.int/en/web/common-european-framework-reference-languages/cefr-reference-level-descriptions-language-by-language-components-and-forerunners | ||
| - https://www.coe.int/en/web/education/-/manual-for-language-test-development-and-examining-1 | ||
| - https://www.coe.int/en/web/common-european-framework-reference-languages/relating-examinations-to-the-cefr | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
115 changes: 115 additions & 0 deletions
115
docs/superpowers/plans/2026-08-27-cefr-language-assessment-profile.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,115 @@ | ||
| # CEFR Language Assessment Profile Implementation Plan | ||
|
|
||
| > **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. | ||
|
|
||
| **Goal:** Publish a rights-safe CEFR assessment blueprint, task, and immutable domain-result contract with executable positive and negative fixtures. | ||
|
|
||
| **Architecture:** The contract repository carries metadata references and conformance evidence only. Learning Content Studio owns authored content, Psychometrics Commons owns sessions/results, fast-mlsirm owns numerical psychometrics, LMS owns placement/completion actions, and TEPP owns longitudinal analysis. | ||
|
|
||
| **Tech Stack:** JSON Schema Draft 2020-12, JSON fixtures, Markdown ADR/doctoring, GitHub Actions with Python 3 standard library. | ||
|
|
||
| **Spec:** `docs/superpowers/specs/2026-08-27-cefr-language-assessment-profile-design.md` | ||
|
|
||
| ## Global Constraints | ||
|
|
||
| - Target branch remains the bootstrap PR branch until PR #1 integrates. | ||
| - No runtime database or application state. | ||
| - No official CEFR descriptor prose, translations, task content, responses, audio, provider payloads or PII. | ||
| - No Python/Rust numerical arithmetic. | ||
| - Every claim and artifact is version-pinned. | ||
| - Exact-head review and required checks remain mandatory. | ||
|
|
||
| --- | ||
|
|
||
| ### Task 1: Common CEFR definitions | ||
|
|
||
| **Files:** | ||
| - Create: `profiles/cwl_cefr_language_assessment/v1/schemas/cefr-common.schema.json` | ||
|
|
||
| **Interfaces:** | ||
| - Produces: CEFR level, activity-domain, communication-mode, language-tag, exact-reference, digest and timestamp definitions. | ||
|
|
||
| - [x] Define Pre-A1, A1/A2/B1/B2/C1/C2 and A2+/B1+/B2+ codes. | ||
| - [x] Define reception, production, interaction and mediation. | ||
| - [x] Define activity domains without descriptor prose. | ||
| - [x] Pin Draft 2020-12, schema version and published `$id`. | ||
|
|
||
| ### Task 2: Blueprint and task contracts | ||
|
|
||
| **Files:** | ||
| - Create: `profiles/cwl_cefr_language_assessment/v1/schemas/assessment-blueprint.schema.json` | ||
| - Create: `profiles/cwl_cefr_language_assessment/v1/schemas/task-specification.schema.json` | ||
|
|
||
| **Interfaces:** | ||
| - Consumes: common definitions from Task 1. | ||
| - Produces: immutable assessment and task metadata boundaries. | ||
|
|
||
| - [x] Require target-language RLD, instrument, scoring, cut-score and validation references. | ||
| - [x] Require standard-setting evidence for high-stakes/certification blueprints. | ||
| - [x] Require mode-specific task evidence and rubric references for constructed responses. | ||
| - [x] Prohibit copied descriptor/task payload fields by closed schemas. | ||
|
|
||
| ### Task 3: Immutable result contract | ||
|
|
||
| **Files:** | ||
| - Create: `profiles/cwl_cefr_language_assessment/v1/schemas/cefr-result-snapshot.schema.json` | ||
|
|
||
| **Interfaces:** | ||
| - Consumes: common definitions from Task 1. | ||
| - Produces: `cwl_cefr_language_assessment/result_snapshot/v1`. | ||
|
|
||
| - [x] Require domain probabilities, uncertainty and descriptor coverage for measured domains. | ||
| - [x] Require explicit non-measurement states instead of invented scores. | ||
| - [x] Gate overall reporting on complete required domains and a reporting policy. | ||
| - [x] Gate `cefr_linked` and certification claims on standard-setting and empirical validation references. | ||
|
|
||
| ### Task 4: Positive and negative fixtures | ||
|
|
||
| **Files:** | ||
| - Create: `profiles/cwl_cefr_language_assessment/v1/conformance/valid/*.json` | ||
| - Create: `profiles/cwl_cefr_language_assessment/v1/conformance/invalid/*.json` | ||
|
|
||
| **Interfaces:** | ||
| - Produces: executable examples for downstream contract tests. | ||
|
|
||
| - [x] Add an English A1–B2 placement blueprint. | ||
| - [x] Add a reference-only reading-task specification. | ||
| - [x] Add profile-only and linked-overall result examples. | ||
| - [x] Add failures for missing standard setting, copied descriptor text, incomplete overall reporting and non-unit probability mass. | ||
|
|
||
| ### Task 5: Governance and traceability | ||
|
|
||
| **Files:** | ||
| - Create: `profiles/cwl_cefr_language_assessment/v1/README.md` | ||
| - Create: `docs/adr/0002-cefr-language-assessment-profile.md` | ||
| - Create: `docs/doctoring/CEFR_LANGUAGE_ASSESSMENT.md` | ||
| - Modify: `README.md` | ||
| - Modify: `CHANGELOG.md` | ||
| - Modify: `docs/ARCHITECTURE.md` | ||
| - Modify: `docs/doctoring/STANDARD_TRACEABILITY.md` | ||
|
|
||
| **Interfaces:** | ||
| - Produces: reviewer-readable authority, rights, claim and research boundaries. | ||
|
|
||
| - [x] Record the 2020 Companion Volume and 2026 revised test-development manual. | ||
| - [x] State that the Council of Europe does not validate provider linking claims. | ||
| - [x] Separate CEFR alignment, linking and certification-decision status. | ||
| - [x] Record downstream repository ownership and next actions. | ||
|
|
||
| ### Task 6: Executable fixture gate | ||
|
|
||
| **Files:** | ||
| - Create: `scripts/validate_cefr_profile.py` | ||
| - Modify: `.github/workflows/quality.yml` | ||
|
|
||
| **Interfaces:** | ||
| - Consumes: all Task 1–4 schemas and fixtures. | ||
| - Produces: `Learning Contracts Quality` exact-head evidence. | ||
|
|
||
| - [x] Parse every JSON file. | ||
| - [x] Verify schema metadata and published paths. | ||
| - [x] Accept all valid fixtures. | ||
| - [x] Reject each negative fixture for its intended reason. | ||
| - [x] Reject forbidden payload fields, duplicate domains and invalid probability mass. | ||
| - [ ] Obtain terminal hosted checks on the unchanged exact head. | ||
| - [ ] Obtain qualifying independent review after the parent branch is ready. |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.