Skip to content
Draft
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
37 commits
Select commit Hold shift + click to select a range
5ee09bc
feat: define rights-safe CEFR language assessment profile
seonghobae Aug 27, 2026
3efff35
test: add overall-reporting blueprint fixture
seonghobae Aug 27, 2026
073bb53
test: reproduce CEFR schema and blueprint authority gaps
seonghobae Aug 27, 2026
e013b5b
test: run CEFR authority regressions before validator
seonghobae Aug 27, 2026
7d22777
build: pin Draft 2020-12 validator dependencies
seonghobae Aug 27, 2026
bde27a7
ci: run hash-locked Draft 2020-12 validation
seonghobae Aug 27, 2026
7de84e4
fix: pin exact language-profile revision in CEFR blueprints
seonghobae Aug 27, 2026
ef4370e
fix: pin English language-profile registry snapshot
seonghobae Aug 27, 2026
61b0a70
fix: pin English overall-blueprint language profile
seonghobae Aug 27, 2026
da33604
fix: keep high-stakes fixture invalid for one reason
seonghobae Aug 27, 2026
b84aa90
fix: bind CEFR claims to blueprint and certification authority
seonghobae Aug 27, 2026
ae9f69c
fix: bind linked overall fixture to authorized blueprint
seonghobae Aug 27, 2026
0c4764d
test: isolate incomplete-domain overall failure
seonghobae Aug 27, 2026
56e96c1
test: reject overall result without blueprint authorization
seonghobae Aug 27, 2026
341240a
fix: enforce Draft 2020-12 and blueprint authority
seonghobae Aug 27, 2026
01054ba
docs: clarify allowed CEFR result-envelope evidence
seonghobae Aug 27, 2026
5556033
docs: align CEFR claims and language-profile revisions
seonghobae Aug 27, 2026
63f6be0
docs: replace moving CEFR references with exact revisions
seonghobae Aug 27, 2026
6614536
docs: record Draft validation and blueprint authorization
seonghobae Aug 27, 2026
4ee2772
docs: align CEFR design with executable validation
seonghobae Aug 27, 2026
c5a8f71
docs: record full CEFR schema and authority gates
seonghobae Aug 27, 2026
ad2be22
docs: record CEFR review-hardening implementation
seonghobae Aug 27, 2026
ec9a2aa
docs: align CEFR profile claims and validator
seonghobae Aug 27, 2026
439f721
merge: synchronize CEFR stack and repair ADR collision
seonghobae Sep 19, 2026
8725481
merge: synchronize proposed bootstrap authority
seonghobae Sep 19, 2026
503e73f
fix: reject mutable CEFR revision aliases
seonghobae Sep 19, 2026
4eac808
fix: require evidence-bound RLD revisions
seonghobae Sep 19, 2026
96ed45d
merge: synchronize anchored authority gate
seonghobae Sep 19, 2026
57c8368
fix: validate CEFR calendar timestamps
seonghobae Sep 19, 2026
fb3c169
merge: synchronize cross-runtime timestamp evidence
seonghobae Sep 19, 2026
ac31799
merge: synchronize generated-artifact gate
seonghobae Sep 19, 2026
fa52c3b
merge: synchronize documentation landing gate
seonghobae Sep 19, 2026
1bddab8
Merge canonical bootstrap text-hygiene gate into CEFR profile
seonghobae Sep 30, 2026
150bbf6
fix(cefr): bind reported levels to blueprint
seonghobae Sep 30, 2026
e98e803
merge: integrate bootstrap PR-range quality gate
seonghobae Oct 3, 2026
2659e18
fix(cefr): replace mutable schema identities
seonghobae Oct 3, 2026
8d7bd65
merge: integrate non-empty envelope references
seonghobae Oct 9, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion .github/workflows/quality.yml
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@ name: Learning Contracts Quality

on:
pull_request:
branches: [develop, main]
branches: [develop, main, agent/bootstrap-learning-contracts]
push:
branches: [develop, main]

Expand Down Expand Up @@ -80,3 +80,5 @@ jobs:
raise SystemExit(f"unresolved bootstrap marker {marker!r} in {path}")

print("learning interoperability bootstrap contract validation passed")
- name: Validate CEFR language-assessment profile
run: python3 scripts/validate_cefr_profile.py
3 changes: 3 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,3 +8,6 @@
- Standards traceability baseline for xAPI, cmi5, LTI, QTI, CASE, Open Badges, CLR, and accessibility.
- Versioned learning-domain event envelope schema.
- Repository agent development rules.
- Rights-safe `cwl_cefr_language_assessment/v1` blueprint, task, and immutable domain-result contracts.
- CEFR positive/negative fixture gate covering standard-setting, overall-reporting, probability-mass, and protected-content boundaries.
- CEFR architecture decision, research doctoring, design specification, and implementation plan.
6 changes: 5 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,10 +6,14 @@ Shared, versioned interoperability contracts for the CWL Learning Platform.

This repository contains schemas, profiles, generated-client contracts, and conformance fixtures shared by the Learning Management Platform, Learning Content Studio, Learning Record Store, Psychometrics Commons, and other CWL consumers.

Initial standards portfolio: xAPI 2.0, cmi5 Quartz compatibility, LTI 1.3, QTI 3, CASE 1.1, Open Badges 3.0, CLR 2.0, and accessibility-related contract metadata.
Initial standards portfolio: xAPI 2.0, cmi5 Quartz compatibility, LTI 1.3, QTI 3, CASE 1.1, Open Badges 3.0, CLR 2.0, CEFR language-assessment metadata, and accessibility-related contract metadata.

It contains no product runtime state and no application database.

## Profiles

- `profiles/cwl_cefr_language_assessment/v1` defines a rights-safe CEFR assessment blueprint, task metadata, immutable domain-result snapshot, and executable fixtures. It stores references rather than official descriptor prose, task content, responses, media, or numerical scoring payloads.

## Branching

Product work targets `develop`; release promotion to `main` occurs only after exact-head review and required checks.
Expand Down
14 changes: 10 additions & 4 deletions docs/ARCHITECTURE.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,12 +2,18 @@

This repository owns versioned learning interoperability contracts and no application runtime state.

Primary families: xAPI 2.0, cmi5 Quartz compatibility, LTI 1.3, QTI 3, CASE 1.1, Open Badges 3.0, and CLR 2.0.
Primary families: xAPI 2.0, cmi5 Quartz compatibility, LTI 1.3, QTI 3, CASE 1.1, Open Badges 3.0, CLR 2.0, and rights-safe CEFR language-assessment metadata.

Authority boundaries:
- Learning Management Platform: offerings, enrollment, progression, completion policy.
- Learning Content Studio: authoring state and immutable releases.
- Learning Management Platform: offerings, enrollment, progression, placement/completion policy, and credential references.
- Learning Content Studio: authoring state, tasks, rubrics, media, rights, and immutable releases.
- Learning Record Store: xAPI statements and document resources.
- Psychometrics Commons: assessment sessions, responses, and score snapshots.
- Psychometrics Commons: assessment blueprints/instrument publication, sessions, responses, and immutable result snapshots.
- fast-mlsirm: psychometric estimation, many-facet calibration, standard-setting/cut-score evidence, linking, DIF, uncertainty, and recovery.
- Semantic Data Portal: rights-aware descriptor/RLD/competency catalog references where adopted.
- TEPP: longitudinal, temporal, multilevel, and multiple-membership language-development analysis.
- contextual-orchestrator: bounded AI-rater orchestration; its observations are evidence, not score authority.

The CEFR profile stores immutable references and result-envelope evidence only. It never stores official descriptor prose, authored task content, raw responses, audio, provider payloads, PII, or numerical psychometric payloads.
Comment thread
seonghobae marked this conversation as resolved.
Outdated

Consumers integrate through versioned contracts; cross-repository database access is not part of the architecture.
56 changes: 56 additions & 0 deletions docs/adr/0002-cefr-language-assessment-profile.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,56 @@
# ADR 0002: Rights-safe CEFR language-assessment profile

## Status

Accepted for the active stacked PR; not protected-`develop` truth until merged.

Approved by: ContextualWisdomLab repository owner
Approval date: 2026-08-27

## Context

CWL can compose learning management, content authoring, assessment hosting, psychometric estimation, model orchestration and longitudinal analysis. It lacks a shared contract for reporting a domain-level language-proficiency profile in relation to the Common European Framework of Reference for Languages (CEFR).

A naive contract would create material risks:

- copying official descriptor prose or translations into a public repository;
- treating CEFR as a single equally spaced numerical score;
- averaging incomplete skill results into one overall label;
- confusing alignment with empirical linking or certification;
- moving scoring arithmetic or result authority into an interoperability repository;
- leaking raw responses, audio, task content, provider payloads or PII.

## Decision

Create `profiles/cwl_cefr_language_assessment/v1` as a metadata-only, versioned interoperability profile.

The profile:

1. references the CEFR Companion Volume, language-specific Reference Level Description, descriptors, task/rubric releases, scoring profile, cut-score revision, standard-setting study and validation evidence by immutable opaque identity;
2. supports Pre-A1, A1, A2, A2+, B1, B1+, B2, B2+, C1 and C2 while keeping domain results explicit;
3. models reception, production, interaction and mediation through typed activity-domain codes;
4. requires level probabilities, uncertainty, credible-level sets and descriptor-coverage references for every measured domain;
5. prohibits a reported overall level unless required domains are complete and a versioned overall-reporting policy is pinned;
6. requires standard-setting and linking-validation references before `cefr_linked` or certification-decision claims;
7. requires standard-setting evidence for high-stakes or certification blueprints;
8. rejects copied descriptor/task/response payload fields in fixtures and the quality gate;
9. leaves numerical scoring in fast-mlsirm, instrument/result authority in Psychometrics Commons, content authority in Learning Content Studio, and learner actions in Learning Management Platform.

## Consequences

### Positive

- Consumers can exchange an auditable proficiency profile without cross-service SQL or payload duplication.
- Domain-level uncertainty and incomplete evidence remain visible.
- Rights and certification claims fail closed.
- Future target languages can use different RLD authorities without changing the common contract.

### Negative

- The profile cannot prove that an assessment is linked to the CEFR.
- Consumers must resolve referenced artifacts through their authorized owning systems.
- Full JSON Schema conformance tooling and generated SDKs remain a later release slice.

## Reversal conditions

A new major profile version is required if the CEFR framework representation, level system, domain taxonomy, claim semantics or result envelope changes incompatibly. New optional evidence references may be added compatibly only with conformance fixtures and consumer contract tests.
61 changes: 61 additions & 0 deletions docs/doctoring/CEFR_LANGUAGE_ASSESSMENT.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,61 @@
# CEFR language-assessment research and standards basis

## Product interpretation

The CEFR is a non-prescriptive reference framework and common meta-language for curriculum, materials and assessment. It describes six common levels and three plus levels, and the Companion Volume adds Pre-A1 and expands descriptors for mediation, online interaction, plurilingual/pluricultural competence, phonology and signing.

This repository does not reproduce the descriptor corpus. A contract stores only an immutable `descriptor_reference` and the exact framework/RLD authority needed by the consumer.

## Assessment-development boundary

The 2026 revised *Manual for Language Test Development and Examining—For use with the CEFR* is the current Council of Europe test-development baseline. The separate Manual for Relating Examinations to the CEFR defines transparent, cumulative procedures for supporting a linking claim. The Council of Europe does not verify or validate an examination provider's claimed link.

The product vocabulary therefore distinguishes:

```text
experimental
→ CEFR-aligned
→ CEFR-linked
→ governed certification decision
```

A label cannot advance by editing narrative copy. It requires exact standard-setting, empirical validation and publication evidence.
Comment thread
seonghobae marked this conversation as resolved.
Outdated

## Language-specific content

Reference Level Descriptions are language-specific inventories of linguistic forms and communicative content. Each target-language blueprint must pin one authorized RLD reference or explicitly record that no adequate RLD exists and route the instrument to research-only status.
Comment thread
seonghobae marked this conversation as resolved.
Outdated

## Measurement governance

The 2014 *Standards for Educational and Psychological Testing* governs intended interpretation and use, validity evidence, reliability/precision, fairness, administration, reporting and the rights of test takers. CEFR alignment alone does not establish these properties.

The downstream scientific owner must evaluate, as applicable:

- multidimensional structure and local dependence;
- rater, task, criterion, occasion and scoring-engine facets;
- standard-setting and cut-score uncertainty;
- classification consistency and decision error;
- form linking and anchor stability;
- language, population, mode and accommodation DIF/invariance;
- human/AI rater drift and adjudication;
- true-parameter and classification recovery;
- longitudinal comparability before change interpretation.

## APA 7th references

American Educational Research Association, American Psychological Association, & National Council on Measurement in Education. (2014). *Standards for educational and psychological testing*. American Educational Research Association.

Association of Language Testers in Europe. (2026). *New revised manual for language test development and examining: For use with the CEFR*. Council of Europe.

Council of Europe. (2009). *Relating language examinations to the Common European Framework of Reference for Languages: Learning, teaching, assessment (CEFR): A manual*. Council of Europe.

Council of Europe. (2020). *Common European Framework of Reference for Languages: Learning, teaching, assessment—Companion volume*. Council of Europe Publishing.

## Official sources

- https://www.coe.int/en/web/common-european-framework-reference-languages/cefr-companion-volume-and-its-language-versions
- https://www.coe.int/en/web/common-european-framework-reference-languages/introduction-and-context
- https://www.coe.int/en/web/common-european-framework-reference-languages/cefr-descriptors
- https://www.coe.int/en/web/common-european-framework-reference-languages/cefr-reference-level-descriptions-language-by-language-components-and-forerunners
- https://www.coe.int/en/web/education/-/manual-for-language-test-development-and-examining-1
- https://www.coe.int/en/web/common-european-framework-reference-languages/relating-examinations-to-the-cefr
7 changes: 7 additions & 0 deletions docs/doctoring/STANDARD_TRACEABILITY.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,14 @@ Adoption status and conformance evidence are intentionally separate. `Adopt` rec
| CASE Service | 1.1 Final | https://standards.1edtech.org/case/ | Competency and learning-outcome interchange | Adopt | Not evidenced |
| Open Badges | 3.0 | https://www.1edtech.org/standards/open-badges | Portable achievement credential | Adopt | Not evidenced |
| Comprehensive Learner Record | 2.0 | https://www.1edtech.org/standards/clr | Portable learner achievement record | Adopt | Not evidenced |
| CEFR Companion Volume | 2020 | https://www.coe.int/en/web/common-european-framework-reference-languages/cefr-companion-volume-and-its-language-versions | Framework, levels, communicative modes, descriptor-reference baseline | Adopt as reference-only profile | Contract fixtures on active PR; no assessment-linking claim |
| Manual for Language Test Development and Examining | Revised 2026 edition | https://www.coe.int/en/web/education/-/manual-for-language-test-development-and-examining-1 | CEFR-related language-test development baseline | Adopt for product/scientific traceability | Documentation only |
| Manual for Relating Examinations to the CEFR | Current published manual and supplements | https://www.coe.int/en/web/common-european-framework-reference-languages/relating-examinations-to-the-cefr | Standard-setting, empirical linking, transparent reporting and continuing validation | Adopt for claim gate | No linking study evidenced |
| CEFR Reference Level Descriptions | Language-specific published revisions | https://www.coe.int/en/web/common-european-framework-reference-languages/cefr-reference-level-descriptions-language-by-language-components-and-forerunners | Target-language content specification reference | Adopt by target-language profile | No RLD content copied; authority reference required |
Comment thread
seonghobae marked this conversation as resolved.
Outdated
| Standards for Educational and Psychological Testing | 2014 | https://www.testingstandards.net/ | Intended interpretation/use, validity, reliability/precision, fairness and reporting | Adopt for assessment governance | Downstream product evidence required |
| WCAG | 2.2, W3C Recommendation 2024-12-12 | https://www.w3.org/TR/WCAG22/ | Accessible learning and contract-facing web content | Adopt | Not evidenced |
| ATAG | 2.0, W3C Recommendation 2015-09-24 | https://www.w3.org/TR/ATAG20/ | Accessible authoring-tool contract | Adopt | Not evidenced |

The Council of Europe does not verify or validate an examination provider's CEFR link. This repository must not use the Council of Europe logo or the European emblem to imply certification or endorsement.

Every implementation PR that claims conformance must link the precise standard revision, normative requirement, implementation location, and executable evidence. Certification claims require the applicable certification process and may not be inferred from implementation alone.
115 changes: 115 additions & 0 deletions docs/superpowers/plans/2026-08-27-cefr-language-assessment-profile.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,115 @@
# CEFR Language Assessment Profile Implementation Plan

> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.

**Goal:** Publish a rights-safe CEFR assessment blueprint, task, and immutable domain-result contract with executable positive and negative fixtures.

**Architecture:** The contract repository carries metadata references and conformance evidence only. Learning Content Studio owns authored content, Psychometrics Commons owns sessions/results, fast-mlsirm owns numerical psychometrics, LMS owns placement/completion actions, and TEPP owns longitudinal analysis.

**Tech Stack:** JSON Schema Draft 2020-12, JSON fixtures, Markdown ADR/doctoring, GitHub Actions with Python 3 standard library.

**Spec:** `docs/superpowers/specs/2026-08-27-cefr-language-assessment-profile-design.md`

## Global Constraints

- Target branch remains the bootstrap PR branch until PR #1 integrates.
- No runtime database or application state.
- No official CEFR descriptor prose, translations, task content, responses, audio, provider payloads or PII.
- No Python/Rust numerical arithmetic.
- Every claim and artifact is version-pinned.
- Exact-head review and required checks remain mandatory.

---

### Task 1: Common CEFR definitions

**Files:**
- Create: `profiles/cwl_cefr_language_assessment/v1/schemas/cefr-common.schema.json`

**Interfaces:**
- Produces: CEFR level, activity-domain, communication-mode, language-tag, exact-reference, digest and timestamp definitions.

- [x] Define Pre-A1, A1/A2/B1/B2/C1/C2 and A2+/B1+/B2+ codes.
- [x] Define reception, production, interaction and mediation.
- [x] Define activity domains without descriptor prose.
- [x] Pin Draft 2020-12, schema version and published `$id`.

### Task 2: Blueprint and task contracts

**Files:**
- Create: `profiles/cwl_cefr_language_assessment/v1/schemas/assessment-blueprint.schema.json`
- Create: `profiles/cwl_cefr_language_assessment/v1/schemas/task-specification.schema.json`

**Interfaces:**
- Consumes: common definitions from Task 1.
- Produces: immutable assessment and task metadata boundaries.

- [x] Require target-language RLD, instrument, scoring, cut-score and validation references.
- [x] Require standard-setting evidence for high-stakes/certification blueprints.
- [x] Require mode-specific task evidence and rubric references for constructed responses.
- [x] Prohibit copied descriptor/task payload fields by closed schemas.

### Task 3: Immutable result contract

**Files:**
- Create: `profiles/cwl_cefr_language_assessment/v1/schemas/cefr-result-snapshot.schema.json`

**Interfaces:**
- Consumes: common definitions from Task 1.
- Produces: `cwl_cefr_language_assessment/result_snapshot/v1`.

- [x] Require domain probabilities, uncertainty and descriptor coverage for measured domains.
- [x] Require explicit non-measurement states instead of invented scores.
- [x] Gate overall reporting on complete required domains and a reporting policy.
- [x] Gate `cefr_linked` and certification claims on standard-setting and empirical validation references.

### Task 4: Positive and negative fixtures

**Files:**
- Create: `profiles/cwl_cefr_language_assessment/v1/conformance/valid/*.json`
- Create: `profiles/cwl_cefr_language_assessment/v1/conformance/invalid/*.json`

**Interfaces:**
- Produces: executable examples for downstream contract tests.

- [x] Add an English A1–B2 placement blueprint.
- [x] Add a reference-only reading-task specification.
- [x] Add profile-only and linked-overall result examples.
- [x] Add failures for missing standard setting, copied descriptor text, incomplete overall reporting and non-unit probability mass.

### Task 5: Governance and traceability

**Files:**
- Create: `profiles/cwl_cefr_language_assessment/v1/README.md`
- Create: `docs/adr/0002-cefr-language-assessment-profile.md`
- Create: `docs/doctoring/CEFR_LANGUAGE_ASSESSMENT.md`
- Modify: `README.md`
- Modify: `CHANGELOG.md`
- Modify: `docs/ARCHITECTURE.md`
- Modify: `docs/doctoring/STANDARD_TRACEABILITY.md`

**Interfaces:**
- Produces: reviewer-readable authority, rights, claim and research boundaries.

- [x] Record the 2020 Companion Volume and 2026 revised test-development manual.
- [x] State that the Council of Europe does not validate provider linking claims.
- [x] Separate CEFR alignment, linking and certification-decision status.
- [x] Record downstream repository ownership and next actions.

### Task 6: Executable fixture gate

**Files:**
- Create: `scripts/validate_cefr_profile.py`
- Modify: `.github/workflows/quality.yml`

**Interfaces:**
- Consumes: all Task 1–4 schemas and fixtures.
- Produces: `Learning Contracts Quality` exact-head evidence.

- [x] Parse every JSON file.
- [x] Verify schema metadata and published paths.
- [x] Accept all valid fixtures.
- [x] Reject each negative fixture for its intended reason.
- [x] Reject forbidden payload fields, duplicate domains and invalid probability mass.
- [ ] Obtain terminal hosted checks on the unchanged exact head.
- [ ] Obtain qualifying independent review after the parent branch is ready.
Loading