Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
23 commits
Select commit Hold shift + click to select a range
5ee09bc
feat: define rights-safe CEFR language assessment profile
seonghobae Aug 27, 2026
3efff35
test: add overall-reporting blueprint fixture
seonghobae Aug 27, 2026
073bb53
test: reproduce CEFR schema and blueprint authority gaps
seonghobae Aug 27, 2026
e013b5b
test: run CEFR authority regressions before validator
seonghobae Aug 27, 2026
7d22777
build: pin Draft 2020-12 validator dependencies
seonghobae Aug 27, 2026
bde27a7
ci: run hash-locked Draft 2020-12 validation
seonghobae Aug 27, 2026
7de84e4
fix: pin exact language-profile revision in CEFR blueprints
seonghobae Aug 27, 2026
ef4370e
fix: pin English language-profile registry snapshot
seonghobae Aug 27, 2026
61b0a70
fix: pin English overall-blueprint language profile
seonghobae Aug 27, 2026
da33604
fix: keep high-stakes fixture invalid for one reason
seonghobae Aug 27, 2026
b84aa90
fix: bind CEFR claims to blueprint and certification authority
seonghobae Aug 27, 2026
ae9f69c
fix: bind linked overall fixture to authorized blueprint
seonghobae Aug 27, 2026
0c4764d
test: isolate incomplete-domain overall failure
seonghobae Aug 27, 2026
56e96c1
test: reject overall result without blueprint authorization
seonghobae Aug 27, 2026
341240a
fix: enforce Draft 2020-12 and blueprint authority
seonghobae Aug 27, 2026
01054ba
docs: clarify allowed CEFR result-envelope evidence
seonghobae Aug 27, 2026
5556033
docs: align CEFR claims and language-profile revisions
seonghobae Aug 27, 2026
63f6be0
docs: replace moving CEFR references with exact revisions
seonghobae Aug 27, 2026
6614536
docs: record Draft validation and blueprint authorization
seonghobae Aug 27, 2026
4ee2772
docs: align CEFR design with executable validation
seonghobae Aug 27, 2026
c5a8f71
docs: record full CEFR schema and authority gates
seonghobae Aug 27, 2026
ad2be22
docs: record CEFR review-hardening implementation
seonghobae Aug 27, 2026
ec9a2aa
docs: align CEFR profile claims and validator
seonghobae Aug 27, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 18 additions & 1 deletion .github/workflows/quality.yml
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@ name: Learning Contracts Quality

on:
pull_request:
branches: [develop, main]
branches: [develop, main, agent/bootstrap-learning-contracts]
push:
branches: [develop, main]

Expand All @@ -22,6 +22,14 @@ jobs:
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
with:
persist-credentials: false
- name: Set up pinned Python
uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
with:
python-version: "3.12"
- name: Install hash-locked contract validator
run: >-
python -m pip install --disable-pip-version-check --only-binary=:all:
--require-hashes -r requirements-contracts-ci-hashes.txt
- name: Validate documentation and schemas
shell: python3 {0}
run: |
Expand All @@ -39,6 +47,7 @@ jobs:
Path("docs/ARCHITECTURE.md"),
Path("docs/adr/0001-contract-authority-boundary.md"),
Path("docs/doctoring/STANDARD_TRACEABILITY.md"),
Path("requirements-contracts-ci-hashes.txt"),
schema_path,
]
missing = [str(path) for path in required if not path.is_file()]
Expand Down Expand Up @@ -80,3 +89,11 @@ jobs:
raise SystemExit(f"unresolved bootstrap marker {marker!r} in {path}")

print("learning interoperability bootstrap contract validation passed")
- name: Run CEFR schema and blueprint authority regressions
run: python -m unittest -v tests/test_validate_cefr_profile.py
- name: Validate CEFR language-assessment profile
run: python scripts/validate_cefr_profile.py
- name: Compile validator and tests
run: python -m compileall -q scripts tests
- name: Verify clean diff
run: git diff --check
6 changes: 6 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,3 +8,9 @@
- Standards traceability baseline for xAPI, cmi5, LTI, QTI, CASE, Open Badges, CLR, and accessibility.
- Versioned learning-domain event envelope schema.
- Repository agent development rules.
- Rights-safe `cwl_cefr_language_assessment/v1` blueprint, task, and immutable domain-result contracts.
- Hash-locked JSON Schema Draft 2020-12 validation with offline `$ref` resolution and metaschema checks.
- CEFR positive/negative fixture gates covering standard-setting, protected content, probability mass, required-domain completeness, blueprint overall-reporting authority, and exact reporting-policy equality.
- Exact target-language profile/RLD revision or dated-snapshot requirement; mutable revision aliases fail closed.
- Distinct `cefr_aligned`, `cefr_linked`, and certification-decision evidence gates, including governed certification authority and policy references.
- CEFR architecture decision, research doctoring, fixed standards traceability, design specification, implementation plan, and focused regression tests.
6 changes: 5 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,10 +6,14 @@ Shared, versioned interoperability contracts for the CWL Learning Platform.

This repository contains schemas, profiles, generated-client contracts, and conformance fixtures shared by the Learning Management Platform, Learning Content Studio, Learning Record Store, Psychometrics Commons, and other CWL consumers.

Initial standards portfolio: xAPI 2.0, cmi5 Quartz compatibility, LTI 1.3, QTI 3, CASE 1.1, Open Badges 3.0, CLR 2.0, and accessibility-related contract metadata.
Initial standards portfolio: xAPI 2.0, cmi5 Quartz compatibility, LTI 1.3, QTI 3, CASE 1.1, Open Badges 3.0, CLR 2.0, CEFR language-assessment metadata, and accessibility-related contract metadata.

It contains no product runtime state and no application database.

## Profiles

- `profiles/cwl_cefr_language_assessment/v1` defines a rights-safe CEFR assessment blueprint, task metadata, immutable domain-result snapshot, and executable fixtures. It stores references rather than official descriptor prose, task content, responses, media, or numerical scoring payloads.

## Branching

Product work targets `develop`; release promotion to `main` occurs only after exact-head review and required checks.
Expand Down
16 changes: 12 additions & 4 deletions docs/ARCHITECTURE.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,12 +2,20 @@

This repository owns versioned learning interoperability contracts and no application runtime state.

Primary families: xAPI 2.0, cmi5 Quartz compatibility, LTI 1.3, QTI 3, CASE 1.1, Open Badges 3.0, and CLR 2.0.
Primary families: xAPI 2.0, cmi5 Quartz compatibility, LTI 1.3, QTI 3, CASE 1.1, Open Badges 3.0, CLR 2.0, and rights-safe CEFR language-assessment metadata.

Authority boundaries:
- Learning Management Platform: offerings, enrollment, progression, completion policy.
- Learning Content Studio: authoring state and immutable releases.
- Learning Management Platform: offerings, enrollment, progression, placement/completion policy, and credential references.
- Learning Content Studio: authoring state, tasks, rubrics, media, rights, and immutable releases.
- Learning Record Store: xAPI statements and document resources.
- Psychometrics Commons: assessment sessions, responses, and score snapshots.
- Psychometrics Commons: assessment blueprints/instrument publication, sessions, responses, and immutable result snapshots.
- fast-mlsirm: psychometric estimation, many-facet calibration, standard-setting/cut-score evidence, linking, DIF, uncertainty, and recovery.
- Semantic Data Portal: rights-aware descriptor/RLD/competency catalog references where adopted.
- TEPP: longitudinal, temporal, multilevel, and multiple-membership language-development analysis.
- contextual-orchestrator: bounded AI-rater orchestration; its observations are evidence, not score authority.

The CEFR profile stores immutable references and bounded result-envelope summaries only. Allowed summaries include domain `level_probabilities`, `credible_level_set`, `standard_error`, measurement status, descriptor-coverage references, claim status, limitations, and evidence identities. The profile never stores official descriptor prose, authored task content, raw responses, audio, provider payloads, PII, item/person/rater parameter arrays, likelihood traces, raw scores, response-level calculations, or other scoring-engine internals.

A reported overall result is not self-authorizing. The executable validator resolves the exact immutable assessment blueprint and accepts an overall result only when that blueprint permits overall reporting, every required domain is measured, and the result cites the blueprint's exact reporting policy.

Consumers integrate through versioned contracts; cross-repository database access is not part of the architecture.
61 changes: 61 additions & 0 deletions docs/adr/0002-cefr-language-assessment-profile.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,61 @@
# ADR 0002: Rights-safe CEFR language-assessment profile

## Status

Accepted for the active stacked PR; not protected-`develop` truth until merged.

Approved by: ContextualWisdomLab repository owner
Approval date: 2026-08-27

## Context

CWL can compose learning management, content authoring, assessment hosting, psychometric estimation, model orchestration and longitudinal analysis. It lacks a shared contract for reporting a domain-level language-proficiency profile in relation to the Common European Framework of Reference for Languages (CEFR).

A naive contract would create material risks:

- copying official descriptor prose or translations into a public repository;
- treating CEFR as a single equally spaced numerical score;
- averaging incomplete skill results into one overall label;
- allowing a result document to self-authorize an overall label;
- confusing alignment with empirical linking or certification;
- moving scoring arithmetic or result authority into an interoperability repository;
- leaking raw responses, audio, task content, provider payloads or PII.

## Decision

Create `profiles/cwl_cefr_language_assessment/v1` as a metadata-only, versioned interoperability profile.

The profile:

1. references the CEFR Companion Volume, an exact target-language RLD/profile source and immutable revision/snapshot, descriptors, task/rubric releases, scoring profile, cut-score revision, standard-setting study and validation evidence by opaque identity;
2. supports Pre-A1, A1, A2, A2+, B1, B1+, B2, B2+, C1 and C2 while keeping domain results explicit;
3. models reception, production, interaction and mediation through typed activity-domain codes;
4. requires level probabilities, uncertainty, credible-level sets and descriptor-coverage references for every measured domain;
5. structurally validates every document with its committed JSON Schema Draft 2020-12 schema before semantic checks;
6. resolves the immutable assessment blueprint before accepting a result and prohibits an overall level unless that blueprint authorizes overall reporting, every required domain is measured and the exact blueprint reporting policy is cited;
7. requires standard-setting and linking-validation references before `cefr_linked` claims;
8. requires an exact certification authority and certification policy, in addition to linked evidence, before a `certification_decision` claim;
9. requires standard-setting evidence for high-stakes or certification blueprints;
10. rejects copied descriptor/task/response payload fields in fixtures and the quality gate;
11. leaves numerical scoring in fast-mlsirm, instrument/result authority in Psychometrics Commons, content authority in Learning Content Studio, and learner actions in Learning Management Platform.

## Consequences

### Positive

- Consumers can exchange an auditable proficiency profile without cross-service SQL or payload duplication.
- Domain-level uncertainty and incomplete evidence remain visible.
- A result cannot escalate its own overall-reporting or certification authority.
- Rights and claim-state boundaries fail closed.
- Future target languages can use different exact RLD/profile authorities without changing the common contract.

### Negative

- The profile cannot prove that an assessment is linked to the CEFR.
- Consumers must resolve referenced artifacts through their authorized owning systems.
- The CI gate adds a small hash-locked Draft 2020-12 validator dependency set.
- Generated SDKs and cross-repository consumer conformance remain a later release slice.

## Reversal conditions

A new major profile version is required if the CEFR framework representation, level system, domain taxonomy, claim semantics, blueprint-authorization semantics or result envelope changes incompatibly. New optional evidence references may be added compatibly only with conformance fixtures and consumer contract tests.
100 changes: 100 additions & 0 deletions docs/doctoring/CEFR_LANGUAGE_ASSESSMENT.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,100 @@
# CEFR language-assessment research and standards basis

## Product interpretation

The CEFR is a non-prescriptive reference framework and common meta-language for curriculum, materials and assessment. It describes six common levels and three plus levels, and the Companion Volume adds Pre-A1 and expands descriptors for mediation, online interaction, plurilingual/pluricultural competence, phonology and signing.

This repository does not reproduce the descriptor corpus. A contract stores only an immutable `descriptor_reference`, exact framework authority, exact target-language profile/RLD source identity, and an immutable publisher revision or dated source snapshot.

## Assessment-development boundary

The 2026 revised *Manual for Language Test Development and Examining—For use with the CEFR* is the Council of Europe/ALTE test-development baseline adopted by this profile. The separate 2009 *Manual for Relating Language Examinations to the CEFR* and its explicitly identified supporting grids provide transparent, cumulative procedures for supporting a linking claim. The Council of Europe does not verify or validate an examination provider's claimed link.

The product claim states therefore have different evidence gates:

```text
experimental
- research-only contract evidence; no operational CEFR interpretation claim

cefr_aligned
- the blueprint references CEFR constructs, descriptors, and a target-language profile;
- no empirical examination-linking claim is made

cefr_linked
- exact standard-setting evidence and empirical linking/classification-validation
evidence are pinned in addition to the aligned blueprint

certification_decision
- the result is already CEFR-linked;
- an exact certification authority and certification policy are pinned;
- the governed decision and publication requirements of that authority are met
```

A label cannot advance by editing narrative copy. A `cefr_aligned` result does not become `cefr_linked` without standard-setting and empirical validation, and a linked result does not become a certification decision without a governed decision authority and policy.

## Language-specific content

Reference Level Descriptions are language-specific inventories of linguistic forms and communicative content. CEFR profile v1 requires every target-language blueprint to pin both:

```text
language_reference_level_description_reference
language_reference_level_description_revision
```

The revision must be an immutable publisher revision, edition, digest-bound snapshot, or dated registry snapshot. Mutable aliases such as `latest` or `current` are rejected. Version 1 has no no-RLD exception; introducing one would require an explicit contract change, new claim semantics, and positive/negative conformance fixtures.

The public Council of Europe English RLD registry identifies the English Profile programme but exposes no single immutable edition identifier. The English fixtures therefore pin the authority as `coe_rld_registry_english_profile` and the dated registry snapshot `english_profile_registry_snapshot_2026_08_27`. That snapshot is a source-discovery identity, not a claim that all English Profile research has become one fixed operational RLD edition.

For content-specification research, the exact published forerunner retained in doctoring is North, Ortega, and Sheehan's *A Core Inventory for General English* (2010; ISBN 978-0-86355-653-1). It is not silently substituted for the English Profile registry identity in the contract.

## Measurement governance

The 2014 *Standards for Educational and Psychological Testing* governs intended interpretation and use, validity evidence, reliability/precision, fairness, administration, reporting and the rights of test takers. CEFR alignment alone does not establish these properties.

The downstream scientific owner must evaluate, as applicable:

- multidimensional structure and local dependence;
- rater, task, criterion, occasion and scoring-engine facets;
- standard-setting and cut-score uncertainty;
- classification consistency and decision error;
- form linking and anchor stability;
- language, population, mode and accommodation DIF/invariance;
- human/AI rater drift and adjudication;
- true-parameter and classification recovery;
- longitudinal comparability before change interpretation.

## Fixed examination-linking sources

This profile does not use mutable labels such as “current manual and supplements.” It pins these exact sources where applicable:

- Council of Europe (2009), *Relating Language Examinations to the CEFR: A Manual*;
- Council of Europe (2012), *Highlights from the Manual for Relating Language Examinations to the CEFR* (ISBN 978-92-871-7169-6);
- Council of Europe, *CEFR Speaking Content Analysis Grid*, February 2014;
- Council of Europe, *CEFR Writing Content Analysis Grid*, version 3.1.

Additional Council materials require their own exact title, version/date, source, digest or snapshot identity before they enter a blueprint or validation record.

## APA 7th references

American Educational Research Association, American Psychological Association, & National Council on Measurement in Education. (2014). *Standards for educational and psychological testing*. American Educational Research Association.

Association of Language Testers in Europe. (2026). *New revised manual for language test development and examining: For use with the CEFR*. Council of Europe.

Council of Europe. (2009). *Relating language examinations to the Common European Framework of Reference for Languages: Learning, teaching, assessment (CEFR): A manual*. Council of Europe.

Council of Europe. (2012). *Highlights from the manual for relating language examinations to the CEFR*. Council of Europe. ISBN 978-92-871-7169-6.

Council of Europe. (2020). *Common European Framework of Reference for Languages: Learning, teaching, assessment—Companion volume*. Council of Europe Publishing.

North, B., Ortega, A., & Sheehan, S. (2010). *A core inventory for general English*. British Council and EAQUALS. ISBN 978-0-86355-653-1.

## Official sources

- https://www.coe.int/en/web/common-european-framework-reference-languages/cefr-companion-volume-and-its-language-versions
- https://www.coe.int/en/web/common-european-framework-reference-languages/introduction-and-context
- https://www.coe.int/en/web/common-european-framework-reference-languages/cefr-descriptors
- https://www.coe.int/en/web/common-european-framework-reference-languages/cefr-reference-level-descriptions-language-by-language-components-and-forerunners
- https://www.coe.int/en/web/common-european-framework-reference-languages/english
- https://www.coe.int/en/web/education/-/manual-for-language-test-development-and-examining-1
- https://www.coe.int/en/web/common-european-framework-reference-languages/relating-examinations-to-the-cefr
- https://www.eaquals.org/resources/a-core-inventory-for-general-english/
Loading