Skip to content

test(linter): Add JSON Schema validation tests for oxlintrc configuration#16822

Closed
Copilot wants to merge 25 commits intomainfrom
copilot/add-json-schema-tests
Closed

test(linter): Add JSON Schema validation tests for oxlintrc configuration#16822
Copilot wants to merge 25 commits intomainfrom
copilot/add-json-schema-tests

Conversation

Copy link
Contributor

Copilot AI commented Dec 14, 2025

  • Add jsonschema crate as dev-dependency to oxc_linter/Cargo.toml
  • Create test fixture directory for schema validation tests
  • Create valid test configuration files
    • Basic valid config with plugins
    • Config with valid categories
    • Full featured valid config (from issue) with jsPlugins
    • Environment config
    • Rules config
    • Ignore patterns config with glob patterns
    • Empty config
  • Create invalid test configuration files
    • Config with invalid plugin name
    • Config with invalid category name
    • Config with invalid severity
    • Config with wrong type for plugins
    • Config with wrong type for env
  • Create schema validation test file
    • Test that loads the generated schema
    • Tests valid configs pass validation
    • Tests invalid configs fail validation
    • Test that schema is valid JSON Schema Draft 7
  • Run and verify tests work correctly
  • Clean up accidentally committed files
  • Format code per project standards
  • Address code review feedback
  • Fix all clippy warnings

💬 We'd love your input! Share your thoughts on Copilot coding agent in our 2 minute survey.

Copilot AI changed the title [WIP] Add tests for JSON Schema validation checks Add JSON Schema validation tests for oxlintrc configuration Dec 14, 2025
Copilot AI requested a review from connorshea December 14, 2025 05:46
@connorshea connorshea force-pushed the copilot/add-json-schema-tests branch from 1916266 to 7d10179 Compare December 14, 2025 05:46
@github-actions github-actions bot added the A-linter Area - Linter label Dec 14, 2025
@connorshea connorshea changed the title Add JSON Schema validation tests for oxlintrc configuration test(linter): Add JSON Schema validation tests for oxlintrc configuration Dec 14, 2025
@github-actions github-actions bot added the C-test Category - Testing. Code is missing test cases, or a PR is adding them label Dec 14, 2025
@codspeed-hq
Copy link

codspeed-hq bot commented Dec 14, 2025

CodSpeed Performance Report

Merging #16822 will not alter performance

Comparing copilot/add-json-schema-tests (decfb4e) with main (bc7aae7)1

Summary

✅ 42 untouched
⏩ 3 skipped2

Footnotes

  1. No successful run was found on main (7005873) during the generation of this report, so bc7aae7 was used instead as the comparison base. There might be some changes unrelated to this pull request in this report.

  2. 3 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports.

Copilot AI requested a review from connorshea December 14, 2025 05:59
@connorshea connorshea marked this pull request as ready for review December 14, 2025 06:41
@connorshea connorshea requested a review from camc314 as a code owner December 14, 2025 06:41
Copilot AI review requested due to automatic review settings December 14, 2025 06:41
Copy link
Member

@connorshea connorshea left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks good to me generally, but I also did part of the work, so can't really approve :)

Copy link

@charliecreates charliecreates bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The test suite adds valuable schema regression coverage, but it currently relies on npm/oxlint/configuration_schema.json existing at test time, which can make cargo test non-hermetic and flaky in CI or clean checkouts. The insta snapshots also capture unstable, dependency-version-sensitive jsonschema error strings, which may cause noisy churn. Consider adding a runtime fallback (or explicit precondition) for schema loading and snapshot a more stable, sorted representation of validation errors.

Additional notes (2)
  • Readability | crates/oxc_linter/tests/schema_validation_test.rs:14-32
    get_project_root().unwrap() is called multiple times. In tests this is fine, but it also means failures will be less informative and you’ll do redundant filesystem work. Consolidating to a single project_root() helper improves readability and makes it easier to change behavior (e.g., fallback logic) in one place.

  • Maintainability | crates/oxc_linter/tests/schema_validation_test.rs:111-126
    snap_name currently includes the .json extension (e.g. invalid_env_wrong_type.json_errors). That’s not wrong, but it’s a bit noisy and makes renames (like switching fixtures to .jsonc in the future) cause snapshot churn. Also, snapshot naming depends on the filename string rather than the path, which becomes awkward if you later enumerate fixtures from the directory.

Summary of changes

What changed

✅ Added JSON Schema validation coverage for .oxlintrc

  • Introduced a new Rust test suite at crates/oxc_linter/tests/schema_validation_test.rs that:
    • Loads the JSON Schema from npm/oxlint/configuration_schema.json.
    • Validates a curated set of valid and invalid config fixtures.
    • Snapshots invalid validation output via insta for stable regression tracking.
    • Verifies the schema declares Draft 7 via the $schema field and is compilable.

🧪 Added fixtures + snapshots

  • Added fixture files under:
    • crates/oxc_linter/tests/fixtures/schema_validation/valid/*.json
    • crates/oxc_linter/tests/fixtures/schema_validation/invalid/*.json
  • Added insta snapshot files under crates/oxc_linter/tests/snapshots/* capturing failing validation results.

📦 Dependency + repo hygiene updates

  • Added jsonschema = "0.37" as a dev-dependency in crates/oxc_linter/Cargo.toml.
  • Updated .gitignore to ignore *.rlib.

Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR adds comprehensive JSON Schema validation tests for the oxlintrc configuration file to prevent regressions in the schema generation process. The tests validate both valid and invalid configuration files against the generated schema, ensuring that the schema correctly accepts proper configs and rejects improper ones.

Key changes:

  • Added jsonschema crate as a dev-dependency for schema validation testing
  • Created 11 valid test fixtures covering various configuration scenarios (plugins, categories, environments, rules, overrides, etc.)
  • Created 7 invalid test fixtures testing common misconfiguration errors
  • Implemented 3 test functions with snapshot testing for validation errors
  • Updated .gitignore to exclude *.rlib build artifacts

Reviewed changes

Copilot reviewed 27 out of 29 changed files in this pull request and generated no comments.

Show a summary per file
File Description
crates/oxc_linter/Cargo.toml Adds jsonschema dev-dependency for validation testing
crates/oxc_linter/tests/schema_validation_test.rs Implements schema validation tests with fixture loading and snapshot testing
crates/oxc_linter/tests/fixtures/schema_validation/valid/*.json Provides 11 valid configuration test cases covering different features
crates/oxc_linter/tests/fixtures/schema_validation/invalid/*.json Provides 7 invalid configuration test cases for error validation
crates/oxc_linter/tests/snapshots/invalid_*.snap Snapshot files capturing expected validation error messages
.gitignore Adds *.rlib to ignore Rust library build artifacts
Cargo.lock Updates lock file with jsonschema and its transitive dependencies

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot AI requested a review from camc314 December 14, 2025 13:18
graphite-app bot pushed a commit that referenced this pull request Dec 15, 2025
…rride struct. (#16870)

Ensure that extra fields are not allowed inside overrides, to make it clear to users when they have added an invalid field.

Pulled out of #16822 since I think it's worth including regardless of that PR.
@connorshea connorshea self-requested a review December 21, 2025 21:43
@connorshea connorshea force-pushed the copilot/add-json-schema-tests branch from bd5614b to 36fc2f4 Compare December 31, 2025 00:08
Copilot AI and others added 25 commits January 1, 2026 19:29
Co-authored-by: connorshea <2977353+connorshea@users.noreply.github.com>
Co-authored-by: connorshea <2977353+connorshea@users.noreply.github.com>
Co-authored-by: connorshea <2977353+connorshea@users.noreply.github.com>
Co-authored-by: connorshea <2977353+connorshea@users.noreply.github.com>
- Remove *.a and *.o from .gitignore (not needed)
- Add more patterns to ignore_patterns.json test fixture including **/*.ts
- Add jsPlugins configuration to full_config.json test fixture
- Fix clippy uninlined_format_args warnings in test file

Co-authored-by: connorshea <2977353+connorshea@users.noreply.github.com>
Ensure that the snapshots exist and can display the validation results.
Add test to validate that this fails when there is an unknown field in an override config.
As requested, removing the *.rlib pattern from .gitignore.

Co-authored-by: camc314 <18101008+camc314@users.noreply.github.com>
@connorshea connorshea force-pushed the copilot/add-json-schema-tests branch from 36fc2f4 to decfb4e Compare January 2, 2026 02:29
@connorshea
Copy link
Member

I'm going to close this for now, I think it's worth exploring later though

@connorshea connorshea closed this Jan 23, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

A-linter Area - Linter C-test Category - Testing. Code is missing test cases, or a PR is adding them

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants