Enhance check_config script with JSON output and fail on warnings

[BenjaminMichaelis]

Proposed change

I run check_config to validate my HA configuration in my CI pipelines.
However, warnings seem to be always treated as non-fatal, missing errors I want to be automatically caught. In addition, there are limited customization options for the validation process.

Type of change

Not totally sure what this falls under since it isn't a full integration, but giving it my best guess :)

• Dependency upgrade
• Bugfix (non-breaking change which fixes an issue)
• New integration (thank you!)
• New feature (which adds functionality to an existing integration)
• Deprecation (breaking change to happen in the future)
• Breaking change (fix/feature causing existing functionality to break)
• Code quality improvements to existing code or addition of tests

Additional information

This pr introduces 2 flags to be able to improve on that without breaking current functionality.

1. --fail-on-warnings - Provides a non-zero exit code if there are any warnings
2. --json - This provides machine readable output of the result so better customizations can be run against structured output

Checklist

• I understand the code I am submitting and can explain how it works.
• The code change is tested and works locally.
• Local tests pass. Your PR cannot be merged unless tests pass
• There is no commented out code in this PR.
• I have followed the development checklist
• I have followed the perfect PR recommendations
• The code has been formatted using Ruff (ruff format homeassistant tests)
• Tests have been added to verify that the new code works.
• Any generated code has been carefully reviewed for correctness and compliance with project standards.

If user exposed functionality or configuration variables are added/changed:

• Documentation added/updated for www.home-assistant.io (in-progress)

If the code communicates with devices, web services, or third-party tools:

• The manifest file has all fields filled out correctly.
  Updated and included derived files by running: python3 -m script.hassfest.
• New or updated dependencies have been added to requirements_all.txt.
  Updated by running python3 -m script.gen_requirements_all.
• For the updated dependencies - a link to the changelog, or at minimum a diff between library versions is added to the PR description.

To help with the load of incoming pull requests:

• I have reviewed two other open pull requests in this repository.

[home-assistant]

Hi @BenjaminMichaelis

It seems you haven't yet signed a CLA. Please do so here.

Once you do that we will be able to review and accept this pull request.

Thanks!

[home-assistant]

Please take a look at the requested changes, and use the Ready for review button when you are done, thanks 👍

Learn more about our pull request process.

[Copilot]

Pull Request Overview

This PR enhances the check_config script by adding two new command-line flags: --json for machine-readable JSON output and --fail-on-warnings to make warnings cause non-zero exit codes. These additions enable better automation and CI/CD integration while maintaining backwards compatibility.

Key changes:

• Added JSON output format that provides structured data including error counts, warnings, and component information
• Added option to treat warnings as failures for stricter validation in automated environments
• Enhanced exit code logic to handle the new warning failure mode

Reviewed Changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

File	Description
homeassistant/scripts/check_config.py	Added argument parsing for new flags, JSON output generation, and modified exit code logic
tests/scripts/test_check_config.py	Comprehensive test suite covering new functionality, flag interactions, and edge cases

Stale

[BenjaminMichaelis]

@kellerza , just wanted to see if you could potentially take a look at this when you get the chance - I notice for this file you are the only one marked as an owner. :)

[kellerza]

Very comprehensive tests!

It seems --secrets is not executed as part of --json. Do you want to include / simply print a warning if --secrets is present during --json output?

[BenjaminMichaelis]

@kellerza - to your secrets comment - good catch - and I don't have a use for secrets as part of the json currently so I printed out a comment in that case. (ac532ec)

but if it is desirable, we could add it with something like this:

        # Include secrets information if requested
        if args.secrets:
            # Build list of missing secrets (referenced but not found)
            missing_secrets = [key for key, val in res["secrets"].items() if val is None]

            # Build list of used secrets (found and used)
            used_secrets = [key for key, val in res["secrets"].items() if val is not None]

            json_object["secrets"] = {
                "secret_files": res["secret_cache"],
                "used_secrets": used_secrets,
                "missing_secrets": missing_secrets,
                "total_secrets": len(res["secrets"]),
                "total_missing": len(missing_secrets)
            }

But also we could wait until that is actually desired and leave as is for now. Thoughts?

[BenjaminMichaelis]

@kellerza need anything else from me to move forward on this?

[kellerza]

        json_object["secrets"] = {
            "secret_files": res["secret_cache"],
            "used_secrets": used_secrets,
            "missing_secrets": missing_secrets,
            "total_secrets": len(res["secrets"]),
            "total_missing": len(missing_secrets)
        }

But also we could wait until that is actually desired and leave as is for now. Thoughts?

That should be best. I don't really want to print out any secret values specifically

[kellerza]

This should be good to merge.

Do you already have a docs PR for the two new command line flags? https://www.home-assistant.io/docs/tools/check_config/

[frenck]

Although I must admit, the use case for this one is slim (not widely adopted probably), the implementation looks solid.

Since it has no impact otherwise, I'm going to add this one in.

Thanks for this fine contribution @BenjaminMichaelis 👍

../Frenck

[kellerza]

Thanks @BenjaminMichaelis !