Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add coverage statistics #5907

Open
wants to merge 62 commits into
base: main
Choose a base branch
from

Conversation

mildm8nnered
Copy link
Collaborator

@mildm8nnered mildm8nnered commented Dec 22, 2024

WIP - Addresses #5906

Adds coverage reporting to SwiftLint, via a --report-coverage command line option, and a report_coverage configuration file setting.

% swiftlint --quiet --report-coverage
Enabled rules coverage: 0.995
    All rules coverage: 0.854

"Enabled rules coverage" is helpful for monitoring whether contributors are simply disabling SwiftLint for large sections of code or entire files (although blanket_disable_command, if not disabled, should help with that already).

"All rules coverage" is more useful - low values here indicate that a lot of rules are disabled (or suppressed), and that you could potentially be getting a lot more value out of SwiftLint.

Related changes - we now cache calls to file.regions(), as we'll call this at least twice if coverage is enabled.

From https://github.com/mildm8nnered/SwiftLint/blob/mildm8nnered-add-coverage-statistics/Source/SwiftLintFramework/Coverage.swift:

Coverage is defined as the sum of the number of rules applied to each line of input, divided by the product of the
number of input lines and the total number of rules.

If all rules are applied to every input line, then coverage would be 1 (or 100%). If half the rules are applied
to all input lines, or if all the rules are applied to half of the input lines, then coverage would be 0.5, and
if no rules are enabled, or there is no input, then coverage would be zero.

No distinction is made between actual lines of Swift source, versus blank lines or comments, as SwiftLint may
apply rules to those as well. Coverage is only calculated over input files, so if you exclude files in your
configuration, they will be ignored. Empty input files, or files containing a single empty line will be ignored, as
SwiftLint ignores these files automatically.

"All rules" can be defined either as all enabled rules, or all available rules, enabled or not, resulting
in two different coverage metrics:

  • "Enabled rules coverage" measures how frequently enabled rules are being disabled by swiftlint:disable
    commands.
  • "All rules coverage" measures how many of all possible rules are actually being applied.

Typically, enabled rules coverage will be close to 1, as swiftlint:disable is used sparingly. All rules
coverage will generally be much lower, as some rules are contradictory, and many rules are optin. With no opt-in
rules enabled, all rules coverage will be about 0.4, rising to 0.8 or more if many opt-in rules are enabled.

When calculating all rules coverage swiftlint:disable commands are still accounted for, but only for enabled
rules.

Coverage figures will be different for linting and analyzing as these use different sets of rules.

The number of enabled rules is determined on a per-file basis, so child and local configurations will be accounted
for.

Custom rules, if enabled and defined, will be counted as first class rules for both enabled and all rules coverage.
If not enabled, custom_rules will be counted as a single rule, even if a configuration exists for it.

When calculating enabled rules coverage, the custom rules in the configuration for each input file (e.g. including
child configurations) will be taken into account. When calculating all rules coverage, only the main configurations
custom rules settings will be used.

@SwiftLintBot
Copy link

SwiftLintBot commented Dec 22, 2024

1 Warning
⚠️ Big PR
17 Messages
📖 Linting Aerial with this PR took 0.96s vs 0.94s on main (2% slower)
📖 Linting Alamofire with this PR took 1.28s vs 1.28s on main (0% slower)
📖 Linting Brave with this PR took 6.68s vs 6.66s on main (0% slower)
📖 Linting DuckDuckGo with this PR took 5.56s vs 5.56s on main (0% slower)
📖 Linting Firefox with this PR took 11.08s vs 11.12s on main (0% faster)
📖 Linting Kickstarter with this PR took 10.4s vs 10.42s on main (0% faster)
📖 Linting Moya with this PR took 0.54s vs 0.54s on main (0% slower)
📖 Linting NetNewsWire with this PR took 2.96s vs 2.96s on main (0% slower)
📖 Linting Nimble with this PR took 0.79s vs 0.8s on main (1% faster)
📖 Linting PocketCasts with this PR took 8.84s vs 8.79s on main (0% slower)
📖 Linting Quick with this PR took 0.45s vs 0.45s on main (0% slower)
📖 Linting Realm with this PR took 4.56s vs 4.54s on main (0% slower)
📖 Linting Sourcery with this PR took 2.37s vs 2.34s on main (1% slower)
📖 Linting Swift with this PR took 4.61s vs 4.59s on main (0% slower)
📖 Linting VLC with this PR took 1.27s vs 1.28s on main (0% faster)
📖 Linting Wire with this PR took 18.44s vs 18.48s on main (0% faster)
📖 Linting WordPress with this PR took 11.58s vs 11.62s on main (0% faster)

Generated by 🚫 Danger

@SimplyDanny SimplyDanny linked an issue Dec 22, 2024 that may be closed by this pull request
2 tasks
@mildm8nnered mildm8nnered force-pushed the mildm8nnered-add-coverage-statistics branch 4 times, most recently from 761d4f4 to 3274fa8 Compare December 27, 2024 14:55
@mildm8nnered mildm8nnered marked this pull request as ready for review December 27, 2024 20:30
@mildm8nnered mildm8nnered force-pushed the mildm8nnered-add-coverage-statistics branch from 17a31e1 to 61d61b5 Compare December 28, 2024 13:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add coverage statistics
2 participants