Skip to content

ci: add coverage report and summary to tests workflow (closes #3) - #6

Merged
xingyaoww merged 6 commits into
mainfrom
openhands/fix-issue-3-coverage
Aug 24, 2025
Merged

xingyaoww merged 6 commits into
mainfrom
openhands/fix-issue-3-coverage

Conversation

@xingyaoww

Copy link
Copy Markdown
Contributor

Summary

  • Add pytest-cov to dev dependencies
  • Update CI to run pytest with coverage and produce Cobertura XML (coverage.xml)
  • Add Code Coverage Summary action to generate a markdown/text summary and badge
  • Post the coverage summary as a sticky PR comment on pull requests
  • Upload coverage artifacts for inspection

Changes

  • pyproject.toml: add pytest-cov in dev dependencies
  • .github/workflows/tests.yml:
    • run pytest with --cov=openhands --cov-report=xml:coverage.xml --cov-report=term-missing
    • add irongut/CodeCoverageSummary@v1.3.0 step with badge output
    • add marocchino/sticky-pull-request-comment@v2 to comment results on PRs
    • upload coverage.xml and code-coverage-results.md as artifacts
    • trigger on both push and pull_request; set minimal permissions

Testing

  • Local: executed pytest with coverage to verify coverage.xml generation (tests currently fail in local env but coverage report generation is validated). CI workflow will generate and publish the summary regardless of test outcomes due to if: always() on reporting steps.

Docs

  • Not applicable.

Checklist

  • CI updated to produce a coverage report
  • Coverage summary included using Marketplace action
  • PR references the issue and will close it when merged

Closes #3

@xingyaoww can click here to continue refining the PR

… workflow\n\n- Add pytest-cov to dev dependencies\n- Generate Cobertura XML via pytest --cov-report xml\n- Post markdown summary and badge using irongut/CodeCoverageSummary\n- Comment coverage results on PRs\n\nCloses #3\n\nCo-authored-by: openhands <openhands@all-hands.dev>
openhands-agent and others added 2 commits August 24, 2025 20:36
…erate coverage.svg badge from coverage.xml in CI\n- Commit badge on push builds; upload badge artifact on PRs\n- Add coverage badge to README\n\nCo-authored-by: openhands <openhands@all-hands.dev>
Comment thread .github/workflows/tests.yml Outdated
@github-actions

Copy link
Copy Markdown
Contributor

Code Coverage

Package Line Rate Health
. 100% ✔
core 86% ✔
core.runtime 86% ✔
core.runtime.tools 100% ✔
core.runtime.tools.execute_bash 100% ✔
core.runtime.tools.str_replace_editor 89% ✔
core.runtime.tools.str_replace_editor.utils 96% ✔
Summary 91% (667 / 736) ✔

@openhands-ai

openhands-ai Bot commented Aug 24, 2025

Copy link
Copy Markdown

Looks like there are a few issues preventing this PR from being merged!

  • GitHub Actions are failing:
    • Run tests

If you'd like me to help, just leave a comment, like

@OpenHands please fix the failing actions on PR #6 at branch `openhands/fix-issue-3-coverage`

Feel free to include any additional details that might help me get this PR into a better state.

You can manage your notification settings

@xingyaoww
xingyaoww marked this pull request as ready for review August 24, 2025 21:00
@xingyaoww
xingyaoww merged commit 36531f2 into main Aug 24, 2025
@xingyaoww
xingyaoww deleted the openhands/fix-issue-3-coverage branch August 24, 2025 21:00
@openhands-ai openhands-ai Bot mentioned this pull request Feb 10, 2026
3 of 5 tasks
xli04 added a commit to xli04/software-agent-sdk that referenced this pull request May 15, 2026
Addresses review OpenHands#6 (test plan item #5) on OpenHands#2911.

Five tests verify the analyzer's output drives ``ConfirmRisky``
correctly across all risk levels. The Conversation state machine
transitions into ``WAITING_FOR_CONFIRMATION`` whenever
``ConfirmRisky.should_confirm(risk)`` returns True, so this is the
operationally meaningful contract:

- HIGH from analyzer -> ConfirmRisky pauses
- MEDIUM from analyzer -> default threshold (HIGH) does NOT pause
- LOW from analyzer -> does NOT pause
- UNKNOWN from analyzer + confirm_unknown=True (default) -> pauses
  (this is what preserves the conservative posture after the
  parse-failure path was switched from HIGH to UNKNOWN in
  review #1)
- UNKNOWN from analyzer + confirm_unknown=False -> does NOT pause

All five tests mock the guardrail LLM, so no network calls.
StressTestor pushed a commit to StressTestor/software-agent-sdk that referenced this pull request Jun 1, 2026
…ds#3) (OpenHands#6)

Co-authored-by: openhands <openhands@all-hands.dev>
Co-authored-by: github-actions <41898282+github-actions[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add coverage report for tests

2 participants