Skip to content

ci: Fix slash command result detection to properly report success/failure - #71062

Merged
Aaron ("AJ") Steers (aaronsteers) merged 1 commit into
masterfrom
devin/1767136944-fix-slash-command-result-detection
Dec 30, 2025
Merged

ci: Fix slash command result detection to properly report success/failure#71062
Aaron ("AJ") Steers (aaronsteers) merged 1 commit into
masterfrom
devin/1767136944-fix-slash-command-result-detection

Conversation

@aaronsteers

Copy link
Copy Markdown
Member

What

Fixes the /run-connector-tests slash command always reporting "job completed successfully" even when tests fail. See this comment for an example where the command reported success despite test failures.

How

The workflow output was using jobs.connector-ci-checks-summary.result which returns the job's completion status (success if the job runs without errors), not the step's output that evaluates whether the actual tests passed or failed.

Fixed by:

  1. Adding an outputs section to the connector-ci-checks-summary job to expose the evaluate-status step output
  2. Updating the workflow output to reference jobs.connector-ci-checks-summary.outputs.result instead of jobs.connector-ci-checks-summary.result

Review guide

  1. .github/workflows/connector-ci-checks.yml - Two small changes:
    • Line 34: Changed workflow output reference from .result to .outputs.result
    • Lines 473-474: Added job outputs section to expose the step output

User Impact

Slash commands like /run-connector-tests will now correctly report failure when tests fail, instead of always showing success.

Can this PR be safely reverted and rolled back?

  • YES 💚

Link to Devin run: https://app.devin.ai/sessions/38464f16eb5f495cb72a77841632c875
Requested by: AJ Steers (Aaron ("AJ") Steers (@aaronsteers))

…lure

The workflow output was using jobs.connector-ci-checks-summary.result
(the job's completion status) instead of the step's output. This caused
the slash command to always report success even when tests failed.

Fixed by:
1. Adding outputs section to connector-ci-checks-summary job to expose
   the evaluate-status step output
2. Updating workflow output to reference jobs.connector-ci-checks-summary.outputs.result

Co-Authored-By: AJ Steers <aj@airbyte.io>
@devin-ai-integration

Copy link
Copy Markdown
Contributor
Original prompt from AJ Steers
@Devin - This slash command output is misleading because it says "job completed successfully. See log for details." when the job result is failure. <https://github.com/airbytehq/airbyte/pull/70221#issuecomment-3599287191>

Use a workflow dispatch github action from the marketplace which is capable of waiting for the workflow to complete and then detecting result of the workflow in order to properly indicate in the slash command appended text whether the job succeeded or failed.
Thread URL: https://airbytehq-team.slack.com/archives/D089P0UPVT4/p1767136806526669

@devin-ai-integration

Copy link
Copy Markdown
Contributor

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

  • Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
  • Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

  • Disable automatic comment and CI monitoring

@github-actions

Copy link
Copy Markdown
Contributor

👋 Greetings, Airbyte Team Member!

Here are some helpful tips and reminders for your convenience.

Helpful Resources

PR Slash Commands

Airbyte Maintainers (that's you!) can execute the following slash commands on your PR:

  • /format-fix - Fixes most formatting issues.
  • /bump-version - Bumps connector versions.
    • You can specify a custom changelog by passing changelog. Example: /bump-version changelog="My cool update"
    • Leaving the changelog arg blank will auto-populate the changelog from the PR title.
  • /bump-progressive-rollout-version - Bumps connector version with an RC suffix for progressive rollouts.
    • Creates a release candidate version (e.g., 2.16.10-rc.1) with enableProgressiveRollout: true
    • Example: /bump-progressive-rollout-version changelog="Add new feature for progressive rollout"
  • /run-cat-tests - Runs legacy CAT tests (Connector Acceptance Tests)
  • /run-regression-tests - Runs regression tests for the modified connector(s).
  • /build-connector-images - Builds and publishes a pre-release docker image for the modified connector(s).
  • /publish-connectors-prerelease - Publishes pre-release connector builds (tagged as {version}-preview.{git-sha}) for all modified connectors in the PR.
  • Connector release lifecycle (AI-powered):
    • /ai-prove-fix - Runs prerelease readiness checks, including testing against customer connections.
    • /ai-canary-prerelease - Rolls out prerelease to 5-10 connections for canary testing.
    • /ai-release-watch - Monitors rollout post-release and tracks sync success rates.
  • JVM connectors:
    • /update-connector-cdk-version connector=<CONNECTOR_NAME> - Updates the specified connector to the latest CDK version.
      Example: /update-connector-cdk-version connector=destination-bigquery
    • /bump-bulk-cdk-version bump=patch changelog='foo' - Bump the Bulk CDK's version. bump can be major/minor/patch.
  • Python connectors:
    • /poe connector source-example lock - Run the Poe lock task on the source-example connector, committing the results back to the branch.
    • /poe source example lock - Alias for /poe connector source-example lock.
    • /poe source example use-cdk-branch my/branch - Pin the source-example CDK reference to the branch name specified.
    • /poe source example use-cdk-latest - Update the source-example CDK dependency to the latest available version.

📝 Edit this welcome message.

@aaronsteers
Aaron ("AJ") Steers (aaronsteers) merged commit 9c91a09 into master Dec 30, 2025
37 checks passed
@aaronsteers
Aaron ("AJ") Steers (aaronsteers) deleted the devin/1767136944-fix-slash-command-result-detection branch December 30, 2025 23:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant