ci: add autoqa reliability workflow for windows

[Minh141120]

This pull request introduces a new GitHub Actions workflow for manually running AutoQA reliability tests across multiple platforms. The workflow provides flexible configuration options for testing different app sources and build types, and automates the installation, testing, and artifact collection processes.

New AutoQA Reliability Workflow:

• Adds a new workflow file .github/workflows/autoqa-reliability.yml that can be triggered manually with customizable inputs for source type, app installer locations, build type, reliability phase, test paths, and artifact names.

Windows Reliability Test Job:

• Defines a reliability-windows job that:
  • Supports both URL and local artifact sources for the Windows installer.
  • Cleans up previous installations, prepares the app, installs dependencies, and runs reliability tests using the specified parameters.
  • Uploads screen recordings and test trajectories as artifacts for further analysis.
  • Ensures cleanup is performed after tests, regardless of test outcomes.

---

Important

Adds a new GitHub Actions workflow for manual AutoQA reliability tests on Windows with configurable inputs and artifact uploads.

• New Workflow:
  • Adds .github/workflows/autoqa-reliability.yml for manual AutoQA reliability tests.
  • Configurable inputs for source type, app installer locations, build type, reliability phase, test paths, and artifact names.
• Windows Job:
  • reliability-windows job supports URL and local artifact sources.
  • Cleans previous installations, installs dependencies, and runs tests.
  • Uploads screen recordings and test trajectories as artifacts.
  • Ensures cleanup after tests, regardless of outcome.

^{This description was created by for 76dfe02. You can customize this summary. It will automatically update as commits are pushed.}

[ellipsis-dev]

Caution

Changes requested ❌

Reviewed everything up to 4059d7a in 2 minutes and 38 seconds. Click for details.

• Reviewed 162 lines of code in 1 files
• Skipped 0 files when reviewing.
• Skipped posting 3 draft comments. View those below.
• Modify your settings and rules to customize what types of comments Ellipsis leaves. And don't forget to react with 👍 or 👎 to teach Ellipsis.

1. .github/workflows/autoqa-reliability.yml:80

• Draft comment:
  Wrap conditional expressions in ${{ }} for clarity (e.g. use ${{ inputs.source_type == 'local' }}).
• Reason this comment was not posted:
  Decided after close inspection that this draft comment was likely wrong and/or not actionable: usefulness confidence = 20% vs. threshold = 50% While the suggestion would make the syntax more consistent with other expressions in the file, both forms are valid in GitHub Actions. The current form is perfectly functional and readable. This seems like a minor stylistic preference rather than a significant improvement. The file is new, but this isn't a critical issue that needs addressing. The suggestion would improve consistency with other expressions in the file. GitHub's own examples often use the ${{ }} syntax for if conditions. While consistency is good, this is a minor stylistic issue that doesn't affect functionality or maintainability significantly. Both syntaxes are well-documented and commonly used. This comment should be removed as it suggests a minor stylistic change that isn't critical enough to warrant a code change request.

2. .github/workflows/autoqa-reliability.yml:101

• Draft comment:
  Inconsistency: IS_NIGHTLY is set only in the local branch; ensure it's set for both branches for downstream steps.
• Reason this comment was not posted:
  Decided after close inspection that this draft comment was likely wrong and/or not actionable: usefulness confidence = 20% vs. threshold = 50% The comment raises a valid concern about consistency, but without seeing the contents of windows_download.ps1, I can't be certain there's actually a problem. The script is passed all the nightly parameters it needs. The fact that the install step works suggests the variable is being set properly in both paths. I don't have visibility into windows_download.ps1 to confirm how it handles IS_NIGHTLY. There could be an actual inconsistency. Given that the script is passed the nightly parameter and the install step works, it's more likely that the variable is being handled correctly in both paths. Without strong evidence of an actual problem, and given that the necessary parameters are being passed, this comment appears to be speculative and should be removed.

3. .github/workflows/autoqa-reliability.yml:110

• Draft comment:
  Ensure that repository variables 'JAN_APP_URL' and 'IS_NIGHTLY' are defined, as they're referenced in the download script.
• Reason this comment was not posted:
  Comment did not seem useful. Confidence is useful = 0% <= threshold 50% The comment is asking the author to ensure that certain variables are defined, which is similar to asking them to double-check or verify something. This violates the rule against asking the author to ensure or verify things. Therefore, this comment should be removed.

Workflow ID: wflow_ypFe5alPPh0ms4zN

^{You can customize by changing your verbosity settings, reacting with 👍 or 👎, replying to comments, or adding code review rules.}

[ellipsis-dev]

Important

Looks good to me! 👍

Reviewed 76dfe02 in 1 minute and 34 seconds. Click for details.

• Reviewed 103 lines of code in 1 files
• Skipped 0 files when reviewing.
• Skipped posting 4 draft comments. View those below.
• Modify your settings and rules to customize what types of comments Ellipsis leaves. And don't forget to react with 👍 or 👎 to teach Ellipsis.

1. .github/workflows/autoqa-reliability.yml:6

• Draft comment:
  The 'source_type' input now only allows 'url', but the description still references artifact selection. Please update the description and documentation to reflect that only URL is supported (removing local support).
• Reason this comment was not posted:
  Comment looked like it was already resolved.

2. .github/workflows/autoqa-reliability.yml:45

• Draft comment:
  Artifact name inputs for Windows, Ubuntu, and macOS have been removed. The PR description still mentions local artifact support. Please update the PR documentation if local sources are no longer supported.
• Reason this comment was not posted:
  Comment did not seem useful. Confidence is useful = 0% <= threshold 50% The comment is asking the PR author to update the PR documentation, which is against the rules. It also mentions the removal of artifact names, but it doesn't provide a specific suggestion or ask for confirmation about a specific code change. Therefore, this comment should be removed.

3. .github/workflows/autoqa-reliability.yml:77

• Draft comment:
  The download artifact step for local sources has been removed. Since the PR description indicates support for both URL and local artifact sources, please ensure that the documentation and intended functionality are aligned.
• Reason this comment was not posted:
  Comment did not seem useful. Confidence is useful = 0% <= threshold 50% The comment is asking the PR author to ensure that documentation and functionality are aligned, which is similar to asking them to confirm their intention or ensure behavior. This violates the rules against asking for confirmation or ensuring behavior. The comment does not provide a specific code suggestion or ask for a specific test to be written.

4. .github/workflows/autoqa-reliability.yml:92

• Draft comment:
  The 'Download/Prepare Jan app' step now unconditionally runs the download script without checking for a local source. Confirm that this change (removal of local branch) is intentional and update the PR description accordingly.
• Reason this comment was not posted:
  Comment was not on a location in the diff, so it can't be submitted as a review comment.

Workflow ID: wflow_5tB1msLkx5KYHQLH

^{You can customize by changing your verbosity settings, reacting with 👍 or 👎, replying to comments, or adding code review rules.}