IBX-9251: Run browser tests on different Ubuntu versions #54
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This was referenced Nov 26, 2024
micszo
changed the title
|
Is there any way to force a certain ubuntu version after a restart/fail ? I fear that here we trade increased coverage for decreased reliability (and im not really a fan of that :( ). |
micszo
force-pushed
the
ibx-9251
branch
4 times, most recently
from
247bab2 to
af8ab41
Compare
|
Added summary in description. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description:
Run on Ubuntu 24.04, 22.04 and 20.04. Chosen randomly for given setup - each job in a setup gets the same OS version.
Ref. https://doc.ibexa.co/en/latest/getting_started/requirements/#operating-system
Pros of approach:
Cons:
For QA:
Q: Can runners in a group be used? (initial idea)
A: Runner groups are meant to be used with self-hosted runners only, not for Github-hosted ones.
Open question: Since Docker is used for different versions of services, what exactly is the impact of different Ubuntu editions?
Summary:
Result of restarting a job:
Using artifacts with a file holding chosen Ubuntu version is not sufficient. Artifacts are not shared between workflow runs which turned out includes restarts.
A possible solution could be to combine artifacts with an external cloud storage. Sounds like overengineering.
The cost in complexity to join increased coverage with good reliability without affecting API limits is too high. If at all possible for all scenarios. For now reliability of tests is key. Topic can be revisited if new GHA mechanisms surface.