Skip to content

fix(test): increase Selenium UI test timeouts #2743

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
anusshukla wants to merge 1 commit into from
Closed

fix(test): increase Selenium UI test timeouts #2743

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 6 additions & 2 deletions docker-compose.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,11 +20,15 @@ services:
- ./out/ui/logs:/home/selenium/.local/share/code-server/logs:rw
restart: "no"
healthcheck:
test: ["CMD", "curl", "-f", "http://localhost:4444"]
test:
[
"CMD-SHELL",
"curl -sf http://localhost:4444 && curl -sf http://localhost:8080",
]
interval: 10s
timeout: 5s
retries: 10
start_period: 10s
start_period: 30s
deploy:
resources:
limits:
Expand Down
4 changes: 4 additions & 0 deletions test/ui/fixtures/ui_fixtures.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -89,6 +89,10 @@ def browser_setup(
log.info(
"Waiting for container %s to be healthy: %s", CONTAINER_NAME, count
)
log.info(
"Container healthy, waiting for code-server extension activation..."
)
time.sleep(30)

browser = os.environ.get("BROWSER_TYPE")
options: FirefoxOptions | ChromeOptions
Expand Down
8 changes: 4 additions & 4 deletions test/ui/test_00_commands.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -40,10 +40,10 @@
"""Test terminal functionality."""
driver, _ = browser_setup

ensure_vscode_ready(driver)

Check failure on line 43 in test/ui/test_00_commands.py

View workflow job for this annotation

GitHub Actions / test (linux) 

test_terminal

selenium.common.exceptions.TimeoutException: Message:
vscode_run_command(driver, ">workbench.action.terminal.new")
vscode_run_command(driver, ">workbench.action.terminal.focus")
time.sleep(3) # allow terminal to start before sending input
time.sleep(5) # allow terminal to start before sending input
vscode_run_command(
driver, ">workbench.action.terminal.sendSequence", "adt --version\\n"
)
Expand All @@ -60,7 +60,7 @@

errors = [NoSuchElementException, ElementNotInteractableException]
wait = WebDriverWait(
driver, timeout=10, poll_frequency=0.5, ignored_exceptions=errors
driver, timeout=30, poll_frequency=1, ignored_exceptions=errors
)
wait.until(lambda _: check_output())

Expand All @@ -78,17 +78,17 @@
"""Test the 'Create an empty Ansible playbook' command."""
driver, _ = browser_setup

ensure_vscode_ready(driver)

Check failure on line 81 in test/ui/test_00_commands.py

View workflow job for this annotation

GitHub Actions / test (linux) 

test_create_empty_playbook

selenium.common.exceptions.TimeoutException: Message:

vscode_run_command(driver, ">ansible.create-empty-playbook")

wait_displayed(
driver,
"//div[contains(@class, 'tab') and contains(., 'Untitled')]",
timeout=2,
timeout=10,
)

time.sleep(1)
time.sleep(2)
view_lines = driver.find_elements("xpath", "//div[@class='view-line']")

# Extract text from the lines (checking first 10 lines is sufficient)
Expand Down
4 changes: 2 additions & 2 deletions test/ui/test_01_dev_webviews.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,11 +20,11 @@
"""Test the devfile creation webview elements and workflow."""
driver, _ = browser_setup

ensure_vscode_ready(driver)

Check failure on line 23 in test/ui/test_01_dev_webviews.py

View workflow job for this annotation

GitHub Actions / test (linux) 

test_devfile_webview

selenium.common.exceptions.TimeoutException: Message:

vscode_run_command(driver, ">Ansible: Create a Devfile")

find_element_across_iframes(driver, "//form[@id='devfile-form']", retries=10)
find_element_across_iframes(driver, "//form[@id='devfile-form']", retries=20)

vscode_textfield_interact(driver, "path-url", "~")

Expand Down Expand Up @@ -70,7 +70,7 @@
find_element_across_iframes(
driver,
"//form[@id='devcontainer-form']",
retries=10,
retries=20,
)

vscode_textfield_interact(driver, "path-url", "~")
Expand Down
13 changes: 11 additions & 2 deletions test/ui/utils/ui_utils.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -78,7 +78,16 @@ def ensure_vscode_ready(driver: WebDriver, timeout: int = 120) -> None:
driver.switch_to.default_content()
if "127.0.0.1:8080" not in driver.current_url:
driver.get("http://127.0.0.1:8080")
wait_displayed(driver, "//a[@aria-label='Ansible']", timeout=60)
max_nav_attempts = 3
for attempt in range(max_nav_attempts):
try:
wait_displayed(driver, "//a[@aria-label='Ansible']", timeout=90)
break
except (TimeoutException, TimeOutError):
if attempt == max_nav_attempts - 1:
raise
driver.refresh()
time.sleep(10)
Comment on lines +81 to +90
Copy link
Copy Markdown

@coderabbitai coderabbitai Bot Apr 16, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor

ensure_vscode_ready currently bypasses its timeout budget.

The new loop uses fixed 90s attempts and 10s sleeps, so total wait can exceed the function’s timeout argument. Use a remaining-time budget tied to timeout.

Proposed change 
-    max_nav_attempts = 3
-    for attempt in range(max_nav_attempts):
+    max_nav_attempts = 3
+    deadline = time.monotonic() + timeout
+    for attempt in range(max_nav_attempts):
+        remaining = int(deadline - time.monotonic())
+        if remaining <= 0:
+            raise TimeOutError("timeout waiting for VSCode Ansible icon")
         try:
-            wait_displayed(driver, "//a[`@aria-label`='Ansible']", timeout=90)
+            wait_displayed(
+                driver,
+                "//a[`@aria-label`='Ansible']",
+                timeout=min(remaining, 90),
+            )
             break
         except (TimeoutException, TimeOutError):
             if attempt == max_nav_attempts - 1:
                 raise
             driver.refresh()
-            time.sleep(10)
+            time.sleep(min(10, max(0, deadline - time.monotonic())))
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
max_nav_attempts = 3
for attempt in range(max_nav_attempts):
try:
wait_displayed(driver, "//a[@aria-label='Ansible']", timeout=90)
break
except (TimeoutException, TimeOutError):
if attempt == max_nav_attempts - 1:
raise
driver.refresh()
time.sleep(10)
max_nav_attempts = 3
deadline = time.monotonic() + timeout
for attempt in range(max_nav_attempts):
remaining = int(deadline - time.monotonic())
if remaining <= 0:
raise TimeOutError("timeout waiting for VSCode Ansible icon")
try:
wait_displayed(
driver,
"//a[`@aria-label`='Ansible']",
timeout=min(remaining, 90),
)
break
except (TimeoutException, TimeOutError):
if attempt == max_nav_attempts - 1:
raise
driver.refresh()
time.sleep(min(10, max(0, deadline - time.monotonic())))
🤖 Prompt for AI Agents 
Verify each finding against the current code and only fix it if needed.

In `@test/ui/utils/ui_utils.py` around lines 81 - 90, The loop in
ensure_vscode_ready uses fixed 90s waits and fixed 10s sleeps, which can exceed
the caller-supplied timeout; change it to track elapsed time (e.g. via
time.monotonic()) and compute a remaining_time budget for each iteration when
calling wait_displayed(driver, "//a[`@aria-label`='Ansible']",
timeout=remaining_time) and for any sleep (sleep only min(10, remaining_time)),
stop retrying and raise if remaining_time <= 0; update handling around
exceptions (TimeoutException, TimeOutError) and the driver.refresh() call so
they respect the remaining timeout budget.

vscode_run_command(driver, ">Ansible: Focus on Ansible Development Tools View")
find_element_across_iframes(
driver,
Expand Down Expand Up @@ -1039,7 +1048,7 @@ def vscode_run_command(
driver,
command_box,
"//input[@aria-controls='quickInput_list']",
timeout=2,
timeout=5,
)
break
except (
Expand Down
Loading