Add guidance for testing conventions in each language by titusfortner · Pull Request #16734 · SeleniumHQ/selenium

titusfortner · 2025-12-15T21:25:13Z

Reason for PR

Each language has its own tribal knowledge right now for how it works, from annotations to helper methods to common locations for things. This is intended to document those things to make it easier to work in each language.

💥 What does this PR do?

Removes language specific testing details from primary README and moves everything to separate TESTING.md files

💡 Additional Considerations

This is in draft because they were auto-generated as a place to start. I updated some of them more than others, but they can all use a lot of work. Please assist, thanks.

Delta456 · 2025-12-15T21:50:29Z

Hyperlinking to each binding readme would also be beneficial.

Yes, this is a good idea. I was going to do that, but our READMEs aren't consistent right now with what they do and don't contain, and I was thinking it might make sense to update them to mostly point to our website docs, and right now the coding agents we're targeting can't access the internet.

I added an issue to track this - #16740

and run specific test with bazel

shs96c · 2025-12-16T12:07:34Z

+| `@NoDriverBeforeTest` | Test needs custom capabilities or tests driver creation itself. Driver is destroyed, must use `createNewDriver(capabilities)` in the test to create one. |
+| `@NoDriverAfterTest` | Test leaves browser in a bad state. Driver is restarted after the test. Also accepts `failedOnly`.                                                       |
+
+For tests needing two browsers simultaneously (e.g., multi-user scenarios), create a second instance with `localDriver = new ChromeDriver()`. This driver is automatically quit after the test.


Nope! This will (at best) create a Chrome instance instead of the browser being used for the tests. The only reason why the browser is closed after tests is because the entire process tree is killed.

The correct way to start a new browser instance is via something like:

RemoteWebDriver.builder().oneOf(Browser.detect()).build();

Thank you! This is exactly why we need this file. (this is much more helpful than the generic warning I made about being careful hard-coding creation of new drivers)

Also, if you name it localDriver, it is automatically checked in after hook: https://github.com/SeleniumHQ/selenium/blob/trunk/java/test/org/openqa/selenium/testing/JupiterTestBase.java#L90

aguspe · 2025-12-16T22:31:53Z

+## Build Files
+
+* Adding tests shouldn't require Bazel changes—`rb_integration_test` uses glob patterns.
+* Make sure `*_spec.rb` files are in a directory with a `BUILD.bazel` containing `rb_integration_test`.


I think we should add a section on how to use environment variables especially for BiDi testing @titusfortner :

Environment Variables

Environment variables control test execution behavior and enable specific features.

BiDi Testing

To run tests with BiDi (Bidirectional) protocol enabled:

# Enable BiDi for all tests WD_REMOTE_BROWSER=chrome BIDI=true bazel test //rb/spec/integration/... # Run BiDi-specific tests bazel test //rb/spec/integration/selenium/webdriver/bidi/...

Available Variables

Variable Purpose Values Example

BIDI Enable BiDi protocol true, false BIDI=true

WD_REMOTE_BROWSER Specify browser for remote tests chrome, firefox, edge, safari WD_REMOTE_BROWSER=firefox

HEADLESS Run tests in headless mode true, false HEADLESS=true

DEBUG Enable debug logging true, false DEBUG=true

Examples

# Run Chrome tests with BiDi enabled BIDI=true bazel test //rb/spec/integration/... --test_tag_filters=chrome # Run headless Firefox tests HEADLESS=true bazel test //rb/spec/integration/... --test_tag_filters=firefox # Run remote tests on Edge with BiDi WD_REMOTE_BROWSER=edge BIDI=true bazel test //rb/spec/integration/... --test_tag_filters=remote # Combine multiple variables BIDI=true HEADLESS=true DEBUG=true bazel test //rb/spec/integration/selenium/webdriver/bidi/...

Testing Guard Behavior

Environment variables interact with test guards. For example:

# This test only runs when BiDi is enabled it 'uses BiDi feature', only: {bidi: true} do # Test code end # This test is excluded when BiDi is enabled it 'classic WebDriver only', exclusive: {bidi: false} do # Test code end

Run with BIDI=true to see these guards in action.

Please add away, you should be able to commit to this PR directly. Everything here - https://github.com/SeleniumHQ/selenium/blob/trunk/README.md#ruby-1 should probably be included.
Thanks!

RenderMichael · 2025-12-21T05:18:35Z

+bazel test //dotnet/... --flaky_test_attempts=3 --pin_browsers=true
+bazel test //dotnet/... --test_output=all --pin_browsers=true
+```
+


A few lines about running tests in the IDE.

Suggested change

### IDE Support

The test infrastructure supports Visual Studio's and Rider's IDE, which can be more convenient in the inner loop when developing features and writing tests. Simply open the solution after building with `bazel` and the IDE should work like normal.

Note that any bugs involving a discrepancy between the `.csproj` and `BUILD.bazel` files would manifest in running the tests through the IDE rather than with `bazel test`; any changes that affect either build systems should be tested with `bazel test` to ensure proper functionality.

Added detailed documentation on environment variables for testing, including BiDi protocol usage and examples.

cgoldberg · 2026-01-21T16:36:30Z

@titusfortner I just updated this branch and fixed the merge conflicts. Is there any more work that needs to be done, or is this ready to merge?

* Enhance Ruby Testing Guide with setup and execution details Expanded the Ruby Testing Guide to include local development setup, test running methods, and environment variables. * Add Steep type signature documentation for Ruby contributors - Added comprehensive Type Signatures with Steep section - Includes instructions for adding RBS files for new classes - Added guide for updating existing type signatures - Documented Steep commands for type checking - Added best practices and common RBS types reference table - Added RubyMine IDE setup instructions - Enhanced debugging section with Bazel and RSpec options - Added interactive REPL usage with bazel run //rb:console * comments addressed * [rb] address review feedback on TESTING.md - Add Bundler install command before `bundle install` (luke-hill) - Add Debugging section with `debug` gem and `rdbg` instructions, moved from trunk README#ruby (titusfortner)

Copilot

Pull request overview

This PR extracts language-specific testing guidance from the top-level README.md into per-language TESTING.md guides, with the goal of documenting language-specific conventions, helpers, and common test commands.

Changes:

Replaces the detailed, language-specific testing section in README.md with a pointer to per-language TESTING.md files.
Adds initial TESTING.md guides for Java, JavaScript, Python, Ruby, .NET, and Rust.
Documents common testing conventions (frameworks, helpers, skipping mechanisms, and example commands) within each language directory.

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 8 comments.

Show a summary per file

File	Description
README.md	Removes embedded per-language test command blocks and points contributors to language-level `TESTING.md` guides.
rust/TESTING.md	Adds a minimal Rust testing guide with a Bazel command and recommended env flags.
rb/TESTING.md	Adds a comprehensive Ruby testing/conventions guide (RSpec, guards, helpers, env vars, tooling).
py/TESTING.md	Adds a Python testing/conventions guide (pytest, markers/fixtures, Bazel + local pytest workflows).
javascript/selenium-webdriver/TESTING.md	Adds a JavaScript testing/conventions guide (Mocha, suite helpers, Bazel commands).
java/TESTING.md	Adds a Java testing/conventions guide (JUnit 5 base classes, annotations, Bazel commands).
dotnet/TESTING.md	Adds a .NET testing/conventions guide (NUnit, fixtures/attributes, Bazel commands).

+
+- `bazel test //rust/...`
+
+Recommended flags: 


+```shell
+# Enable BiDi for all tests
+WD_REMOTE_BROWSER=chrome BIDI=true bazel test //rb/spec/integration/...
+
+# Run BiDi-specific tests
+bazel test //rb/spec/integration/selenium/webdriver/bidi/...
+


+| Variable | Purpose | Values | Example |
+| --- | --- | --- | --- |
+| `BIDI` | Enable BiDi protocol | `true`, `false` | `BIDI=true` |
+| `WD_REMOTE_BROWSER` | Specify browser for remote tests | `chrome`, `firefox`, `edge`, `safari` | `WD_REMOTE_BROWSER=firefox` |
+| `HEADLESS` | Run tests in headless mode | `true`, `false` | `HEADLESS=true` |


+        ├── chrome/
+        ├── firefox/
+        ├── safari/
+        ├── bidi`


+bazel test //py:common-chrome  # Common tests with Chrome
+
+# A single test file with Chrome:
+bazel test //py:common-chrome-test/selenium/webdriver/common/alerts_tests.py
+
+# With BiDi protocol
+bazel test //py:common-chrome-bidi


+# A single test file with Chrome:
+bazel test //py:common-chrome-test/selenium/webdriver/common/alerts_tests.py
+
+# With BiDi protocol
+bazel test //py:common-chrome-bidi
+
+# Test filters
+bazel test //py/... --test_tag_filters=chrome
+
+# Additional arguments
+bazel test //py/... --flaky_test_attempts=3
+bazel test //py/... --test_output=all
+bazel test //py/... --test_output=streamed  # Live output for debugging
+bazel test //py:test-chrome --headless
+
+# Run a specific test in a test file
+bazel test //py:common-chrome-bidi-test/selenium/webdriver/common/bidi_browsing_context_tests.py \
+  --test_arg=-k \
+  --test_arg=test_get_tree_with_child \
+


+```python
+def test_element_is_displayed(driver, pages):
+    pages.load("javascriptPage.html")
+
+    element = driver.find_element(By.ID, "displayed")
+    assert element.is_displayed() is True
+
+@pytest.mark.xfail_safari(reason="Safari doesn't support this")
+def test_something_safari_fails(driver, pages):
+    # Expected to fail on Safari


+bazel test //javascript/selenium-webdriver:all  # All tests
+
+# Specific browser tests
+bazel test //javascript/selenium-webdriver:element-finding-test-chrome
+bazel test //javascript/selenium-webdriver:element-finding-test-firefox
+
+# Additional Arguments
+bazel test //javascript/selenium-webdriver:... --flaky_test_attempts=3
+bazel test //javascript/selenium-webdriver:... --test_output=all


Add guidance for testing conventions in each language

f54b076

titusfortner requested review from RenderMichael, aguspe, asolntsev, cgoldberg, diemol, harsha509, navin772, nvborisenko and p0deje December 15, 2025 21:25

titusfortner added C-py Python Bindings C-rb Ruby Bindings C-dotnet .NET Bindings C-java Java Bindings C-nodejs JavaScript Bindings C-rust Rust code is mostly Selenium Manager labels Dec 15, 2025

selenium-ci added the B-manager Selenium Manager label Dec 15, 2025

titusfortner mentioned this pull request Dec 15, 2025

Canonical approach to supporting AI agent directions #16735

Merged

Delta456 reviewed Dec 15, 2025

View reviewed changes

update rust

844b9ce

titusfortner requested a review from bonigarcia December 15, 2025 21:55

Python updates and additions

00a62a3

RenderMichael reviewed Dec 15, 2025

View reviewed changes

Comment thread dotnet/TESTING.md

navin772 reviewed Dec 16, 2025

View reviewed changes

Comment thread py/TESTING.md

navin772 added 2 commits December 16, 2025 12:50

add docs to run tests directly with pytest

3bc48df

and run specific test with bazel

Merge branch 'trunk' into testing_guides

58a693b

diemol approved these changes Dec 16, 2025

View reviewed changes

run formatter

09277b3

bonigarcia approved these changes Dec 16, 2025

View reviewed changes

shs96c reviewed Dec 16, 2025

View reviewed changes

cgoldberg approved these changes Dec 17, 2025

View reviewed changes

navin772 approved these changes Dec 18, 2025

View reviewed changes

aguspe reviewed Dec 19, 2025

View reviewed changes

RenderMichael reviewed Dec 21, 2025

View reviewed changes

aguspe and others added 2 commits December 29, 2025 15:06

Enhance TESTING.md with environment variable details

392f4cd

Added detailed documentation on environment variables for testing, including BiDi protocol usage and examples.

Merge trunk and fix conflicts

69be16d

cgoldberg mentioned this pull request Jan 22, 2026

[py] make bazel test target names consistent with other languages #16969

Merged

Copilot AI review requested due to automatic review settings May 10, 2026 09:18

Copilot started reviewing on behalf of diemol May 10, 2026 09:19 View session

Copilot AI reviewed May 10, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Add guidance for testing conventions in each language#16734

Add guidance for testing conventions in each language#16734
titusfortner wants to merge 9 commits into
trunkfrom
testing_guides

titusfortner commented Dec 15, 2025

Uh oh!

Delta456 Dec 15, 2025

Uh oh!

titusfortner Dec 15, 2025

Uh oh!

titusfortner Dec 16, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

shs96c Dec 16, 2025

Uh oh!

titusfortner Dec 16, 2025

Uh oh!

aguspe Dec 16, 2025

Uh oh!

titusfortner Dec 19, 2025

Uh oh!

RenderMichael Dec 21, 2025

Uh oh!

cgoldberg commented Jan 21, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

12 participants

Variable	Purpose	Values	Example
`BIDI`	Enable BiDi protocol	`true`, `false`	`BIDI=true`
`WD_REMOTE_BROWSER`	Specify browser for remote tests	`chrome`, `firefox`, `edge`, `safari`	`WD_REMOTE_BROWSER=firefox`
`HEADLESS`	Run tests in headless mode	`true`, `false`	`HEADLESS=true`
`DEBUG`	Enable debug logging	`true`, `false`	`DEBUG=true`

+### IDE Support
+The test infrastructure supports Visual Studio's and Rider's IDE, which can be more convenient in the inner loop when developing features and writing tests. Simply open the solution after building with `bazel` and the IDE should work like normal.
+Note that any bugs involving a discrepancy between the `.csproj` and `BUILD.bazel` files would manifest in running the tests through the IDE rather than with `bazel test`; any changes that affect either build systems should be tested with `bazel test` to ensure proper functionality.

Uh oh!

Conversation

titusfortner commented Dec 15, 2025

Reason for PR

💥 What does this PR do?

💡 Additional Considerations

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Environment Variables

BiDi Testing

Available Variables

Examples

Testing Guard Behavior

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cgoldberg commented Jan 21, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

12 participants