Upgrades Elastic search client and `urlib3` libraries. by fressi-elastic · Pull Request #2024 · elastic/rally

fressi-elastic · 2026-01-13T14:59:20Z

This PR upgrades the Elasticsearch client libraries and urllib3 to newer major versions, updating from Elasticsearch 8.x to 9.x and urllib3 1.x to 2.x. The changes include API parameter updates, refactoring of header handling logic, and deprecation of certain track dependencies.

Changes:

It sets the minimal Elasticsearch server version to 8.0.0 and update documentation.
It updates python package requirements:
- elasticsearch[async] from 8.6.1 to 9.2.1
- elastic-transport from 8.4.1 to 9.2.1
- urllib3 from 1.26.19 to 2.6.3
- docker from 6.0.0 to 7.1.0 (due to incompatibility with newer urllib3 releases)
It renames maxsize parameter to connections_per_node in client initialization to match the new API
It removes most of the logic that diverged with the official Elasticsearch client implementation.
It allows to specify compatibility mode (8, 9, etc.) in the perform_request method.
It refactors header handling logic into reusable functions and improved MIME type compatibility handling
It replaces obsolete docker-compose command with docker compose to avoid conflicts with new urlib3 version.
It fixes it/docker_dev_image_test.py and update elasticsearch server to version 9.2.4.
It update unit tests accordly.

This PR replaces the now obsolete #2003 and it requires PR elastic/rally-tracks#1000

…pgrade of elastisearch client libraries.

Copilot

Pull request overview

This PR upgrades the Elasticsearch client libraries and urllib3 to newer major versions, updating from Elasticsearch 8.x to 9.x and urllib3 1.x to 2.x. The changes include API parameter updates, refactoring of header handling logic, and deprecation of certain track dependencies.

Changes:

Updated elasticsearch[async] from 8.6.1 to 9.2.1, elastic-transport from 8.4.1 to 9.2.1, and urllib3 from 1.26.19 to 2.6.3
Renamed maxsize parameter to connections_per_node in client initialization to match the new API
Refactored header handling logic into reusable functions and improved MIME type compatibility handling

Reviewed changes

Copilot reviewed 11 out of 12 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
pyproject.toml	Updated dependency versions for elasticsearch, elastic-transport, urllib3, and pip
esrally/client/factory.py	Changed `maxsize` to `connections_per_node` parameter
tests/client/factory_test.py	Updated test to use `connections_per_node` parameter
esrally/client/common.py	Refactored header combining and MIME type compatibility logic into dedicated functions
esrally/client/synchronous.py	Updated to use new header handling functions and added endpoint_id/path_parts parameters
esrally/client/asynchronous.py	Updated RallyAsyncElasticsearch signature and header handling to use new functions
esrally/track/loader.py	Added deprecation handling for elasticsearch and elastic-transport dependencies in tracks
esrally/utils/versions.py	Added type hints to is_version_identifier function
it/init.py	Updated test distributions to include 9.2.4 and removed 6.8.0/ARM logic
tests/client/common_test.py	Removed test file (likely obsolete after refactoring)
tests/driver/runner_test.py	Added typing import and @typing.no_type_check decorator to test

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

esrally/client/common.py

…ocker-compose`.

…ocker-compose` (second part).

Copilot

Pull request overview

Copilot reviewed 20 out of 21 changed files in this pull request and generated 2 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

esrally/track/loader.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Concerns lifted after internal discussion.

…rform_request methods.

fressi-elastic · 2026-02-03T11:23:52Z

@gbanasiak I followed up the suggestion received during past code review interactions and I moved the logic out of _perform_request method. This highly reduces the code required to implement perform_request methods. I also added a table in the documentation (assuming this change will be introduced along development of version 2.14.0, and that we will create release 2.13.0 just before merging this change.

gbanasiak

Many thanks for simplifications and getting rid of the product checker code! Comments after a more thorough pass below.

docs/adding_tracks.rst

docs/command_line_reference.rst

docs/migrate.rst

docs/versions.rst

docs/adding_tracks.rst

esrally/client/common.py

gbanasiak · 2026-02-04T12:28:11Z

esrally/client/common.py

+    if body is not None:
+        # It ensures content-type and accept headers are initialized.
+        mimetype = "application/json"
+        if path is not None and path.endswith("/_bulk"):
+            # Server version 9 is picky about this. This should improve compatibility.
+            mimetype = "application/x-ndjson"
+        for header in ("content-type", "accept"):
+            headers.setdefault(header, mimetype)


This seems weird. Why is this required? Is there anything similar in Elasticsearch Python client code? If 9.x version of the client can talk to 9.x version of the server by default, why do we need any extra modifications, e.g. for _bulk?

Older versions of server where assuming JSON type has to be assumed if not specified when the body was clearly a Json. Newer versions are getting more strict, so we have to provide the headers which the track is not providing to allow posting jsons or bulk objects to the server. Version 9 is more picky than version 8.

Rally already had this bofore my change for the reason I explained here.

BulkIndex runner calls Python client's bulk() which sets the right headers, so the above code seems redundant. Does this address a specific failure? In which test, track, operation?

esrally/client/synchronous.py

it/__init__.py

tests/client/synchronous_test.py

- Update common_test.py for get_compatibility_mode and ensure_mimetype_headers API - Update synchronous_test.py and asynchronous_test.py with perform_request tests - Use monkeypatch and create_autospec; fix type hints and assertions - Client fixes for mypy/pylint Co-authored-by: Cursor <cursoragent@cursor.com>

… error. Refactor test cases.

gbanasiak · 2026-02-05T11:03:40Z

@fressi-elastic I've worked on style and content of EOL policy, see cdc94e9.

fressi-elastic force-pushed the esclient-upgrade-2 branch 3 times, most recently from ee26164 to 1e8aeeb Compare January 13, 2026 17:47

Upgrade Elastic search client libraries and urlib3.

1421ad0

fressi-elastic force-pushed the esclient-upgrade-2 branch from 1e8aeeb to 1421ad0 Compare January 13, 2026 18:34

Try enabling client compatibility mode to support ES version 8

de1c485

fressi-elastic force-pushed the esclient-upgrade-2 branch from 553574d to de1c485 Compare January 14, 2026 13:03

fressi-elastic added 8 commits January 19, 2026 11:34

Update client compatibility version handling.

1011049

Drop support for distribution version < 8, add support for version 9

31a59e3

Remove obsolete unit test.

169050a

Change track dependency installation: add pip to venv and deprecate u…

494a501

…pgrade of elastisearch client libraries.

It removes unnecessary changes.

478846c

Skip installing dependencies, when all have been removed.

be8fe13

Use mimetype "application/x-ndjson" as default for bulk requests.

2a05ff1

Rewrite how headers are being reconbined in perform_request method

46c6f19

fressi-elastic force-pushed the esclient-upgrade-2 branch from b9c230c to 46c6f19 Compare January 19, 2026 17:39

fressi-elastic requested a review from Copilot January 19, 2026 19:03

Copilot AI reviewed Jan 19, 2026

View reviewed changes

esrally/client/common.py Outdated Show resolved Hide resolved

fressi-elastic added 2 commits January 19, 2026 22:27

Update requests version.

e9018e7

It uses docker compose command instead of the now deprecated old `d…

933bc1d

…ocker-compose`.

fressi-elastic changed the title ~~Upgrade Elastic search client libraries and urlib3.~~ It upgrades Elastic search client and urlib3 libraries. Jan 20, 2026

fressi-elastic added 2 commits January 20, 2026 04:37

It removes unnecessary dependency.

38aafd8

It uses docker compose command instead of the now deprecated old `d…

24f6479

…ocker-compose` (second part).

fressi-elastic force-pushed the esclient-upgrade-2 branch from b1a7027 to 24f6479 Compare January 20, 2026 03:38

fressi-elastic requested a review from Copilot January 20, 2026 03:43

Copilot AI reviewed Jan 20, 2026

View reviewed changes

esrally/track/loader.py Outdated Show resolved Hide resolved

esrally/track/loader.py Outdated Show resolved Hide resolved

fressi-elastic and others added 4 commits January 20, 2026 04:47

Corrected capitalization of 'rally' to 'Rally'.

79e299b

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Simplify slice dependency[len(deprecated) : len(deprecated) + 1]

4ec0a24

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Refactor and test function _filter_requirements.

63dabea

Restore docker as Python dependency.

5c86570

Remove _ProductChecker class

6b1132b

fressi-elastic added 6 commits February 3, 2026 10:54

Simplify the methods overriding by taking headers handling out of _pe…

c40749c

…rform_request methods.

Add unit tests for common.py

718e10b

Add test case for RallySyncElasticsearch.perform_request

501ff35

Add test for perform_request method to asynchronous_test.py

0b9e0f7

Update rally documentation.

8bfbe88

It handles the no-headers corner case.

826601f

Debug it test cases.

4f783e5

gbanasiak reviewed Feb 4, 2026

View reviewed changes

fressi-elastic and others added 3 commits February 4, 2026 14:34

Fix documentation removing mentions to OSS release download.

9c7f4b8

Revert changes to docs/migrate.rst and docs/add_tracks.rst

1ee9dfa

fressi-elastic force-pushed the esclient-upgrade-2 branch 2 times, most recently from 32d3650 to 92aaaf9 Compare February 4, 2026 14:08

Update docs/version.rst

7a83814

fressi-elastic force-pushed the esclient-upgrade-2 branch from 92aaaf9 to 7a83814 Compare February 4, 2026 14:11

fressi-elastic and others added 8 commits February 4, 2026 15:25

Add docstrings to perform_request methods.

6b5c657

Test against the minimal supported version of ElasticSearch server.

16fe066

Correct some comments style

60798c6

Update the code for clarity.

554db60

Write warning when get_compatibility_mode fails instead of raising an…

a217eff

… error. Refactor test cases.

Fix test cases.

4c899e8

Pin dependency version: elasticsearch[async]==9.3.0

0b2a789

Clarify EOL policy

cdc94e9

Drop old versions from new track instructions

ea1311a

gbanasiak mentioned this pull request Feb 5, 2026

Remove support for mapping types #2036

Open

fressi-elastic changed the title ~~It upgrades Elastic search client and urlib3 libraries.~~ Upgrades Elastic search client and urlib3 libraries. Feb 10, 2026

Comments

Conversation

fressi-elastic commented Jan 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

fressi-elastic commented Feb 3, 2026

Uh oh!

gbanasiak left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

gbanasiak Feb 4, 2026

Choose a reason for hiding this comment

Uh oh!

fressi-elastic Feb 4, 2026

Choose a reason for hiding this comment

Uh oh!

fressi-elastic Feb 4, 2026

Choose a reason for hiding this comment

Uh oh!

gbanasiak Feb 5, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

gbanasiak commented Feb 5, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

fressi-elastic commented Jan 13, 2026 •

edited

Loading