Support retries in MultiClusterWrapper#737
Conversation
|
I couldn't make unit tests work locally for me without Python and package upgrades. |
| prereq: | ||
| pyenv install --skip-existing $(PY38) | ||
| pyenv local $(PY38) | ||
| pyenv install --skip-existing $(PY312) |
There was a problem hiding this comment.
How come we jump to 3.12 instead of 3.9 ?
There was a problem hiding this comment.
This follows dev environment preparation from Rally's Makefile, where 3.12 is used as well. It aligns both.
There was a problem hiding this comment.
Ah, I hadnt spotted we had switched there (We used to use 3.8, then when we upgraded to 3.12 we jumped straight to 3.12). I suppose from a testing perspective we still test on 3.9, so thats ok I guess
There was a problem hiding this comment.
Exactly, we test 3.9-3.12, but for dev purposes configure 3.12.
| # call original runner assuming a single-cluster one, see https://github.com/elastic/rally/pull/488 | ||
| # and https://github.com/elastic/rally/pull/1563 for additional context |
There was a problem hiding this comment.
Actually we cannot assume that, this requires a modification.
There was a problem hiding this comment.
I've generalized in e3f42ea. Today multi-cluster-wrapper is only used with built-in Rally runners which are exclusively single-cluster. But I can imagine a situation where someone wants to iterate through all the clusters providing a dictionary of clusters for the runner to choose from. Perhaps that's too much.
The
multi-cluster-wrappercustom runner unwraps the runner before calling it for every remote cluster. This strips the wrapper responsible for retries (src) from the runner making operation parameters such asretry-until-successignored.Another wrapper removed during the unwrap is
MultiClientRunner(src, src). This wrapper assumes that ES client is specified through a{"<cluster-name>": <client>}dictionary, and for single-cluster runners (all built-in Rally runners) this wrapper extracts the client fromdefaultcluster, while for multi-cluster runners (custom ones), provides entire dictionary.This PR tries to address it by resigning from the unwrap. For this to work ES client is passed in a dictionary with
defaultkey which assumes the underlying runner will be a single-cluster one.