Skip to content

chore(consume): add retries for connection refused from post requests#2195

Closed
spencer-tb wants to merge 1 commit intomainfrom
rpc-post-request-connection-refused
Closed

chore(consume): add retries for connection refused from post requests#2195
spencer-tb wants to merge 1 commit intomainfrom
rpc-post-request-connection-refused

Conversation

@spencer-tb
Copy link
Collaborator

@spencer-tb spencer-tb commented Sep 23, 2025

🗒️ Description

Attempts to fix sporadic connection refused errors in Hive consume simulators by adding retry logic to post requests sent to clients. This aims to address an assumed race condition where Hive reports a client as ready but the RPC service isn't quite available yet during startup, causing tests to fail with "Connection refused" errors.

🔗 Related Issues or PRs

Sporadic fails for Besu: https://hive.ethpandaops.io/#/compare/fusaka?runs=1758510938-01cbc0c3af77726a9969c9050108697a,1758441660-b4e01a5f7cea5a2ed8fc8d8c2bb4380e,1758389473-146b809e40427cf98675ec853360495e,1758207887-90de6bb128a9349d19d3b793802ef7e6,1758155899-75a28ff1edfb65c9297bf0be9f2e9f09,1758078900-e2c9d1724030ef8fac4a181f1928bd54,1757992304-a4c773a17b794af44eb5b00958f71b5c,1757903263-fa62cb0a95645c31420886e8d4f4c27a,1757817185-941983bc143dff74b7ca8a778195036f,1757733636-e801251e63ebb945ceddfc627a8e690c

✅ Checklist

  • All: Ran fast tox checks to avoid unnecessary CI fails, see also Code Standards and Enabling Pre-commit Checks:
    uvx --with=tox-uv tox -e lint,typecheck,spellcheck,markdownlint
  • All: PR title adheres to the repo standard - it will be used as the squash commit message and should start type(scope):.
  • All: Considered adding an entry to CHANGELOG.md.
  • All: Considered updating the online docs in the ./docs/ directory.
  • All: Set appropriate labels for the changes (only maintainers can apply labels).

@spencer-tb spencer-tb added type:chore Type: Chore scope:consume Scope: Consume command suite labels Sep 23, 2025
@spencer-tb spencer-tb changed the title chore(rpc): add retries for connection refused from post requests. chore(consume): add retries for connection refused from post requests. Sep 23, 2025
@spencer-tb spencer-tb changed the title chore(consume): add retries for connection refused from post requests. chore(consume): add retries for connection refused from post requests Sep 23, 2025
Copy link
Member

@danceratopz danceratopz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey, not such a big fan of letting these retry loops pervade more of our code. Will post an alternative with tenacity. And we can evaluate the options, see also:

result = response_json["result"]
return result
except requests.ConnectionError as e:
if ("Connection refused" in str(e) or "Errno 111" in str(e)) and attempt < 4:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this has an off by one:

Suggested change
if ("Connection refused" in str(e) or "Errno 111" in str(e)) and attempt < 4:
if ("Connection refused" in str(e) or "Errno 111" in str(e)) and attempt < 5:

@danceratopz
Copy link
Member

@spencer-tb please check #2205 as an alternative.

@spencer-tb
Copy link
Collaborator Author

CIFO #2205

@spencer-tb spencer-tb closed this Sep 26, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

scope:consume Scope: Consume command suite type:chore Type: Chore

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

Comments