Skip to content

fix: release lock guards before awaiting channel send (#869) - #1003

Merged
zmanian merged 3 commits into
nearai:stagingfrom
zmanian:pr-905-staging
Mar 12, 2026
Merged

zmanian merged 3 commits into
nearai:stagingfrom
zmanian:pr-905-staging

Conversation

@zmanian

@zmanian zmanian commented Mar 12, 2026

Copy link
Copy Markdown
Collaborator

Summary

Supersedes #905 for the staging merge train.

qbit-glitch and others added 3 commits March 11, 2026 17:53
Clone `mpsc::Sender` out of `RwLock` before `.send().await` to prevent
read guards from blocking write lock acquisition (shutdown/start) when
the channel buffer is full.

Fixed call sites:
- src/channels/http.rs: process_message()
- src/channels/web/server.rs: chat_send_handler(), chat_approval_handler()
- src/channels/web/handlers/chat.rs: chat_send_handler(), chat_approval_handler()
- src/channels/web/ws.rs: handle_client_message() (2 sites)
- src/channels/wasm/wrapper.rs: process_emitted_messages() (2 impls, also
  scoped rate_limiter write lock per-iteration)

Includes regression test: shutdown_completes_while_process_message_blocked

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
(cherry picked from commit 84802e1)
The regression-test-check workflow failed because origin/main wasn't
available as a ref in the CI environment. actions/checkout@v4 fetches
the PR merge ref history but doesn't make the base branch ref available
for three-dot diff comparisons.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
(cherry picked from commit 1d5a7bd)
@github-actions github-actions Bot added scope: channel/web Web gateway channel scope: channel/wasm WASM channel runtime scope: ci CI/CD workflows size: L 200-499 changed lines labels Mar 12, 2026
@zmanian
zmanian enabled auto-merge (squash) March 12, 2026 00:54
@zmanian zmanian added the skip-regression-check Bypass regression test CI gate (tests exist but not in tests/ dir) label Mar 12, 2026
@github-actions github-actions Bot added risk: medium Business logic, config, or moderate-risk modules contributor: core 20+ merged PRs labels Mar 12, 2026
@gemini-code-assist

Copy link
Copy Markdown
Contributor

Summary of Changes

Hello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request cherry-picks critical fixes from a previous PR (#905) to resolve a deadlock issue in asynchronous channel operations. The core problem involved RwLock guards being held across await points during message sending, which could block system shutdowns. The changes ensure that these locks are released promptly before asynchronous send operations, thereby preventing deadlocks and improving system stability. A new regression test has been added to verify this corrected behavior.

Highlights

  • Deadlock Prevention: Addressed a deadlock scenario by ensuring RwLock guards are released before awaiting channel send operations across various channel implementations.
  • Regression Testing: Introduced a new regression test in src/channels/http.rs to validate the fix for the deadlock during channel shutdown.
  • Code Refactoring: Refactored message sending logic in wasm, web/handlers, web/server, and web/ws modules to prevent holding locks across await points.
  • CI/Workflow Updates: Included a CI-only skip marker commit and carried over a regression-check workflow fix from a superseded pull request (fix: release lock guards before awaiting channel send (#869) #905).
Changelog
  • src/channels/http.rs
    • Added a regression test to verify that channel shutdown completes even when process_message is blocked due to a full channel buffer, which previously caused a deadlock.
  • src/channels/wasm/wrapper.rs
    • Modified WasmChannel::emit_messages to clone the message sender (tx) before iterating and sending messages, ensuring the RwLock read guard on message_tx is not held across send().await.
    • Refactored rate limiting logic in WasmChannel::emit_messages to acquire and release the rate_limiter write lock within a smaller scope, preventing it from being held across send().await.
    • Updated comments to clarify that no locks are held across send().await.
    • Applied similar changes to process_emitted_messages to clone the sender and scope the rate limiter lock.
  • src/channels/web/handlers/chat.rs
    • Modified chat_send_handler and chat_approval_handler to clone the message sender (tx) after acquiring it from the msg_tx RwLock, ensuring the read guard is released before tx.send(msg).await.
  • src/channels/web/server.rs
    • Modified chat_send_handler and chat_approval_handler to clone the message sender (tx) after acquiring it from the msg_tx RwLock, ensuring the read guard is released before tx.send(msg).await.
  • src/channels/web/ws.rs
    • Modified handle_client_message in two locations to clone the message sender (tx) after acquiring it from the msg_tx RwLock, ensuring the read guard is released before tx.send(incoming).await.
Ignored Files
  • Ignored by pattern: .github/workflows/** (1)
    • .github/workflows/regression-test-check.yml
Activity
Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature Command Description
Code Review /gemini review Performs a code review for the current pull request in its current state.
Pull Request Summary /gemini summary Provides a summary of the current pull request in its current state.
Comment @gemini-code-assist Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help /gemini help Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

  1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

@zmanian

zmanian commented Mar 12, 2026

Copy link
Copy Markdown
Collaborator Author

Staging replacement for #905. This PR cherry-picks the actual lock-guard fix plus the regression-check workflow fix onto current staging, without carrying over the unrelated main merge history from the original branch.\n\nThe regression test gate on #905 was a false negative; this branch includes a CI-only [skip-regression-check] empty commit and the skip-regression-check label so the required check can pass while the rest of the matrix validates the real code changes.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request correctly addresses a potential deadlock issue by ensuring RwLock read guards are not held across .await points. The fix is consistently applied across multiple channel implementations (http, wasm, web). Cloning the mpsc::Sender and scoping lock acquisitions are idiomatic and effective solutions in async Rust. The inclusion of a regression test in src/channels/http.rs is particularly commendable, as it specifically targets the deadlock scenario and ensures the fix is robust. The changes are clean, well-commented, and significantly improve the concurrency safety of the channel implementations.

@zmanian
zmanian merged commit ef34943 into nearai:staging Mar 12, 2026
15 checks passed
bkutasi pushed a commit to bkutasi/ironclaw that referenced this pull request Mar 28, 2026
…earai#1003)

* fix: release lock guards before awaiting channel send (nearai#869)

Clone `mpsc::Sender` out of `RwLock` before `.send().await` to prevent
read guards from blocking write lock acquisition (shutdown/start) when
the channel buffer is full.

Fixed call sites:
- src/channels/http.rs: process_message()
- src/channels/web/server.rs: chat_send_handler(), chat_approval_handler()
- src/channels/web/handlers/chat.rs: chat_send_handler(), chat_approval_handler()
- src/channels/web/ws.rs: handle_client_message() (2 sites)
- src/channels/wasm/wrapper.rs: process_emitted_messages() (2 impls, also
  scoped rate_limiter write lock per-iteration)

Includes regression test: shutdown_completes_while_process_message_blocked

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
(cherry picked from commit 84802e1)

* ci: fetch base branch before regression test check

The regression-test-check workflow failed because origin/main wasn't
available as a ref in the CI environment. actions/checkout@v4 fetches
the PR merge ref history but doesn't make the base branch ref available
for three-dot diff comparisons.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
(cherry picked from commit 1d5a7bd)

* chore(ci): rerun regression gate [skip-regression-check]

(cherry picked from commit 784d444)

---------

Co-authored-by: Umesh Kumar Singh <brijbiharisingh1971@outlook.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
drchirag1991 pushed a commit to drchirag1991/ironclaw that referenced this pull request Apr 8, 2026
…earai#1003)

* fix: release lock guards before awaiting channel send (nearai#869)

Clone `mpsc::Sender` out of `RwLock` before `.send().await` to prevent
read guards from blocking write lock acquisition (shutdown/start) when
the channel buffer is full.

Fixed call sites:
- src/channels/http.rs: process_message()
- src/channels/web/server.rs: chat_send_handler(), chat_approval_handler()
- src/channels/web/handlers/chat.rs: chat_send_handler(), chat_approval_handler()
- src/channels/web/ws.rs: handle_client_message() (2 sites)
- src/channels/wasm/wrapper.rs: process_emitted_messages() (2 impls, also
  scoped rate_limiter write lock per-iteration)

Includes regression test: shutdown_completes_while_process_message_blocked

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
(cherry picked from commit 84802e1)

* ci: fetch base branch before regression test check

The regression-test-check workflow failed because origin/main wasn't
available as a ref in the CI environment. actions/checkout@v4 fetches
the PR merge ref history but doesn't make the base branch ref available
for three-dot diff comparisons.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
(cherry picked from commit 1d5a7bd)

* chore(ci): rerun regression gate [skip-regression-check]

(cherry picked from commit 784d444)

---------

Co-authored-by: Umesh Kumar Singh <brijbiharisingh1971@outlook.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

contributor: core 20+ merged PRs risk: medium Business logic, config, or moderate-risk modules scope: channel/wasm WASM channel runtime scope: channel/web Web gateway channel scope: ci CI/CD workflows size: L 200-499 changed lines skip-regression-check Bypass regression test CI gate (tests exist but not in tests/ dir)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants