Skip to content

Conversation

@michaelsproul
Copy link
Member

Issue Addressed

Our Holesky nodes running with the light client enabled were logging messages about full queues:

Feb 12 22:09:28.949 ERRO Work queue is full queue: unknown_light_client_optimistic_update, queue_len: 128, msg: the system has insufficient resources for load, service: bproc

I thought this might be genuine overload, but it turns out this queue was never being read from!

Proposed Changes

  • Rename light-client related queues in the beacon processor for clarity.
  • Ensure all light-client related queues are being popped from.

Additional Info

I noticed that the sampling_result_queue is never being read from either, but I've added a TODO(das) to address that in future.

@michaelsproul michaelsproul added bug Something isn't working light-client v7.0.0-beta.0 New release c. Q1 2025 labels Feb 12, 2025
Copy link
Member

@jimmygchen jimmygchen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch, LGTM!

@jimmygchen jimmygchen added the ready-for-merge This PR is ready to merge. label Feb 13, 2025
@michaelsproul
Copy link
Member Author

For the record: the impact of the bug isn't that we never processing light client updates from gossip. It's that we never process them once queued. It takes a while for the queue to build up, because if the processor is not at capacity, we will still schedule and run processing immediately here:

match work {
_ if can_spawn => self.spawn_worker(work, idle_tx),

@michaelsproul
Copy link
Member Author

@mergify queue

@mergify
Copy link

mergify bot commented Feb 13, 2025

queue

✅ The pull request has been merged automatically

Details

The pull request has been merged automatically at 25f804a

@mergify mergify bot merged commit 25f804a into sigp:release-v7.0.0-beta.0 Feb 13, 2025
30 checks passed
@michaelsproul michaelsproul deleted the light-client-plumbing branch February 13, 2025 01:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working light-client ready-for-merge This PR is ready to merge. v7.0.0-beta.0 New release c. Q1 2025

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants