Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix https_batch deadlock due to golang timer changes #648

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

nicklas-dohrn
Copy link
Contributor

@nicklas-dohrn nicklas-dohrn commented Dec 13, 2024

What is changed:

This changes the way the syslog batches are triggered.
The new implementation no longer uses Timers that need to be reset and checked, it just ticks once every second per http_batch drain to send a batch at least once a second.
The new logic is as follows:

  • If there is a log present in the queue, send it at last one second later with every log from the same second.
  • If there are more logs in one second than the 'batchSize' allows, send the batch immediately

The problem fixed by this change:

The old implementation was able to deadlock itself due to the way the time channel was reset. There are even more changes to this behaviour to be expected by the golang 1.23 update. (https://tip.golang.org/doc/go1.23#timer-changes)

Impact:

The new implementation will tick more often, but the overhead will be pretty minimal, due to not doing anything for empty drains.

Type of change

  • Bug fix (non-breaking change which fixes an issue)

Testing performed?

  • Unit tests
  • Tested in dev cluster

Checklist:

  • This PR is being made against the main branch, or relevant version branch
  • I have made corresponding changes to the documentation
  • I have added testing for my changes

@nicklas-dohrn nicklas-dohrn requested a review from a team as a code owner December 13, 2024 13:08
@nicklas-dohrn nicklas-dohrn force-pushed the main branch 2 times, most recently from f2dac59 to 5e8242f Compare December 13, 2024 13:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Development

Successfully merging this pull request may close these issues.

1 participant