Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: resolve flight connection leak #13866

Merged
merged 3 commits into from
Dec 1, 2023

Conversation

dantengsky
Copy link
Member

@dantengsky dantengsky commented Nov 30, 2023

I hereby agree to the terms of the CLA available at: https://databend.rs/dev/policies/cla/

Summary

Execution of distributed tasks may result in a leak of Flight TCP connections.


How to reproduce:

  • 3 node cluster (local fs storage)

  • set up test data create table t as select * from numbers(10000000)

  • execute the following script several times (typically, 10 times is sufficent)

bendsql  --dsn "databend://root:@localhost:8000/?sslmode=disable&enable_distributed_replace_into=1" \
         --query="replace into t on(number) select * from t"

turns out the primary reason for the Flight RPC connection leak is the mismatch between runtimes:

the runtime used to create the FlightClient and the one used to drop it are not the same.

Note: I cannot confirm that this completely resolves all issues related to Flight connection leaks.


This PR that havs been tested through the following scenarios:

These tests show that the number of open file descriptors is under controlled.

  • Closes #issue

This change is Reviewable

@github-actions github-actions bot added the pr-bugfix this PR patches a bug in codebase label Nov 30, 2023
@dantengsky dantengsky force-pushed the fix-filght-client-leak branch from 49573ec to b1ee5af Compare November 30, 2023 04:59
@dantengsky dantengsky changed the title fix: resolve flight client leak when killing in-progress tasks with 'kill query' fix: resolve QueryCoordinator leak when killing in-progress tasks with 'kill query' Nov 30, 2023
@dantengsky dantengsky closed this Nov 30, 2023
@dantengsky dantengsky changed the title fix: resolve QueryCoordinator leak when killing in-progress tasks with 'kill query' fix: resolve flight connection leak Nov 30, 2023
@dantengsky dantengsky reopened this Nov 30, 2023
@dantengsky dantengsky added the ci-benchmark Benchmark: run all test label Nov 30, 2023
Copy link
Contributor

Docker Image for PR

  • tag: pr-13866-e4c3c76

note: this image tag is only available for internal use,
please check the internal doc for more details.

Copy link
Contributor

@dantengsky dantengsky marked this pull request as ready for review December 1, 2023 04:11
@dantengsky
Copy link
Member Author

@zhang2014 PTAL, thanks

@dantengsky dantengsky added this pull request to the merge queue Dec 1, 2023
@dantengsky dantengsky removed this pull request from the merge queue due to a manual request Dec 1, 2023
@BohuTANG BohuTANG merged commit bad39b9 into databendlabs:main Dec 1, 2023
80 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
ci-benchmark Benchmark: run all test pr-bugfix this PR patches a bug in codebase
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants