ClickHouse Socket is not connected

[zKoz210]

Self-Hosted Version

24.3.0

CPU Architecture

x86_64

Docker Version

Docker version 24.0.7, build afdd53b

Docker Compose Version

Docker Compose version v2.21.0

Steps to Reproduce

Update 24.2.0 to 24.3.0

docker compose logs clickhouse -f -n 200

Expected Result

ClickHouse work

Actual Result

sentry-self-hosted-clickhouse-1  | 2024.03.16 19:00:59.837194 [ 46 ] {} <Error> ServerErrorHandler: Poco::Exception. Code: 1000, e.code() = 107, e.displayText() = Net Exception: Socket is not connected, Stack trace (when copying this message, always include the lines below):
sentry-self-hosted-clickhouse-1  | 
sentry-self-hosted-clickhouse-1  | 0. Poco::Net::SocketImpl::error(int, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) @ 0x13c4ee8e in /usr/bin/clickhouse
sentry-self-hosted-clickhouse-1  | 1. Poco::Net::SocketImpl::peerAddress() @ 0x13c510d6 in /usr/bin/clickhouse
sentry-self-hosted-clickhouse-1  | 2. DB::HTTPServerRequest::HTTPServerRequest(std::__1::shared_ptr<DB::Context const>, DB::HTTPServerResponse&, Poco::Net::HTTPServerSession&) @ 0x110e6f0b in /usr/bin/clickhouse
sentry-self-hosted-clickhouse-1  | 3. DB::HTTPServerConnection::run() @ 0x110e5d6e in /usr/bin/clickhouse
sentry-self-hosted-clickhouse-1  | 4. Poco::Net::TCPServerConnection::start() @ 0x13c5614f in /usr/bin/clickhouse
sentry-self-hosted-clickhouse-1  | 5. Poco::Net::TCPServerDispatcher::run() @ 0x13c57bda in /usr/bin/clickhouse
sentry-self-hosted-clickhouse-1  | 6. Poco::PooledThread::run() @ 0x13d89e59 in /usr/bin/clickhouse
sentry-self-hosted-clickhouse-1  | 7. Poco::ThreadImpl::runnableEntry(void*) @ 0x13d860ea in /usr/bin/clickhouse
sentry-self-hosted-clickhouse-1  | 8. start_thread @ 0x9609 in /usr/lib/x86_64-linux-gnu/libpthread-2.31.so
sentry-self-hosted-clickhouse-1  | 9. clone @ 0x122293 in /usr/lib/x86_64-linux-gnu/libc-2.31.so
sentry-self-hosted-clickhouse-1  |  (version 21.8.13.1.altinitystable (altinity build))

Event ID

No response

[zKoz210]

sentry-self-hosted-snuba-generic-metrics-distributions-consumer-1  | {"timestamp":"2024-03-16T19:03:06.499325Z","level":"INFO","fields":{"message":"Inserted 1431 rows"},"target":"rust_snuba::strategies::clickhouse"}
sentry-self-hosted-snuba-spans-consumer-1                          | {"timestamp":"2024-03-16T19:03:06.549370Z","level":"INFO","fields":{"message":"Inserted 387 rows"},"target":"rust_snuba::strategies::clickhouse"}
sentry-self-hosted-snuba-transactions-consumer-1                   | {"timestamp":"2024-03-16T19:03:06.954095Z","level":"INFO","fields":{"message":"Inserted 12 rows"},"target":"rust_snuba::strategies::clickhouse"}
sentry-self-hosted-snuba-generic-metrics-distributions-consumer-1  | {"timestamp":"2024-03-16T19:03:07.372832Z","level":"INFO","fields":{"message":"Inserted 2 rows"},"target":"rust_snuba::strategies::clickhouse"}
sentry-self-hosted-snuba-spans-consumer-1                          | {"timestamp":"2024-03-16T19:03:07.582966Z","level":"INFO","fields":{"message":"Inserted 455 rows"},"target":"rust_snuba::strategies::clickhouse"}
sentry-self-hosted-snuba-transactions-consumer-1                   | {"timestamp":"2024-03-16T19:03:07.702908Z","level":"INFO","fields":{"message":"Inserted 12 rows"},"target":"rust_snuba::strategies::clickhouse"}
sentry-self-hosted-worker-1                                        | 19:03:08 [INFO] sentry.tasks.post_process: handle_owner_assignment.issue_owners_exist (group=28569 project=53 organization=1 reason='issue_owners_exist')
sentry-self-hosted-worker-1                                        | 19:03:08 [INFO] sentry.models.projectownership: handle_auto_assignment.no_issue_owner (event_id='504e30e18a034df187ebfad6642cc2ef' group_id='28569' project_id='53' organization_id=1 source='post_process' ownership=<ProjectOwnership at 0x7f525e7adf10: id=None, project_id=53, is_active=True> autoassignment_types=[1, 2])
sentry-self-hosted-snuba-transactions-consumer-1                   | {"timestamp":"2024-03-16T19:03:08.453861Z","level":"INFO","fields":{"message":"Inserted 6 rows"},"target":"rust_snuba::strategies::clickhouse"}
sentry-self-hosted-clickhouse-1                                    | 2024.03.16 19:03:08.454539 [ 503 ] {} <Error> ServerErrorHandler: Poco::Exception. Code: 1000, e.code() = 107, e.displayText() = Net Exception: Socket is not connected, Stack trace (when copying this message, always include the lines below):
sentry-self-hosted-clickhouse-1                                    | 
sentry-self-hosted-clickhouse-1                                    | 0. Poco::Net::SocketImpl::error(int, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) @ 0x13c4ee8e in /usr/bin/clickhouse
sentry-self-hosted-clickhouse-1                                    | 1. Poco::Net::SocketImpl::peerAddress() @ 0x13c510d6 in /usr/bin/clickhouse
sentry-self-hosted-clickhouse-1                                    | 2. DB::HTTPServerRequest::HTTPServerRequest(std::__1::shared_ptr<DB::Context const>, DB::HTTPServerResponse&, Poco::Net::HTTPServerSession&) @ 0x110e6f0b in /usr/bin/clickhouse
sentry-self-hosted-clickhouse-1                                    | 3. DB::HTTPServerConnection::run() @ 0x110e5d6e in /usr/bin/clickhouse
sentry-self-hosted-clickhouse-1                                    | 4. Poco::Net::TCPServerConnection::start() @ 0x13c5614f in /usr/bin/clickhouse
sentry-self-hosted-clickhouse-1                                    | 5. Poco::Net::TCPServerDispatcher::run() @ 0x13c57bda in /usr/bin/clickhouse
sentry-self-hosted-clickhouse-1                                    | 6. Poco::PooledThread::run() @ 0x13d89e59 in /usr/bin/clickhouse
sentry-self-hosted-clickhouse-1                                    | 7. Poco::ThreadImpl::runnableEntry(void*) @ 0x13d860ea in /usr/bin/clickhouse
sentry-self-hosted-clickhouse-1                                    | 8. start_thread @ 0x9609 in /usr/lib/x86_64-linux-gnu/libpthread-2.31.so
sentry-self-hosted-clickhouse-1                                    | 9. clone @ 0x122293 in /usr/lib/x86_64-linux-gnu/libc-2.31.so
sentry-self-hosted-clickhouse-1                                    |  (version 21.8.13.1.altinitystable (altinity build))
sentry-self-hosted-snuba-spans-consumer-1                          | {"timestamp":"2024-03-16T19:03:08.695734Z","level":"INFO","fields":{"message":"Inserted 928 rows"},"target":"rust_snuba::strategies::clickhouse"}
sentry-self-hosted-web-1                                           | 19:03:09 [INFO] sentry.access.api: api.access (method='POST' view='sentry.api.endpoints.relay.project_configs.RelayProjectConfigsEndpoint' response=200 user_id='None' is_app='None' token_type='None' is_frontend_request='False' organization_id='None' auth_id='None' path='/api/0/relays/projectconfigs/' caller_ip='192.168.176.51' user_agent='None' rate_limited='False' rate_limit_category='None' request_duration_seconds=0.12395715713500977 rate_limit_type='DNE' concurrent_limit='None' concurrent_requests='None' reset_time='None' group='None' limit='None' remaining='None')
sentry-self-hosted-snuba-transactions-consumer-1                   | {"timestamp":"2024-03-16T19:03:09.210059Z","level":"INFO","fields":{"message":"Inserted 16 rows"},"target":"rust_snuba::strategies::clickhouse"}
sentry-self-hosted-clickhouse-1                                    | 2024.03.16 19:03:09.210605 [ 511 ] {} <Error> ServerErrorHandler: Poco::Exception. Code: 1000, e.code() = 107, e.displayText() = Net Exception: Socket is not connected, Stack trace (when copying this message, always include the lines below):
sentry-self-hosted-clickhouse-1                                    | 
sentry-self-hosted-clickhouse-1                                    | 0. Poco::Net::SocketImpl::error(int, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) @ 0x13c4ee8e in /usr/bin/clickhouse
sentry-self-hosted-clickhouse-1                                    | 1. Poco::Net::SocketImpl::peerAddress() @ 0x13c510d6 in /usr/bin/clickhouse
sentry-self-hosted-clickhouse-1                                    | 2. DB::HTTPServerRequest::HTTPServerRequest(std::__1::shared_ptr<DB::Context const>, DB::HTTPServerResponse&, Poco::Net::HTTPServerSession&) @ 0x110e6f0b in /usr/bin/clickhouse
sentry-self-hosted-clickhouse-1                                    | 3. DB::HTTPServerConnection::run() @ 0x110e5d6e in /usr/bin/clickhouse
sentry-self-hosted-clickhouse-1                                    | 4. Poco::Net::TCPServerConnection::start() @ 0x13c5614f in /usr/bin/clickhouse
sentry-self-hosted-clickhouse-1                                    | 5. Poco::Net::TCPServerDispatcher::run() @ 0x13c57bda in /usr/bin/clickhouse
sentry-self-hosted-clickhouse-1                                    | 6. Poco::PooledThread::run() @ 0x13d89e59 in /usr/bin/clickhouse
sentry-self-hosted-clickhouse-1                                    | 7. Poco::ThreadImpl::runnableEntry(void*) @ 0x13d860ea in /usr/bin/clickhouse
sentry-self-hosted-clickhouse-1                                    | 8. start_thread @ 0x9609 in /usr/lib/x86_64-linux-gnu/libpthread-2.31.so
sentry-self-hosted-clickhouse-1                                    | 9. clone @ 0x122293 in /usr/lib/x86_64-linux-gnu/libc-2.31.so
sentry-self-hosted-clickhouse-1                                    |  (version 21.8.13.1.altinitystable (altinity build))

[aldy505]

This shouldn't be an issue. Based on ClickHouse/ClickHouse#23059 (comment) and ClickHouse/ClickHouse#27297 (comment), the client (in this case snuba) has closed the connection prematurely.

Since this is a snuba issue, would you guys want to transfer this to the snuba repo? @untitaker @lynnagara

[aldy505]

I'm gonna close this one in favor of #2876