Sentry stopped accepting transaction data

[ingria]

Self-Hosted Version

24.3.0.dev0

CPU Architecture

x86_x64

Docker Version

24.0.4

Docker Compose Version

24.0.4

Steps to Reproduce

Update to the latest master

Expected Result

Everything works fine

Actual Result

Performance page shows zeros for the time period since the update and until now:

Project page shows the correct info about transactions and errors:

Stats page shows 49k transactions of which 49k are dropped:

Same for errors:

Event ID

No response

UPD

there are a lot of errors in clickhouse container:

2024.03.10 23:40:34.789282 [ 46 ] {} <Error> ServerErrorHandler: Poco::Exception. Code: 1000, e.code() = 107, e.displayText() = Net Exception: Socket is not connected, Stack trace (when copying this message, always include the lines below):

0. Poco::Net::SocketImpl::error(int, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) @ 0x13c4ee8e in /usr/bin/clickhouse
1. Poco::Net::SocketImpl::peerAddress() @ 0x13c510d6 in /usr/bin/clickhouse
2. DB::ReadBufferFromPocoSocket::ReadBufferFromPocoSocket(Poco::Net::Socket&, unsigned long) @ 0x101540cd in /usr/bin/clickhouse
3. DB::HTTPServerRequest::HTTPServerRequest(std::__1::shared_ptr<DB::Context const>, DB::HTTPServerResponse&, Poco::Net::HTTPServerSession&) @ 0x110e6fd5 in /usr/bin/clickhouse
4. DB::HTTPServerConnection::run() @ 0x110e5d6e in /usr/bin/clickhouse
5. Poco::Net::TCPServerConnection::start() @ 0x13c5614f in /usr/bin/clickhouse
6. Poco::Net::TCPServerDispatcher::run() @ 0x13c57bda in /usr/bin/clickhouse
7. Poco::PooledThread::run() @ 0x13d89e59 in /usr/bin/clickhouse
8. Poco::ThreadImpl::runnableEntry(void*) @ 0x13d860ea in /usr/bin/clickhouse
9. start_thread @ 0x9609 in /usr/lib/x86_64-linux-gnu/libpthread-2.31.so
10. clone @ 0x122293 in /usr/lib/x86_64-linux-gnu/libc-2.31.so
 (version 21.8.13.1.altinitystable (altinity build))

[ingria]

Also, for some reason Sentry started dropping incoming errors some time ago (as if I was using saas sentry):

[barisyild]

Did you change the port?
I had the same situation when I changed the port.

[ingria]

Yes, I have the relay port exposed to the host network. How did you manage to fix the problem?

[barisyild]

Yes, I have the relay port exposed to the host network. How did you manage to fix the problem?

When I reverted the port change the problem was resolved.

[ingria]

Nope, didn't help. Doesn't work even with default config. Thanks for the tip though

[hubertdeng123]

Are there any logs in your web container that can help? Are you sure you are receiving the event envelopes? You should be able to see that activity in your nginx container.

[linxiaowang]

Same here, on the browser side, there is a request sent with an event type of "transaction", but there is no data displayed under "performance", and the number of transactions in the project is also 0.

[linxiaowang]

Same here, on the browser side, there is a request sent with an event type of "transaction", but there is no data displayed under "performance", and the number of transactions in the project is also 0.

Problem solved, server time not match the sdk time.

[ingria]

I can see that there are successful requests to /api/2/envelope:

Also I can see transaction statistics on the projects page:

Number 394k for the last 24 hours is about right.

[hubertdeng123]

Are you on a nightly version of self-hosted? What does your sentry.conf.py look like? We've added some feature flags there to support the new performance features

[ingria]

I'm using docker with the latest commit from this repository. Bottom of the page says Sentry 24.3.0.dev0 unknown. So I guess that's nightly.

I've updated sentry.conf.py to match the most recent version from this repo - now the only difference is in SENTRY_SINGLE_ORGANIZATION and CSRF_TRUSTED_ORIGINS variables.

After that, errors have also disappeared:

[williamdes]

I can confirm that the clickhouse errors are due to the Rust workers, reverting the workers part of #2831 and #2861
made the errors disappear.
But still I have a too high dropping of transactions since the upgrade.

Worker code: https://github.com/getsentry/snuba/blob/359878fbe030a63945914ef05e705224680b453c/rust_snuba/src/strategies/clickhouse.rs#L61

Workers logs show that insert is done (is it ?): "timestamp":"2024-03-16T11:40:52.491448Z","level":"INFO","fields":{"message":"Inserted 29 rows"},

[aldy505]

The error is caused by connection being prematurely closed. See #2900

[LvckyAPI]

Same issue on latest 24.3.0

[LvckyAPI]

errors are not logged to

[aldy505]

Okay so I'm able to replicate this issue on my instance (24.3.0). What happen is that Sentry does accept transaction/errors/profiles/replays/attachments data, but it doesn't record it on the statistics. So your stats of ingested events might be displayed as is there were no events being recorded, but actually the events are there -- it's processed by Snuba and you can view it on the web UI.

Can anyone reading this confirm that that's what happened on your instances as well? (I don't want to ping everybody)

If the answer to that 👆🏻 is "yes", that means something (a module, container, or something) that ingest the events didn't do data insertion correctly for it to be queried as statistics. I don't know for sure whether it's the responsibility of Snuba consumers (as we moved to rust-consumer just on 24.3.0) or Sentry workers, but I'd assume it's Snuba consumers.

A few solution (well not really but I hope this would get rid of this issue) for this is, either:

1. Fix the issue somewhere, cut a patch release.
2. If it's caused by rust-consumers, then we might rollback the usage of rust-consumer and just go back to old Python ones.

[LvckyAPI]

Okay so I'm able to replicate this issue on my instance (24.3.0). What happen is that Sentry does accept transaction/errors/profiles/replays/attachments data, but it doesn't record it on the statistics. So your stats of ingested events might be displayed as is there were no events being recorded, but actually the events are there -- it's processed by Snuba and you can view it on the web UI.

Can anyone reading this confirm that that's what happened on your instances as well? (I don't want to ping everybody)

If the answer to that 👆🏻 is "yes", that means something (a module, container, or something) that ingest the events didn't do data insertion correctly for it to be queried as statistics. I don't know for sure whether it's the responsibility of Snuba consumers (as we moved to rust-consumer just on 24.3.0) or Sentry workers, but I'd assume it's Snuba consumers.

A few solution (well not really but I hope this would get rid of this issue) for this is, either:

1. Fix the issue somewhere, cut a patch release.
2. If it's caused by rust-consumers, then we might rollback the usage of rust-consumer and just go back to old Python ones.

I didn't see any errors in the Issues tab. I had to rebuild a Server Snapshot to “fix” this problem. So it wasn't just the statistics that were affected.

[williamdes]

Feel free to join my Discord thread: https://discord.com/channels/621778831602221064/1243470946904445008
Discord join link: https://discord.gg/TvshEMuG
For now I found a solution that works pretty well.
When the node starts complaining about the config I run the following: docker exec -it sentry-self-hosted-redis-1 sh -c 'redis-cli DEL relay_config'

Then do a docker compose down and docker compose up.
Please let me know if this works.

[liukch]

I think this PR #2908 would fix this issue，I don't know why it is still not be merged for a long time.

[williamdes]

I think this PR #2908 would fix this issue，I don't know why it is still not be merged for a long time.

Reverting to old Python code is not a solution. Fix the Rust code is a good one.

[rojinebrahimi]

Hey everyone!
I reverted the consumers to python but unfortunately I am still not able to see my transactions. Besides, I have come across some issues on ClickHouse:

2024.07.07 11:11:58.974263 [ 19750 ] {7760523f29425408e575ceb5fbd61469} <Error> TCPHandler: Code: 46. DB::Exception: Unknown function notHandled: While processing ((client_timestamp AS _snuba_timestamp) >= toDateTime('2024-06-27T11:11:58', 'Universal')) AND (_snuba_timestamp < toDateTime('2024-07-07T11:11:55', 'Universal')) AND ((project_id AS _snuba_project_id) IN tuple(17)) AND (notHandled() = 1) AND ((occurrence_type_id AS _snuba_occurrence_type_id) IN (4001, 4002, 4003)) AND (_snuba_project_id IN tuple(17)) AND ((group_id AS _snuba_group_id) IN (187756,...

Does anyone possibly know the solution or the root cause of this problem?

[hheexx]

I have this problem again after updating to 24.8.0!

[hheexx]

---except that reverting to python snuba does not work. No more errors but still does not work.

[khassad]

I have this problem again after updating to 24.8.0!

Same behavior here, transactions showed up partially (some data in stats area) but not in global projects pages or individual project page 😢

[DarkByteZero]

I had issues with stopped ingestion, but my issue was that I didn't have COMPOSE_PROFILES=feature-complete in my custom env

[hubertdeng123]

I had issues with stopped ingestion, but my issue was that I didn't have COMPOSE_PROFILES=feature-complete in my custom env

Ah yeah, that'll do it. Without that you'll only be ingesting errors.

[liukch]

This issue has been present for several months and remains unresolved.
Do we have a schedule to fix this issue?
Due to this issue, we are experiencing significant difficulties with the upgrade of our self-hosted Sentry version.
@hubertdeng123

[aldy505]

---except that reverting to python snuba does not work. No more errors but still does not work.

@hheexx I helped someone on Discord a few days ago, both regular snuba consumer and snuba rust-consumer didn't work for him. He tried upgrading their server instance to a higher spec (previously 4 cores CPU + 16 GB RAM [AWS EC2 m6a.xlarge] --> 8 cores CPU + 32 GB RAM [AWS EC2 m6a.2xlarge]). See the Discord thread here: https://discord.com/channels/621778831602221064/1286099840480182272

Obviously I know bumping their server spec is not for everyone, even my initial hunch was on the IO/s (or IOps) limit.

[hheexx]

thanks @aldy505, you are right. I fixed it by moving msl to seperate ssd storage (vm is on hdd)

[klemen-df]

Same issues, nothing helps :(

[Mordreak]

Same issue here, on fresh install and latest commit on master, I do get a 200 response with ID of the transaction, but nothing shows in the performance tab panel.
Tried several proposed solutions without any luck