All Events tab doesn't work for issue with lots of events

[stefangr]

Self-Hosted Version

23.1.0

CPU Architecture

x86_64

Docker Version

20.10.23

Docker Compose Version

2.15.1

Steps to Reproduce

1. Have an issue with more than ~50 individual events
2. Open the "All events" tab

Expected Result

The all events page shows the list of all events paginated

Actual Result

The browser gets a HTTP 502 Bad Gateway returned.

The spinner animation on the "All events" tab is permanent.

Relevant nginx log output:

sentry-self-hosted-nginx-1                                     | 2023/01/20 07:28:12 [error] 27#27: *2985 upstream sent too big header while reading response header from upstream, client: yy.yy.yy.yy, server: , request: "GET /api/0/issues/13177/attachments/?per_page=50&types=event.minidump&event_id=d58d7624c3214558810510c8ec0e4e8c&event_id=241c7e629ba74abcae9bd4374095eae1&event_id=f77a3d03635e467688f10fc744395270&event_id=75d53c47b96c4194a1a6f6ba67f0f192&event_id=11784f253ffc41c6a656b1e47a06262d&event_id=9c2c886a05374dfabf7c0c58c97e661c&event_id=8719783fe3d74a2aa48587f705170cbc&event_id=bdf25f593208444d89934f6963a9b62a&event_id=8369b6c2408b42ae95f5b082de6dca0b&event_id=f4674866da5e43bbad4dc3003ece4124&event_id=4c50f2863cfc42aaa1d8ff4e4a7bc364&event_id=8d5bf7e16a72496b9fc7c2fa118b13b9&event_id=3c7b52123f284ea8806ab65b4c8e10e4&event_id=53df552adf254d47bd3f727e7d657342&event_id=d6bc29cb49004ce19ecd3e24a409dee1&event_id=a00b60bbb9d64634bd86944060058991&event_id=3e699fe3bf03439c9ed409aeb7a299ac&event_id=d0799b9456be46c59a8cd8ab443fff8b&event_id=59f5efa1af74495d8ac3ec767bd4d4b7&event_id=86d1528f022a4a9eae0ef7b825d09be7&event_id=76e5878ff51249be9230b888f1b70756&event_id=02279bb8966d4fb4857b2f59e9ebc08f&event_id=e1cd1684eeae48fcb7d15f15b4b2e3db&event_id=3542672ce48a401abb05740a54c8cfad&event_id=d18b40a3c3f740f798823ef8228a7eb5&event_id=8c43f7e36730440394f777dd834c09d8&event_id=cb7ed9d24b1645bdbf067f9814520dba&event_id=4d1350f02fdf4dd79dbe05f67a412ffe&event_id=c085713affa347f2ac5a740f6a80e46d&event_id=2f4ffc32960e4e6aa74961faea5eaf0b&event_id=90b1ba37848546948872c20bb475abd1&event_id=fe094d5a8197413c8bbc777140017fd4&event_id=eac8d43f1ab244838a00e2c85cda59a8&event_id=799c86012448496b9624db39e5d37a3f&event_id=a256fc31028245f5b6f1176b698fcc09&event_id=15e97942653a4ffe8d69cee68a4aedfc&event_id=7557606f2af94905a4b97e60b33b0e44&event_id=3bbc4b4346214bd3a9470166c923b429&event_id=1870a844883148b58d5ce2a922e379b9&event_id=1001faab384140ad90a8897186532295&event_id=6154f30380fd460292d205b6d44068a4&event_id=d9c514620d36413abdaacebbe7915444&event_id=18ac58cab74945a0b0beb1a70c71d55c&event_id=ffcd
sentry-self-hosted-nginx-1                                     | yy.yy.yy.yy - - [20/Jan/2023:07:28:12 +0000] "GET /api/0/issues/13177/attachments/?per_page=50&types=event.minidump&event_id=d58d7624c3214558810510c8ec0e4e8c&event_id=241c7e629ba74abcae9bd4374095eae1&event_id=f77a3d03635e467688f10fc744395270&event_id=75d53c47b96c4194a1a6f6ba67f0f192&event_id=11784f253ffc41c6a656b1e47a06262d&event_id=9c2c886a05374dfabf7c0c58c97e661c&event_id=8719783fe3d74a2aa48587f705170cbc&event_id=bdf25f593208444d89934f6963a9b62a&event_id=8369b6c2408b42ae95f5b082de6dca0b&event_id=f4674866da5e43bbad4dc3003ece4124&event_id=4c50f2863cfc42aaa1d8ff4e4a7bc364&event_id=8d5bf7e16a72496b9fc7c2fa118b13b9&event_id=3c7b52123f284ea8806ab65b4c8e10e4&event_id=53df552adf254d47bd3f727e7d657342&event_id=d6bc29cb49004ce19ecd3e24a409dee1&event_id=a00b60bbb9d64634bd86944060058991&event_id=3e699fe3bf03439c9ed409aeb7a299ac&event_id=d0799b9456be46c59a8cd8ab443fff8b&event_id=59f5efa1af74495d8ac3ec767bd4d4b7&event_id=86d1528f022a4a9eae0ef7b825d09be7&event_id=76e5878ff51249be9230b888f1b70756&event_id=02279bb8966d4fb4857b2f59e9ebc08f&event_id=e1cd1684eeae48fcb7d15f15b4b2e3db&event_id=3542672ce48a401abb05740a54c8cfad&event_id=d18b40a3c3f740f798823ef8228a7eb5&event_id=8c43f7e36730440394f777dd834c09d8&event_id=cb7ed9d24b1645bdbf067f9814520dba&event_id=4d1350f02fdf4dd79dbe05f67a412ffe&event_id=c085713affa347f2ac5a740f6a80e46d&event_id=2f4ffc32960e4e6aa74961faea5eaf0b&event_id=90b1ba37848546948872c20bb475abd1&event_id=fe094d5a8197413c8bbc777140017fd4&event_id=eac8d43f1ab244838a00e2c85cda59a8&event_id=799c86012448496b9624db39e5d37a3f&event_id=a256fc31028245f5b6f1176b698fcc09&event_id=15e97942653a4ffe8d69cee68a4aedfc&event_id=7557606f2af94905a4b97e60b33b0e44&event_id=3bbc4b4346214bd3a9470166c923b429&event_id=1870a844883148b58d5ce2a922e379b9&event_id=1001faab384140ad90a8897186532295&event_id=6154f30380fd460292d205b6d44068a4&event_id=d9c514620d36413abdaacebbe7915444&event_id=18ac58cab74945a0b0beb1a70c71d55c&event_id=ffcdbd5a18704d6aa492912e93abb147&event_id=263e3b9e9b7b41cabadda69f485e36d2&event_id=4765139fbd7f4129891ce64d3eba4567&event_id=b4a595b71a4249c09bfdad7a4d0d053b&event_id=177a622c76b44971b2048807bdabffa8&event_id=13f9622d6f654194875eeb7be3082168&event_id=7be914ef1be9455ea27c3043e3ed7448 HTTP/1.0" 502 150 "https://sentry.ourdomain.ext/organizations/organization-name/issues/13177/events/?project=3&referrer=issue-stream&sort=freq&statsPeriod=14d" "Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Firefox/102.0" "xx.xx.xx.xx"

Event ID

No response

[hubertdeng123]

This seems like it can be fixed by tweaking the nginx config. See here
https://www.cyberciti.biz/faq/nginx-upstream-sent-too-big-header-while-reading-response-header-from-upstream/

[stefangr]

Thanks for that. I tried it and the problem is persistent even with larger buffer sizes.

This call just worked in version 22.12.0 without having to change the nginx configuration.

[cocorossello]

Yes, same here, it was working fine. Changing the buffer I can clear the nginx error but the events tab does not load

[jllado]

Same here 🙏

[hubertdeng123]

Ah, that's good to know. So 23.1.0 potentially broke this. @cocorossello @jllado are you both on 23.1.0?

[cocorossello]

Ah, that's good to know. So 23.1.0 potentially broke this. @cocorossello @jllado are you both on 23.1.0?

Yes, when I moved from 22.12.0 to 23.1.0 the problem started

[hubertdeng123]

Would any of you be able to post web/relay logs that might be helpful in determining what the issue might be after resolving the nginx error? I'm wondering if #1929 might be related

[cocorossello]

The only error present in the logs is the nginx one. I've not been able to remove it, I'll try to see if I can workaround the nginx error

Attaching to sentry_onpremise_nginx_1, sentry_onpremise_relay_1, sentry_onpremise_sentry-cleanup_1, sentry_onpremise_web_1, sentry_onpremise_post-process-forwarder-errors_1, sentry_onpremise_post-process-forwarder-transactions_1, sentry_onpremise_cron_1, sentry_onpremise_ingest-consumer_1, sentry_onpremise_subscription-consumer-transactions_1, sentry_onpremise_worker_1, sentry_onpremise_subscription-consumer-events_1, sentry_onpremise_snuba-replacer_1, sentry_onpremise_snuba-subscription-consumer-events_1, sentry_onpremise_snuba-sessions-consumer_1, sentry_onpremise_snuba-cleanup_1, sentry_onpremise_snuba-consumer_1, sentry_onpremise_snuba-subscription-consumer-transactions_1, sentry_onpremise_snuba-api_1, sentry_onpremise_snuba-transactions-cleanup_1, sentry_onpremise_snuba-outcomes-consumer_1, sentry_onpremise_snuba-transactions-consumer_1, sentry_onpremise_clickhouse_1, sentry_onpremise_symbolicator-cleanup_1, sentry_onpremise_smtp_1, sentry_onpremise_symbolicator_1, sentry_onpremise_memcached_1, sentry_onpremise_postgres_1, sentry_onpremise_kafka_1, sentry_onpremise_zookeeper_1, sentry_onpremise_redis_1
clickhouse_1                                | 2023-01-25T09:00:40.170350386Z 2023.01.25 09:00:40.170113 [ 241 ] {f8bf6abd-6f3e-4f83-b615-446616ea9395} <Information> executeQuery: Read 2 rows, 124.00 B in 0.006 sec., 313 rows/sec., 18.98 KiB/sec.
clickhouse_1                                | 2023-01-25T09:00:40.170531990Z 2023.01.25 09:00:40.170432 [ 241 ] {} <Information> HTTPHandler: Done processing query
snuba-outcomes-consumer_1                   | 2023-01-25T09:00:40.171599614Z 2023-01-25 09:00:40,171 Completed processing <Batch: 2 messages, open for 1.01 seconds>.
clickhouse_1                                | 2023-01-25T09:00:40.173083048Z 2023.01.25 09:00:40.172986 [ 120 ] {ff501bdc-3957-4f9c-ae3f-6c1b042454ec} <Information> executeQuery: Read 2 rows, 5.66 KiB in 0.009 sec., 228 rows/sec., 646.14 KiB/sec.
clickhouse_1                                | 2023-01-25T09:00:40.173766263Z 2023.01.25 09:00:40.173690 [ 120 ] {} <Information> HTTPHandler: Done processing query
snuba-consumer_1                            | 2023-01-25T09:00:40.175175695Z 2023-01-25 09:00:40,174 Completed processing <Batch: 2 messages, open for 1.02 seconds>.
web_1                                       | 2023-01-25T09:00:40.293406769Z 09:00:40 [INFO] sentry.access.api: api.access (method='GET' view='sentry.api.endpoints.group_attachments.GroupAttachmentsEndpoint' response=200 user_id='1' is_app='None' token_type='None' is_frontend_request='True' organization_id='1' auth_id='None' path='/api/0/issues/4/attachments/' caller_ip='2.139.195.191' user_agent='Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36' rate_limited='False' rate_limit_category='None' request_duration_seconds=0.03198647499084473 rate_limit_type='DNE' concurrent_limit='None' concurrent_requests='None' reset_time='None' group='None' limit='None' remaining='None')
nginx_1                                     | 2023-01-25T09:00:40.294884303Z 2023/01/25 09:00:40 [error] 33#33: *15745 upstream sent too big header while reading response header from upstream, client: 192.168.130.4, server: , request: "GET /api/0/issues/4/attachments/?per_page=50&types=event.minidump&event_id=94b2014c013b4fdf8a6658d6e155b125&event_id=6d27b8a6fb5f4bf9b39cefe2dbb1981f&event_id=1c70b27da43a497eaf0670ce9dd98be2&event_id=e489d2131b024d3dab82a29659af5097&event_id=7fa15f6449f8488aaf11b1af9e6f1868&event_id=f86b6eb3862e4ac4b57acac2986c5e1f&event_id=9de64a414a504d5f9b68c4da2614816f&event_id=a4246ca96cc7490586ae8c73ac19ef39&event_id=1474c86b647e4e39a4aa4c8d12a5ec62&event_id=14116e0b4ff24155a82d1237a82e4d22&event_id=077f971535b9413fb4941e9ec3fa69c6&event_id=7d455795af984146a92d6da1ef2dda4d&event_id=986618ce44594f97844d79368d64017c&event_id=d0dfb538f1434a298d9dac7b8d4911af&event_id=f04fedb9ab304244b646ee9bb08e6abf&event_id=5345531abf894189b4c527ca2a3d737f&event_id=c7c64a5c197e4264b78949ba5f183ee2&event_id=aee7a5eeb36b4e49b178171c4b4a5131&event_id=3ac4591ddbab4481a44c08bd571bd1d5&event_id=efe185df9b754efa8bd282918fb22c37&event_id=646e95cf31a84f03b15fa8e295f945e3&event_id=03c13d1ad31c42d499874089683d39b6&event_id=3c7ada9b8b3f478aa07ddc97f74945f4&event_id=6c5c6b831f584643a1cfe20e571e851a&event_id=261356cfb3e4441d8b5d92dc282a79bc&event_id=9101d6372da14788a8dc03abe9c5bcee&event_id=d893c2d9ff7b4a4eaeb58a57adfabc2c&event_id=4f6c17d0de7c47eeb20cf86a8f220d22&event_id=413f15b3aaf04969b733d6d7a9fef99c&event_id=4659e3a30d7f42a2bd1987ad771dd076&event_id=85a6d263b05f482f8d3816bb334c1c31&event_id=9b5968c75eac43c699d0104ae773a0e6&event_id=82e8bba6e54c4bd48567eb98e971a805&event_id=d12795c77a9948da85e6a8f3c7390042&event_id=9cd0588d8354405b8de16928e1283a86&event_id=a5182a403b55419492a0c26029dcb21e&event_id=d95b4f454c9b4781badefbcf8158198a&event_id=245f7a6e2dfc424ca16837ee63a6cfcf&event_id=4a90197869d34c61a3bd52a56286cd00&event_id=4654bbd833df4c4f95ecf0298b4208c8&event_id=a5c16b10a98c4094bd572ad5b76980ac&event_id=25baa7dd8fe548c9898d18e48b099ae5&event_id=bae857025e894fa4a078ab9a6fd4fbb2&event_id=3669f9c
nginx_1                                     | 2023-01-25T09:00:40.294991705Z 192.168.130.4 - - [25/Jan/2023:09:00:40 +0000] "GET /api/0/issues/4/attachments/?per_page=50&types=event.minidump&event_id=94b2014c013b4fdf8a6658d6e155b125&event_id=6d27b8a6fb5f4bf9b39cefe2dbb1981f&event_id=1c70b27da43a497eaf0670ce9dd98be2&event_id=e489d2131b024d3dab82a29659af5097&event_id=7fa15f6449f8488aaf11b1af9e6f1868&event_id=f86b6eb3862e4ac4b57acac2986c5e1f&event_id=9de64a414a504d5f9b68c4da2614816f&event_id=a4246ca96cc7490586ae8c73ac19ef39&event_id=1474c86b647e4e39a4aa4c8d12a5ec62&event_id=14116e0b4ff24155a82d1237a82e4d22&event_id=077f971535b9413fb4941e9ec3fa69c6&event_id=7d455795af984146a92d6da1ef2dda4d&event_id=986618ce44594f97844d79368d64017c&event_id=d0dfb538f1434a298d9dac7b8d4911af&event_id=f04fedb9ab304244b646ee9bb08e6abf&event_id=5345531abf894189b4c527ca2a3d737f&event_id=c7c64a5c197e4264b78949ba5f183ee2&event_id=aee7a5eeb36b4e49b178171c4b4a5131&event_id=3ac4591ddbab4481a44c08bd571bd1d5&event_id=efe185df9b754efa8bd282918fb22c37&event_id=646e95cf31a84f03b15fa8e295f945e3&event_id=03c13d1ad31c42d499874089683d39b6&event_id=3c7ada9b8b3f478aa07ddc97f74945f4&event_id=6c5c6b831f584643a1cfe20e571e851a&event_id=261356cfb3e4441d8b5d92dc282a79bc&event_id=9101d6372da14788a8dc03abe9c5bcee&event_id=d893c2d9ff7b4a4eaeb58a57adfabc2c&event_id=4f6c17d0de7c47eeb20cf86a8f220d22&event_id=413f15b3aaf04969b733d6d7a9fef99c&event_id=4659e3a30d7f42a2bd1987ad771dd076&event_id=85a6d263b05f482f8d3816bb334c1c31&event_id=9b5968c75eac43c699d0104ae773a0e6&event_id=82e8bba6e54c4bd48567eb98e971a805&event_id=d12795c77a9948da85e6a8f3c7390042&event_id=9cd0588d8354405b8de16928e1283a86&event_id=a5182a403b55419492a0c26029dcb21e&event_id=d95b4f454c9b4781badefbcf8158198a&event_id=245f7a6e2dfc424ca16837ee63a6cfcf&event_id=4a90197869d34c61a3bd52a56286cd00&event_id=4654bbd833df4c4f95ecf0298b4208c8&event_id=a5c16b10a98c4094bd572ad5b76980ac&event_id=25baa7dd8fe548c9898d18e48b099ae5&event_id=bae857025e894fa4a078ab9a6fd4fbb2&event_id=3669f9cb317b4e718e95dd72889fa9e1&event_id=cc18236ca79d4a278efdd506b31a844c&event_id=efe94cd7fac04350bc48b368b3654eef&event_id=b3abe5644ab043f6af46a6f999ddeb4b&event_id=d3014d470c724194ae7f5f0279e79ebf&event_id=926ff5dd18ba487c9f07842dbc37f79f&event_id=c562f7bc69b94ccfbb8eeeb9d162e4fe HTTP/1.1" 502 552 "https://sentry.travelcdev.com/organizations/sentry/issues/4/events/?project=2&referrer=issue-stream" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36" "2.139.195.191, 188.114.111.72"
web_1                                       | 2023-01-25T09:00:40.296475238Z [uwsgi-http key:  client_addr: 192.168.16.31 client_port: 4784] hr_read(): Connection reset by peer [plugins/http/http.c line 918]
post-process-forwarder-errors_1             | 2023-01-25T09:00:41.002122697Z 09:00:41 [INFO] sentry.batching-kafka-consumer: Flushing 1 items (from {('events', 0): [333517517, 333517517]}): forced:False size:True time:False
post-process-forwarder-errors_1             | 2023-01-25T09:00:41.005560175Z 09:00:41 [INFO] sentry.batching-kafka-consumer: Worker flush took 1ms
post-process-forwarder-errors_1             | 2023-01-25T09:00:41.007269414Z 09:00:41 [INFO] sentry.batching-kafka-consumer: Flushing 1 items (from {('events', 0): [333517518, 333517518]}): forced:False size:True time:False
post-process-forwarder-errors_1             | 2023-01-25T09:00:41.010444686Z 09:00:41 [INFO] sentry.batching-kafka-consumer: Worker flush took 2ms
ingest-consumer_1                           | 2023-01-25T09:00:42.874841009Z 09:00:42 [INFO] sentry.batching-kafka-consumer: Flushing 5 items (from {('ingest-events', 0): [5837086, 5837090]}): forced:False size:False time:True
ingest-consumer_1                           | 2023-01-25T09:00:42.905566903Z 09:00:42 [INFO] sentry.batching-kafka-consumer: Worker flush took 30ms
web_1                                       | 2023-01-25T09:00:43.643789367Z 09:00:43 [INFO] sentry.access.api: api.access (method='GET' view='sentry.api.endpoints.group_attachments.GroupAttachmentsEndpoint' response=200 user_id='1' is_app='None' token_type='None' is_frontend_request='True' organization_id='1' auth_id='None' path='/api/0/issues/4/attachments/' caller_ip='2.139.195.191' user_agent='Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36' rate_limited='False' rate_limit_category='None' request_duration_seconds=0.03460073471069336 rate_limit_type='DNE' concurrent_limit='None' concurrent_requests='None' reset_time='None' group='None' limit='None' remaining='None')
nginx_1                                     | 2023-01-25T09:00:43.645719210Z 2023/01/25 09:00:43 [error] 33#33: *15757 upstream sent too big header while reading response header from upstream, client: 192.168.130.4, server: , request: "GET /api/0/issues/4/attachments/?per_page=50&types=event.minidump&event_id=94b2014c013b4fdf8a6658d6e155b125&event_id=6d27b8a6fb5f4bf9b39cefe2dbb1981f&event_id=1c70b27da43a497eaf0670ce9dd98be2&event_id=e489d2131b024d3dab82a29659af5097&event_id=7fa15f6449f8488aaf11b1af9e6f1868&event_id=f86b6eb3862e4ac4b57acac2986c5e1f&event_id=9de64a414a504d5f9b68c4da2614816f&event_id=a4246ca96cc7490586ae8c73ac19ef39&event_id=1474c86b647e4e39a4aa4c8d12a5ec62&event_id=14116e0b4ff24155a82d1237a82e4d22&event_id=077f971535b9413fb4941e9ec3fa69c6&event_id=7d455795af984146a92d6da1ef2dda4d&event_id=986618ce44594f97844d79368d64017c&event_id=d0dfb538f1434a298d9dac7b8d4911af&event_id=f04fedb9ab304244b646ee9bb08e6abf&event_id=5345531abf894189b4c527ca2a3d737f&event_id=c7c64a5c197e4264b78949ba5f183ee2&event_id=aee7a5eeb36b4e49b178171c4b4a5131&event_id=3ac4591ddbab4481a44c08bd571bd1d5&event_id=efe185df9b754efa8bd282918fb22c37&event_id=646e95cf31a84f03b15fa8e295f945e3&event_id=03c13d1ad31c42d499874089683d39b6&event_id=3c7ada9b8b3f478aa07ddc97f74945f4&event_id=6c5c6b831f584643a1cfe20e571e851a&event_id=261356cfb3e4441d8b5d92dc282a79bc&event_id=9101d6372da14788a8dc03abe9c5bcee&event_id=d893c2d9ff7b4a4eaeb58a57adfabc2c&event_id=4f6c17d0de7c47eeb20cf86a8f220d22&event_id=413f15b3aaf04969b733d6d7a9fef99c&event_id=4659e3a30d7f42a2bd1987ad771dd076&event_id=85a6d263b05f482f8d3816bb334c1c31&event_id=9b5968c75eac43c699d0104ae773a0e6&event_id=82e8bba6e54c4bd48567eb98e971a805&event_id=d12795c77a9948da85e6a8f3c7390042&event_id=9cd0588d8354405b8de16928e1283a86&event_id=a5182a403b55419492a0c26029dcb21e&event_id=d95b4f454c9b4781badefbcf8158198a&event_id=245f7a6e2dfc424ca16837ee63a6cfcf&event_id=4a90197869d34c61a3bd52a56286cd00&event_id=4654bbd833df4c4f95ecf0298b4208c8&event_id=a5c16b10a98c4094bd572ad5b76980ac&event_id=25baa7dd8fe548c9898d18e48b099ae5&event_id=bae857025e894fa4a078ab9a6fd4fbb2&event_id=3669f9c
nginx_1                                     | 2023-01-25T09:00:43.645767511Z 192.168.130.4 - - [25/Jan/2023:09:00:43 +0000] "GET /api/0/issues/4/attachments/?per_page=50&types=event.minidump&event_id=94b2014c013b4fdf8a6658d6e155b125&event_id=6d27b8a6fb5f4bf9b39cefe2dbb1981f&event_id=1c70b27da43a497eaf0670ce9dd98be2&event_id=e489d2131b024d3dab82a29659af5097&event_id=7fa15f6449f8488aaf11b1af9e6f1868&event_id=f86b6eb3862e4ac4b57acac2986c5e1f&event_id=9de64a414a504d5f9b68c4da2614816f&event_id=a4246ca96cc7490586ae8c73ac19ef39&event_id=1474c86b647e4e39a4aa4c8d12a5ec62&event_id=14116e0b4ff24155a82d1237a82e4d22&event_id=077f971535b9413fb4941e9ec3fa69c6&event_id=7d455795af984146a92d6da1ef2dda4d&event_id=986618ce44594f97844d79368d64017c&event_id=d0dfb538f1434a298d9dac7b8d4911af&event_id=f04fedb9ab304244b646ee9bb08e6abf&event_id=5345531abf894189b4c527ca2a3d737f&event_id=c7c64a5c197e4264b78949ba5f183ee2&event_id=aee7a5eeb36b4e49b178171c4b4a5131&event_id=3ac4591ddbab4481a44c08bd571bd1d5&event_id=efe185df9b754efa8bd282918fb22c37&event_id=646e95cf31a84f03b15fa8e295f945e3&event_id=03c13d1ad31c42d499874089683d39b6&event_id=3c7ada9b8b3f478aa07ddc97f74945f4&event_id=6c5c6b831f584643a1cfe20e571e851a&event_id=261356cfb3e4441d8b5d92dc282a79bc&event_id=9101d6372da14788a8dc03abe9c5bcee&event_id=d893c2d9ff7b4a4eaeb58a57adfabc2c&event_id=4f6c17d0de7c47eeb20cf86a8f220d22&event_id=413f15b3aaf04969b733d6d7a9fef99c&event_id=4659e3a30d7f42a2bd1987ad771dd076&event_id=85a6d263b05f482f8d3816bb334c1c31&event_id=9b5968c75eac43c699d0104ae773a0e6&event_id=82e8bba6e54c4bd48567eb98e971a805&event_id=d12795c77a9948da85e6a8f3c7390042&event_id=9cd0588d8354405b8de16928e1283a86&event_id=a5182a403b55419492a0c26029dcb21e&event_id=d95b4f454c9b4781badefbcf8158198a&event_id=245f7a6e2dfc424ca16837ee63a6cfcf&event_id=4a90197869d34c61a3bd52a56286cd00&event_id=4654bbd833df4c4f95ecf0298b4208c8&event_id=a5c16b10a98c4094bd572ad5b76980ac&event_id=25baa7dd8fe548c9898d18e48b099ae5&event_id=bae857025e894fa4a078ab9a6fd4fbb2&event_id=3669f9cb317b4e718e95dd72889fa9e1&event_id=cc18236ca79d4a278efdd506b31a844c&event_id=efe94cd7fac04350bc48b368b3654eef&event_id=b3abe5644ab043f6af46a6f999ddeb4b&event_id=d3014d470c724194ae7f5f0279e79ebf&event_id=926ff5dd18ba487c9f07842dbc37f79f&event_id=c562f7bc69b94ccfbb8eeeb9d162e4fe HTTP/1.1" 502 552 "https://sentry.travelcdev.com/organizations/sentry/issues/4/events/?project=2&referrer=issue-stream" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36" "2.139.195.191, 188.114.111.73"
web_1                                       | 2023-01-25T09:00:43.647463950Z [uwsgi-http key:  client_addr: 192.168.16.31 client_port: 5808] hr_read(): Connection reset by peer [plugins/http/http.c line 918]
clickhouse_1                                | 2023-01-25T09:00:44.025354479Z 2023.01.25 09:00:44.025148 [ 121 ] {bb5816e3-8431-4eae-815b-aef52ae92e0c} <Information> executeQuery: Read 5 rows, 283.00 B in 0.006 sec., 867 rows/sec., 47.93 KiB/sec.
clickhouse_1                                | 2023-01-25T09:00:44.025602084Z 2023.01.25 09:00:44.025507 [ 121 ] {} <Information> HTTPHandler: Done processing query
snuba-outcomes-consumer_1                   | 2023-01-25T09:00:44.026241499Z 2023-01-25 09:00:44,026 Completed processing <Batch: 5 messages, open for 1.05 seconds>.
clickhouse_1                                | 2023-01-25T09:00:44.032469339Z 2023.01.25 09:00:44.032273 [ 212 ] {042ef020-f43a-4c1b-be9d-ac36cfeccfb2} <Information> executeQuery: Read 5 rows, 41.18 KiB in 0.011 sec., 435 rows/sec., 3.50 MiB/sec.
clickhouse_1                                | 2023-01-25T09:00:44.032788946Z 2023.01.25 09:00:44.032682 [ 212 ] {} <Information> HTTPHandler: Done processing query
snuba-consumer_1                            | 2023-01-25T09:00:44.034201578Z 2023-01-25 09:00:44,033 Completed processing <Batch: 5 messages, open for 1.05 seconds>.
post-process-forwarder-errors_1             | 2023-01-25T09:00:45.006475508Z 09:00:45 [INFO] sentry.batching-kafka-consumer: Flushing 1 items (from {('events', 0): [333517519, 333517519]}): forced:False size:True time:False
post-process-forwarder-errors_1             | 2023-01-25T09:00:45.010108790Z 09:00:45 [INFO] sentry.batching-kafka-consumer: Worker flush took 3ms
post-process-forwarder-errors_1             | 2023-01-25T09:00:45.011684725Z 09:00:45 [INFO] sentry.batching-kafka-consumer: Flushing 1 items (from {('events', 0): [333517520, 333517520]}): forced:False size:True time:False
post-process-forwarder-errors_1             | 2023-01-25T09:00:45.014962299Z 09:00:45 [INFO] sentry.batching-kafka-consumer: Worker flush took 1ms
post-process-forwarder-errors_1             | 2023-01-25T09:00:45.016451233Z 09:00:45 [INFO] sentry.batching-kafka-consumer: Flushing 1 items (from {('events', 0): [333517521, 333517521]}): forced:False size:True time:False
post-process-forwarder-errors_1             | 2023-01-25T09:00:45.019512002Z 09:00:45 [INFO] sentry.batching-kafka-consumer: Worker flush took 2ms
post-process-forwarder-errors_1             | 2023-01-25T09:00:45.021886155Z 09:00:45 [INFO] sentry.batching-kafka-consumer: Flushing 1 items (from {('events', 0): [333517522, 333517522]}): forced:False size:True time:False
post-process-forwarder-errors_1             | 2023-01-25T09:00:45.024361611Z 09:00:45 [INFO] sentry.batching-kafka-consumer: Worker flush took 1ms
post-process-forwarder-errors_1             | 2023-01-25T09:00:45.025852845Z 09:00:45 [INFO] sentry.batching-kafka-consumer: Flushing 1 items (from {('event

[cocorossello]

Ok, it is working now. I only had to increase the nginx proxy buffer size (the one from sentry and also all intermediate ones, in our case we have one more nginx in front).

Just change:
proxy_buffering off;

to:

    proxy_buffers    16 16k;
    proxy_buffer_size   32k;

[hubertdeng123]

Ah, glad to hear that you found a workaround