Revert "Add params for secure ClickHouse connections."

[evanh]

Reverts #2018

[codecov-commenter]

Codecov Report

Merging #2033 (a700917) into master (39cdb11) will increase coverage by 0.16%.
The diff coverage is 100.00%.

❗ Current head a700917 differs from pull request most recent head 1169fa0. Consider uploading reports for the commit 1169fa0 to get more accurate results

@@            Coverage Diff             @@
##           master    #2033      +/-   ##
==========================================
+ Coverage   90.93%   91.10%   +0.16%     
==========================================
  Files         499      499              
  Lines       21576    21564      -12     
==========================================
+ Hits        19621    19646      +25     
+ Misses       1955     1918      -37     

Impacted Files	Coverage Δ
snuba/cli/cleanup.py	0.00% <ø> (ø)
snuba/cli/migrations.py	0.00% <ø> (ø)
snuba/cli/optimize.py	0.00% <ø> (ø)
snuba/clickhouse/native.py	73.04% <ø> (-0.47%)	⬇️
snuba/clusters/cluster.py	93.63% <ø> (-1.90%)	⬇️
snuba/migrations/runner.py	91.66% <ø> (ø)
snuba/settings/__init__.py	92.50% <ø> (ø)
snuba/settings/settings_distributed.py	0.00% <ø> (-100.00%)	⬇️
tests/clusters/test_cluster.py	100.00% <ø> (ø)
tests/migrations/test_table_engines.py	100.00% <ø> (ø)
... and 16 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 39cdb11...1169fa0. Read the comment docs.

[katsil]

Hi! Can you tell me why this issue reverted, i guess cliclhouse secure connection is very important issue

[katsil]

@fpacifici @evanh

[fpacifici]

Hi, sorry this was reverted because of a push safety issue with the original PR that caused a production issue when deployed.
The original PR is still desirable and should be fixed and re-issued. @evanh did you have a chance to nail down the root cause ?

[katsil]

@evanh hi, im really waiting this feautre asap because we are trying to launch sentry 20 in production in destributed mode and DBaas databases (clickhouse) which supports only ssl/tls mode..

[evanh]

When we deployed this it caused Snuba to go into a crashloop and I haven't had time to spin up an environment to try and debug why. @katsil Did you test your changes outside of the tests included in Snuba?

[katsil]

@evanh i dont even tryed to run tests - just insert all changes inside snuba code, build docker image and push them to private registry, then got SSL CA CERT ERRORs..

Like:

2021-08-23 09:41:35,706 Failed to connect to my-ch-db-host:8443
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/clickhouse_driver/connection.py", line 260, in connect
    return self._init_connection(host, port)
  File "/usr/local/lib/python3.8/dist-packages/clickhouse_driver/connection.py", line 226, in _init_connection
    self.socket = self._create_socket(host, port)
  File "/usr/local/lib/python3.8/dist-packages/clickhouse_driver/connection.py", line 221, in _create_socket
    raise err
  File "/usr/local/lib/python3.8/dist-packages/clickhouse_driver/connection.py", line 212, in _create_socket
    sock.connect(sa)
  File "/usr/lib/python3.8/ssl.py", line 1342, in connect
    self._real_connect(addr, False)
  File "/usr/lib/python3.8/ssl.py", line 1333, in _real_connect
    self.do_handshake()
  File "/usr/lib/python3.8/ssl.py", line 1309, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1108)

clickhouse_driver.errors.NetworkError: Code: 210. [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1108) (sas-wfwqv2yebnxbtfrx.db.yandex.net:8443)

Can you tell me on which port i should connect - clickhouse own native port (9000/9440 for tls) or http/https clickhouse port (8123 or 8443 for https) *in my setup only 8443/9440 ports are available

[katsil]

@evanh hi, maybe any news? or we can reopen issue, i guess supporting clickhouse ssl is very important feature for snuba

[evanh]

So looking at the error, it looks to be because the certificate is self signed and Clickhouse has to be configured correctly to support self signed certificates. https://clickhouse.tech/docs/en/operations/server-configuration-parameters/settings/#server_configuration_parameters-openssl

Can you connect directly to Clickhouse with your cert? Without going through Snuba?

As for the port, that is also configured explicitly in Clickhouse. https://clickhouse.tech/docs/en/operations/server-configuration-parameters/settings/#http-porthttps-port

If you can connect correctly to Clickhouse, then I would try adding your changes back to Snuba, and testing that you can connect and run queries in Snuba with HTTPS.

[katsil]

@evanh If i run multiple snuba instances with multiple crons gosu snuba snuba cleanup --storage transactions --dry-run False is there any native snuba locks (if multiple cronjobs will run simultaneously?)

[evanh]

@katsil There aren't any internal locks I'm aware of. I'm not familiar with running Snuba in this way but the only contention I could think of is that all of the instances will be listening for connections on the same port.

[konstantin-popov]

Hi @evanh, actually I successfully connected to ClickHouse using SSL. May be you can somehow debug the issue on your deploy?
@katsil, I guess your problem is in client-side cert validation error (disliking self-signed certificate), not server-side.

[evanh]

@konstantin-popov Sorry are you saying you tested your SSL change in Snuba and were able to successfully run tests with SSL enabled?

[konstantin-popov]

No, I didn't change testing environment to use SSL for ClickHouse connections (it would require some configuration effort to create certificates for CH and I thought it was not the thing worth testing).
What I meant is that I was able to start Snuba instances working with ClickHouse using SSL and it appears to function ok.
Is there a way I can help you in debugging the issue?

[patsevanton]

Has anyone had any luck connecting snuba to clickhouse over SSL? How did you do it (besides fixing the source code) ? Have you tried adding a certificate to the python certifi package ?