ClickHouse issue: Unexpected packet from server (got Data)

[charles-001]

Describe the bug
I get the error below when reladiff tries to connect to ClickHouse. It doesn't really seem like the error lies with reladiff since the error is coming from the driver, but thought I'd ask here just in case it could be reladiff somehow.

Traceback (most recent call last):
  File "/Users/cthompson/.pyenv/versions/3.11.9/bin/reladiff", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/Users/cthompson/.pyenv/versions/3.11.9/lib/python3.11/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/cthompson/.pyenv/versions/3.11.9/lib/python3.11/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/Users/cthompson/.pyenv/versions/3.11.9/lib/python3.11/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/cthompson/.pyenv/versions/3.11.9/lib/python3.11/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/cthompson/.pyenv/versions/3.11.9/lib/python3.11/site-packages/reladiff/__main__.py", line 218, in main
    return _main(**kw)
           ^^^^^^^^^^^
  File "/Users/cthompson/.pyenv/versions/3.11.9/lib/python3.11/site-packages/reladiff/__main__.py", line 367, in _main
    schemas = list(differ._thread_map(_get_schema, safezip(dbs, table_paths)))
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/cthompson/.pyenv/versions/3.11.9/lib/python3.11/concurrent/futures/_base.py", line 619, in result_iterator
    yield _result_or_cancel(fs.pop())
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/cthompson/.pyenv/versions/3.11.9/lib/python3.11/concurrent/futures/_base.py", line 317, in _result_or_cancel
    return fut.result(timeout)
           ^^^^^^^^^^^^^^^^^^^
  File "/Users/cthompson/.pyenv/versions/3.11.9/lib/python3.11/concurrent/futures/_base.py", line 449, in result
    return self.__get_result()
           ^^^^^^^^^^^^^^^^^^^
  File "/Users/cthompson/.pyenv/versions/3.11.9/lib/python3.11/concurrent/futures/_base.py", line 401, in __get_result
    raise self._exception
  File "/Users/cthompson/.pyenv/versions/3.11.9/lib/python3.11/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/cthompson/.pyenv/versions/3.11.9/lib/python3.11/site-packages/reladiff/__main__.py", line 48, in _get_schema
    return db.query_table_schema(table_path)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/cthompson/.pyenv/versions/3.11.9/lib/python3.11/site-packages/sqeleton/databases/base.py", line 449, in query_table_schema
    rows = self.query(self.select_table_schema(path), list)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/cthompson/.pyenv/versions/3.11.9/lib/python3.11/site-packages/sqeleton/databases/base.py", line 386, in query
    res = self._query(sql_code)
          ^^^^^^^^^^^^^^^^^^^^^
  File "/Users/cthompson/.pyenv/versions/3.11.9/lib/python3.11/site-packages/sqeleton/databases/base.py", line 630, in _query
    return r.result()
           ^^^^^^^^^^
  File "/Users/cthompson/.pyenv/versions/3.11.9/lib/python3.11/concurrent/futures/_base.py", line 456, in result
    return self.__get_result()
           ^^^^^^^^^^^^^^^^^^^
  File "/Users/cthompson/.pyenv/versions/3.11.9/lib/python3.11/concurrent/futures/_base.py", line 401, in __get_result
    raise self._exception
  File "/Users/cthompson/.pyenv/versions/3.11.9/lib/python3.11/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/cthompson/.pyenv/versions/3.11.9/lib/python3.11/site-packages/sqeleton/databases/base.py", line 636, in _query_in_worker
    return self._query_conn(self.thread_local.conn, sql_code)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/cthompson/.pyenv/versions/3.11.9/lib/python3.11/site-packages/sqeleton/databases/base.py", line 584, in _query_conn
    return apply_query(callback, sql_code)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/cthompson/.pyenv/versions/3.11.9/lib/python3.11/site-packages/sqeleton/databases/base.py", line 135, in apply_query
    return callback(sql_code)
           ^^^^^^^^^^^^^^^^^^
  File "/Users/cthompson/.pyenv/versions/3.11.9/lib/python3.11/site-packages/sqeleton/databases/base.py", line 569, in _query_cursor
    c.execute(sql_code.code, sql_code.args)
  File "/Users/cthompson/.pyenv/versions/3.11.9/lib/python3.11/site-packages/clickhouse_driver/dbapi/cursor.py", line 117, in execute
    raise OperationalError(orig)
clickhouse_driver.dbapi.errors.OperationalError: Code: 102. Unexpected packet from server $host:9000 (expected ProfileEvents, Progress, Log, Exception or TimezoneUpdate, got Data)

To Reproduce
Run reladiff like this:

reladiff --verbose --debug -i --case-sensitive \
mysql://"$MYSQL_USER":"$MYSQL_PASSWORD"@$MYSQL_HOST:$PORT/$DATABASE $TABLE \
clickhouse://"$CLICKHOUSE_USER":"$CLICKHOUSE_PASSWORD"@"$CLICKHOUSE_HOST:9000/$DATABASE" $TABLE \
--bisection-factor 500 \
--threads 5 \
--update-column created_at \
--min-age 10min \
--columns created_at \
--columns id \
--columns user_id

Versions
Package: 0.5.3
ClickHouse: 24.8.4
Python: 3.11.9

[erezsh]

It doesn't seem like an issue in reladiff, but no harm in documenting errors. Maybe someone else is having a similar issue.

[charles-001]

FYI: I submitted an issue for the driver here

[charles-001]

@erezsh - could sqeleton possibly be the issue here?

[charles-001]

I tried using datadiff which does not exhibit this issue FWIW

[erezsh]

The most likely difference is a different version of the clickhouse driver.

[charles-001]

Unfortunately, that didn't work. I installed 0.2.6 which data-diff uses and got the error:

[12:10:34] ERROR - Code: 102. Unexpected packet from server $host:9000 (expected Exception, EndOfStream or Log, got Data)

[charles-001]

@erezsh Can you please try to reproduce on your end?

[erezsh]

I will try to find time for it soon.

[charles-001]

Hi @erezsh - hope you had good holidays! Any chance you can take a look at this? It would be awesome to be able to use reladiff for our use case.

[erezsh]

Hi @charles-001 , I think this PR should solve the issue: erezsh/sqeleton#29

Any chance you can try it? (you just need to replace your sqeleton installation)

[dankow]

Thank you @erezsh! On behalf of Charles, I gave this a try using the dev branch of sqeleton, but got this error:

[20:38:03] DEBUG - Available mutual columns: {'measurement_ts', 'id', 'created_at', 'bytes', 'project', 'user_id', 'region'}
[20:38:03] INFO - Diffing using columns: key=('id',) update=created_at extra=().
[20:38:03] INFO - Using algorithm 'hashdiff'.
[20:38:03] ERROR - Function 'Compiler.compile_node' not found for signature (<class 'sqeleton.queries.compiler.Compiler'>, <class 'sqeleton.queries.extras.ApplyFuncAndNormalizeAsString'>)
Traceback (most recent call last):
  File "/Users/dankow/Desktop/reladiff/.venv/lib/python3.10/site-packages/runtype/dispatch.py", line 169, in find_function_cached
    return self._cache[tuple(map(self._get_type, args))]
KeyError: (<class 'sqeleton.queries.compiler.Compiler'>, <class 'sqeleton.queries.extras.ApplyFuncAndNormalizeAsString'>)

Here's the command I'm using (with details redacted):

reladiff --verbose --debug -i --case-sensitive \
mysql://<username>:"$MYSQL_PASSWORD"@<mysql_host>:3306/<database> <table>\
clickhouse://<username>:"$CLICKHOUSE_PASSWORD"@<clickhouse_host>:9000/<database> <table> \
--bisection-factor 500 \
--threads 5 \
--update-column created_at \
--min-age 10min \
--columns id

I'm not sure if that error means I did something wrong with using sqeleton from the branch, or something else.

[charles-001]

Hi @erezsh - thank you for looking into this! It worked for me :) Not sure why it didn't for Dan, but if you can, please publish this to pip. Thanks again!

[erezsh]

@dankow Can you please try this PR? I think it should fix your error.

#59

[dankow]

@erezsh That works, thank you very much!