3.1.1: TypeError: cannot use a string pattern on a bytes-like object

[esabol]

Hi. The README.md says Python 3.7 and above is supported? I'm using Python 3.8.5, but I'm getting the following error:

Traceback (most recent call last):
  File "/path/to/home/.local/bin/pg_activity", line 8, in <module>
    sys.exit(main())
  File "/path/to/home/.local/lib/python3.8/site-packages/pgactivity/cli.py", line 389, in main
    dataobj = data.pg_connect(
  File "/path/to/home/.local/lib/python3.8/site-packages/pgactivity/data.py", line 518, in pg_connect
    data = Data.pg_connect(
  File "/path/to/home/.local/lib/python3.8/site-packages/pgactivity/data.py", line 107, in pg_connect
    pg_version = pg_get_short_version(pg_get_version(pg_conn))
  File "/path/to/home/.local/lib/python3.8/site-packages/pgactivity/data.py", line 57, in pg_get_short_version
    res = re.match(
  File "/usr1/local/anaconda_py3/miniconda38/lib/python3.8/re.py", line 191, in match
    return _compile(pattern, flags).match(string)
TypeError: cannot use a string pattern on a bytes-like object

> ~/.local/bin/pg_activity --version
pg_activity 3.1.1
> /usr1/local/anaconda_py3/miniconda38/bin/python3 --version
Python 3.8.5

I installed it using the command python3 -m pip install "pg_activity[psycopg]" --user.

Thanks,
Ed

[dlax]

That's strange, I cannot reproduce (tried to install miniconda, which is Python 3.8.16 at the moment).

Can you try to run this python session from /usr1/local/anaconda_py3/miniconda38/bin/python3 (the python3 executable you used during installation?) and report back?

>>> from pgactivity import pg
>>> pg.__version__
>>> conn = pg.connect()
>>> pg.fetchone(conn, "select version()")

[esabol]

Well, I get

psycopg.OperationalError: connection is bad: No such file or directory
        Is the server running locally and accepting connections on that socket?

if I do that, but that's because my server is not local. I have to use a connection string, like this:

>>> conn = pg.connect("host='myhost' dbname='mydatabase' user='postgres'")
>>> pg.fetchone(conn, "select version()");
{'version': b'PostgreSQL 10.23 on x86_64-pc-linux-gnu, compiled by gcc (GCC) 4.4.7 20120313 (Red Hat 4.4.7-18), 64-bit'}

When I start pg_activity, I specify the command line arguments -h myhost -d mydatabase -U postgres, and my password is in ~/.pgpass.

[dlax]

Ok; then can you try?

>>> import psycopg
>>> conn = psycopg.connect("<your connection string>")
>>> conn.execute("select version()").fetchone()

[esabol]

(b'PostgreSQL 10.23 on x86_64-pc-linux-gnu, compiled by gcc (GCC) 4.4.7 20120313 (Red Hat 4.4.7-18), 64-bit',)

[esabol]

Using .decode() to convert the bytes to a string works just fine, also:

>>> conn.execute("select version()").fetchone()[0].decode()
'PostgreSQL 10.23 on x86_64-pc-linux-gnu, compiled by gcc (GCC) 4.4.7 20120313 (Red Hat 4.4.7-18), 64-bit'

[dlax]

Indeed; but all these calls should return str values, not bytes.

[dlax]

There's something strange with your Python environment or your locale setup.

[esabol]

Any idea what? I've tried with LANG=en_US.UTF-8 and unsetenv-ing LANG, and I get the same result.

[esabol]

Well, it seems to be a bug in psycopg:

psycopg/psycopg#503

Looks like it was fixed on February 1 (psycopg/psycopg#504), but a new version hasn't been released since then. I'm using the latest released version, 3.1.8, from January 28th.

[dlax]

Indeed; the client encoding is probably a good lead.

Can you try adding client_encoding=utf-8 to your connection string?

On my side:

>>> import psycopg
>>> conn = psycopg.connect("client_encoding=sql_ascii")
>>> conn.info.encoding
'ascii'
>>> conn.execute("select version()").fetchone()
(b'PostgreSQL 15.2 (Debian 15.2-1.pgdg110+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 10.2.1-6) 10.2.1 20210110, 64-bit',)
>>>
>>> conn = psycopg.connect()
>>> conn.info.encoding
'utf-8'
>>> conn.execute("select version()").fetchone()
('PostgreSQL 15.2 (Debian 15.2-1.pgdg110+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 10.2.1-6) 10.2.1 20210110, 64-bit',)

However, the fix in psycopg you mention would not resolve this particular issue as it does not apply to all executed statements but only to TypeInfo.fetch(). So I think we should catch this early on our side and ... do something.

[esabol]

Yes, appending " client_encoding=utf-8" to my connection string seems to work!

>>> import psychopg
>>> conn = psycopg.connect("host=myhost dbname=mydatabase user=postgres client_encoding=utf-8")
>>> conn.info.encoding
'utf-8'
>>> conn.execute("select version()").fetchone()
('PostgreSQL 10.23 on x86_64-pc-linux-gnu, compiled by gcc (GCC) 4.4.7 20120313 (Red Hat 4.4.7-18), 64-bit',)

[esabol]

If I do that with pg_activity, I still get an error, unfortunately:

> ~/.local/bin/pg_activity "host=myhost dbname=mydatabase user=postgres client_encoding=utf-8"

Traceback (most recent call last):
  File "/path/to/.local/bin/pg_activity", line 8, in <module>
    sys.exit(main())
  File "/path/to/.local/lib/python3.8/site-packages/pgactivity/cli.py", line 408, in main
    ui.main(term, dataobj, host, args)
  File "/path/to/.local/lib/python3.8/site-packages/pgactivity/ui.py", line 204, in main
    pg_procs.set_items(data.pg_get_activities(ui.duration_mode))
  File "/path/to/.local/lib/python3.8/site-packages/pgactivity/data.py", line 413, in pg_get_activities
    return pg.fetchall(
  File "/path/to/.local/lib/python3.8/site-packages/pgactivity/pg.py", line 147, in fetchall
    return cur.execute(query, args, prepare=True).fetchall()
  File "/path/to/.local/lib/python3.8/site-packages/psycopg/cursor.py", line 851, in fetchall
    records = self._tx.load_rows(self._pos, self.pgresult.ntuples, self._make_row)
  File "psycopg_binary/_psycopg/transform.pyx", line 490, in psycopg_binary._psycopg.Transformer.load_rows
  File "/path/to/.local/lib/python3.8/site-packages/psycopg/rows.py", line 208, in kwargs_row__
    return func(**dict(zip(names, values)))  # type: ignore[arg-type]
  File "/path/to/.local/lib/python3.8/site-packages/pgactivity/types.py", line 933, in from_bytes
    kwargs[name] = value.decode(enc, errors="replace")
LookupError: unknown encoding: SQL_ASCII

[dlax]

If I do that with pg_activity, I still get an error, unfortunately:

Yes, I noticed that on my side too yesterday; that's a broader issue. I'm working on a fix.

Thank you for your patience.

[dlax]

@esabol I've stack all pending changes in a branch issue348 in our git repo; can you try to install pg_activity as python3 -m pip install "git+https://github.com/dalibo/pg_activity@issue348" and tell me how this works for you?

[esabol]

@dlax : It works! Thank you very much!

Unrelated (apologies): Not sure if this is an issue with my terminal or what, but there are blue â ("a" with caret) characters at the lines at the top, like this:

-/4 logical workers â
 -/8 parallel workers
   Other processes & info: 0/3 autovacuum workers â
 0/10 wal senders â
 0 wal receivers â
 -/10 repl. slots

What are those supposed to be? Is that how it's supposed to look?

Also, every ~5 seconds or so, a lot of text is printed to the screen, but then it vanishes so quickly that I can't read what it says. Is that to be expected?

[dlax]

Thank you for confirming, I merged the changes and will release a new version later today.

About rendering issues, no it's not how it's supposed to look like, see the screenshot on project homepage. These are blue ⋅ normally.

What terminal are you using?

[esabol]

What terminal are you using?

$TERM is "screen". But if I detach from my screen, my $TERM is "xterm-color", and I still get the blue â ("a" with caret) instead of the blue bullets/dots there. Ultimately, I'm inside a macOS Terminal window. $LANG is en_US.UTF-8. EDIT: I tried in an xterm window. It looks much nicer there, but no bullets/dots at all. I'll open a new issue.... EDIT #2: I figured out why I was getting â instead of the bullets/dots. My macOS Terminal window was set for Latin-1 instead of UTF-8. If I change it to UTF-8, I get the blue bullets/dots.

I think the text that flashes and then disappears every ~5 seconds are warnings about the ascii encoding.... EDIT: Upgrading to 3.2.0 seems to have stopped this.