Unicode issues with output to stdout

[jcftang]

This relates to

#141

After that the above patch, output to stdout still causes an issue with hanging...

warning: deprecated conversion from string constant to 'UTF8* {aka char*}' [-Wwrite-strings]
Exception in thread Thread-6:
Traceback (most recent call last):
  File "/usr/lib64/python3.4/threading.py", line 911, in _bootstrap_inner
    self.run()
  File "/usr/lib64/python3.4/threading.py", line 859, in run
    self._target(*self._args, **self._kwargs)
  File "/home/build/polly/bin/detail/call.py", line 24, in fanout
    s = line.decode(sys.stdout.encoding) # use the same encoding as stdout/stderr
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe4 in position 54: ordinal not in range(128)

The above was generated from a build that was executed inside a docker container using python 3.4 on centos7. It looks like the encoding in my container is always set to ANSI_X3.4-1968

[ruslo]

using python 3.4

How you install it?

[jcftang]

I installed it via the vendor and epel repositories

[ruslo]

I'm not very experienced with CentOS, it would be nice if you give me the exact instructions :) Thanks.

[jcftang]

@ruslo if you have a centos7 vm or container running...

yum install -y epel-release
yum install -y python34 cmake3 gcc-c++ gcc

The above should give you an environment to run cmake and gcc/g++, I have a bunch of other dependencies that I have which I wont paste into above as its not relevant. Hope the above helps.

[ruslo]

It works fine for me.

> cat /etc/redhat-release
CentOS Linux release 7.3.1611 (Core)

> which cmake3
/usr/bin/cmake3

> cmake3 --version
cmake3 version 3.6.3

> which python3
/usr/bin/python3

> python3 --version
Python 3.4.5

> cat CMakeLists.txt
cmake_minimum_required(VERSION 3.0)
project(foo)

add_executable(foo foo.cpp)

> cat foo.cpp
#include <iostream>

int main() {
  std::cout << "Hello" << std::endl;
}

> polly.py --verbose --clear --config Debug --cmake cmake3

Working directory with Russian characters:

~/work/test/locale/привет

Seems that stdout encoding is UTF8:

> python3 -c 'import sys; print(sys.stdout.encoding)'
UTF-8

[ruslo]

Just for your info I still can't find the way to change the keyboard layout to another language :) I have to copy/paste string from another machine. Any hints welcome.

[jcftang]

May be copy and paste from here?

四 十 五

Or else I can gist it at https://gist.github.com/jcftang/89792312f73309c91116554b6b8e3bce

[ruslo]

Same test, works fine:

[100%] Built target foo
gmake[1]: Leaving directory `/home/ruslo/work/test/locale-2/四 十 五/_builds/default-Debug'
/usr/bin/cmake3 -E cmake_progress_start "/home/ruslo/work/test/locale-2/四 十 五/_builds/default-Debug/CMakeFiles" 0
-
Log saved: /home/ruslo/work/test/locale-2/四 十 五/_logs/polly/default/log.txt
-
Generate: 0:00:02.099576s
Build: 0:00:01.266635s
-
Total: 0:00:03.367197s
-
SUCCESS

Try to run this from console:

> python3 -c 'import sys; print(sys.stdout.encoding)'
UTF-8

[jcftang]

@ruslo the problem exhibits itself when I run the scripts inside a centos7 docker container,...

bash-4.2$ python3 -c 'import sys; print(sys.stdout.encoding)'
ANSI_X3.4-1968

[ruslo]

What about

> echo $LC_ALL
en_US.UTF-8

?

[jcftang]

Doing that gives me an empty value

bash-4.2$ echo $LC_ALL

bash-4.2$

the whole env just has this...

bash-4.2$ env
HOSTNAME=74932611f4c9
TERM=xterm
SSH_AUTH_SOCK=/ssh-agent-sock
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
PWD=/home/build/
SHLVL=1
HOME=/
_=/usr/bin/env
OLDPWD=/home/build

[ruslo]

Can you try export LC_ALL=en_US.UTF-8 and run polly.py again.

[jcftang]

setting the LC_ALL environment variable does not help at all

Exception in thread Thread-8:
Traceback (most recent call last):
  File "/usr/lib64/python3.4/threading.py", line 911, in _bootstrap_inner
    self.run()
  File "/usr/lib64/python3.4/threading.py", line 859, in run
    self._target(*self._args, **self._kwargs)
  File "/home/build/polly/bin/detail/call.py", line 24, in fanout
    s = line.decode(sys.stdout.encoding) # use the same encoding as stdout/stderr
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe4 in position 54: invalid continuation byte

^CTraceback (most recent call last):
  File "./polly/bin/polly", line 7, in <module>
    exec(open(polly_py, 'r').read())
  File "<string>", line 428, in <module>
  File "/home/build/polly/bin/detail/call.py", line 85, in call
    x = teed_call(call_args, logging)
  File "/home/build/polly/bin/detail/call.py", line 64, in teed_call
    t.join() # wait for IO completion
  File "/usr/lib64/python3.4/threading.py", line 1051, in join
    self._wait_for_tstate_lock()
  File "/usr/lib64/python3.4/threading.py", line 1067, in _wait_for_tstate_lock
    elif lock.acquire(block, timeout):
KeyboardInterrupt
bash-4.2$ env
HOSTNAME=74932611f4c9
TERM=xterm
OLDPWD=/home/build
LC_ALL=en_US.UTF-8
SSH_AUTH_SOCK=/ssh-agent-sock
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
PWD=/home/build
SHLVL=1
HOME=/
_=/usr/bin/env
bash-4.2$

[ruslo]

I'm running out of ideas. Let's try replace strategy, please try pr.encoding branch.

I'm still not able to reproduce the error, can you share your docker image? E.g. by https://quay.io/ service.

[jcftang]

@ruslo the branch https://github.com/ruslo/polly/tree/pr.encoding appears to have addressed the hanging, my builds at least succeed. I think its probably safe to merge that PR

[ruslo]

I think its probably safe to merge that PR

It's merged to master now. However I still thinking about reproducing your error. Don't want to move in circles in case this part will change again later for some reason.

[jcftang]

I tend to agree this bug is a bit obscure and hard to reproduce. I can't share my container that I am using to reproduce this. I will see if I can create a test case for you to reproduce if time permits.