Add Unicode support to the doctesting framework

[jdemeyer]

Support

1. Doctest (i.e. sage:) lines containing non-ASCII characters.
2. Doctest results containing non-ASCII characters.
3. Print statements in doctests outputting unicode.
4. Source files with a PEP 0263 encoding declaration.

see also #18370

CC: @nexttime @embray @jdemeyer @hivert

Component: doctest framework

Keywords: unicode

Author: Frédéric Chapoton

Branch/Commit: ed1e3b5

Reviewer: Erik Bray, Volker Braun, David Roe

Issue created by migration from https://trac.sagemath.org/ticket/14153

[jdemeyer]

Description changed:

--- 
+++ 
@@ -1,5 +1,5 @@
 Support
 1. Doctest (i.e. `sage:`) lines containing non-ASCII characters.
 2. Doctest results containing non-ASCII characters.
-3. Print statements outputting non-ASCII characters inside doctests.
+3. Print statements in doctests outputting unicode.
 4. Source files with a [PEP 0263](http://www.python.org/dev/peps/pep-0263/) encoding declaration.

[nexttime]

comment:2

Attachment: doctest_unicode.patch.gz

[kiwifb]

comment:3

I just tried this patch on the suspicion it could help with the test weirdness I reported in 5.9.beta0 on sage-release

sage -t --long devel/sage-main/sage/misc/interpreter.py
**********************************************************************
File "devel/sage-main/sage/misc/interpreter.py", line 150, in 
sage.misc.interpreter.sage_prompt
Failed example:
    shell.run_cell('sage_prompt()')
Expected:
    u'sage'
Got:
    u'sage'
**********************************************************************
File "devel/sage-main/sage/misc/interpreter.py", line 187, in 
sage.misc.interpreter.SageInteractiveShell.system_raw
Failed example:
    shell.system_raw('R --version')
Expected:
    R version ...
Got:
    WARNING: ignoring environment value of R_HOME
    R version 2.15.2 (2012-10-26) -- "Trick or Treat"
    Copyright (C) 2012 The R Foundation for Statistical Computing
    ISBN 3-900051-07-0
    Platform: x86_64-unknown-linux-gnu (64-bit)
    <BLANKLINE>
    R is free software and comes with ABSOLUTELY NO WARRANTY.
    You are welcome to redistribute it under the terms of the
    GNU General Public License versions 2 or 3.
    For more information about these matters see
    http://www.gnu.org/licenses/.
    <BLANKLINE>
**********************************************************************
File "devel/sage-main/sage/misc/interpreter.py", line 566, in 
sage.misc.interpreter.interface_shell_embed
Failed example:
    shell.run_cell('List( [1..10], IsPrime )')
Expected:
    [ false, true, true, false, true, false, true, false, false, false ]
Got:
    [ false, true, true, false, true, false, true, false, false, false ]
**********************************************************************
3 items had failures:
   1 of  15 in sage.misc.interpreter.SageInteractiveShell.system_raw
   1 of   4 in sage.misc.interpreter.interface_shell_embed
   1 of   4 in sage.misc.interpreter.sage_prompt
    [107 tests, 3 failures, 2.4 s]

The R error was fixed by unsetting R_HOME before running the test. Then I applied the patch and here is the result

sage -t --long devel/sage-main/sage/misc/interpreter.py
**********************************************************************
File "devel/sage-main/sage/misc/interpreter.py", line 150, in sage.misc.interpreter.sage_prompt
Failed example:
    shell.run_cell('sage_prompt()')
Expected:
    u'sage'
Got:
    u'sage'
    ---------------------------------------------------------------------------
    AttributeError                            Traceback (most recent call last)
    <ipython-input-1-e17c2c2153ac> in <module>()
    ----> 1 sage_prompt()
    
    /home/work/fbissey/sandbox/sage-5.9.beta0/local/lib/python2.7/site-packages/IPython/core/displayhook.pyc in __call__(self, result)
        240             self.update_user_ns(result)
        241             self.log_output(format_dict)
    --> 242             self.finish_displayhook()
        243 
        244     def flush(self):
    <BLANKLINE>
    /home/work/fbissey/sandbox/sage-5.9.beta0/local/lib/python2.7/site-packages/IPython/core/displayhook.pyc in finish_displayhook(self)
        224         """Finish up all displayhook activities."""
        225         io.stdout.write(self.shell.separate_out2)
    --> 226         io.stdout.flush()
        227 
        228     def __call__(self, result=None):
    <BLANKLINE>
    AttributeError: IOStream instance has no attribute 'flush'
**********************************************************************
File "devel/sage-main/sage/misc/interpreter.py", line 566, in sage.misc.interpreter.interface_shell_embed
Failed example:
    shell.run_cell('List( [1..10], IsPrime )')
Expected:
    [ false, true, true, false, true, false, true, false, false, false ]
Got:
    [ false, true, true, false, true, false, true, false, false, false ]
    ---------------------------------------------------------------------------
    AttributeError                            Traceback (most recent call last)
    <ipython-input-1-a772973ec1ce> in <module>()
    ----> 1 sage.misc.all.logstr("""[ false, true, true, false, true, false, true, false, false, false ]""")
    
    /home/work/fbissey/sandbox/sage-5.9.beta0/local/lib/python2.7/site-packages/IPython/core/displayhook.pyc in __call__(self, result)
        240             self.update_user_ns(result)
        241             self.log_output(format_dict)
    --> 242             self.finish_displayhook()
        243 
        244     def flush(self):
    <BLANKLINE>
    /home/work/fbissey/sandbox/sage-5.9.beta0/local/lib/python2.7/site-packages/IPython/core/displayhook.pyc in finish_displayhook(self)
        224         """Finish up all displayhook activities."""
        225         io.stdout.write(self.shell.separate_out2)
    --> 226         io.stdout.flush()
        227 
        228     def __call__(self, result=None):
    <BLANKLINE>
    AttributeError: IOStream instance has no attribute 'flush'
**********************************************************************
2 items had failures:
   1 of   4 in sage.misc.interpreter.interface_shell_embed
   1 of   4 in sage.misc.interpreter.sage_prompt
    [107 tests, 2 failures, 3.0 s]