Debugging tests using pytest shows"TypeError: message must be set" message

[ChadBailey]

Environment data

Version: 1.49.1 (user setup)
Commit: 58bb7b2331731bf72587010e943852e13e6fd3cf
Date: 2020-09-16T23:27:51.792Z
Electron: 9.2.1
Chrome: 83.0.4103.122
Node.js: 12.14.1
V8: 8.3.110.13-electron.0
OS: Windows_NT x64 10.0.18363

• VS Code version: 1.4
• Extension version (available under the Extensions sidebar): v2020.9.111407
• OS and version: Windows 10
• Python version (& distribution if applicable, e.g. Anaconda): 3.8.5 (64 bit)
• Type of virtual environment used (N/A | venv | virtualenv | conda | ...): N/A
• Relevant/affected Python packages and their versions: N/A
• Relevant/affected Python-related VS Code extensions and their versions: N/A
• Value of the python.languageServer setting: 0.5.51.0

Full language server log

[Info  - 6:26:45 PM] Analysis cache path: C:\Users\Chad\AppData\Local\Microsoft\Python Language Server
[Info  - 6:26:45 PM] Microsoft Python Language Server version 0.5.51.0
[Info  - 6:26:45 PM] Workspace root: c:\Users\Chad\Documents\code\VSCode_Debug
[Info  - 6:26:46 PM] GetCurrentSearchPaths C:\Python38\python.exe 
[Info  - 6:26:46 PM] Interpreter search paths:
[Info  - 6:26:46 PM]     c:\python38\dlls
[Info  - 6:26:46 PM]     c:\python38\lib
[Info  - 6:26:46 PM]     c:\python38
[Info  - 6:26:46 PM]     c:\users\chad\appdata\roaming\python\python38\site-packages
[Info  - 6:26:46 PM]     c:\python38\lib\site-packages
[Info  - 6:26:46 PM] User search paths:
2020-09-23 17:26:46.6932135 [W:onnxruntime:, graph.cc:814 onnxruntime::Graph::Graph] Initializer embeddings_matrix:0_quantized appears in graph inputs and will not be treated as constant value/weight. This may fail some of the graph optimizations, like const folding. Move it out of graph inputs if there is no need to override it, by either re-generating the model with latest exporter/converter or with the tool onnxruntime/tools/python/remove_initializer_from_input.py.
2020-09-23 17:26:46.6934091 [W:onnxruntime:, graph.cc:814 onnxruntime::Graph::Graph] Initializer embeddings_matrix:0_scale appears in graph inputs and will not be treated as constant value/weight. This may fail some of the graph optimizations, like const folding. Move it out of graph inputs if there is no need to override it, by either re-generating the model with latest exporter/converter or with the tool onnxruntime/tools/python/remove_initializer_from_input.py.
2020-09-23 17:26:46.6934436 [W:onnxruntime:, graph.cc:814 onnxruntime::Graph::Graph] Initializer embeddings_matrix:0_zero_point appears in graph inputs and will not be treated as constant value/weight. This may fail some of the graph optimizations, like const folding. Move it out of graph inputs if there is no need to override it, by either re-generating the model with latest exporter/converter or with the tool onnxruntime/tools/python/remove_initializer_from_input.py.
2020-09-23 17:26:46.6934786 [W:onnxruntime:, graph.cc:814 onnxruntime::Graph::Graph] Initializer fixed_quantization_range_uint8 appears in graph inputs and will not be treated as constant value/weight. This may fail some of the graph optimizations, like const folding. Move it out of graph inputs if there is no need to override it, by either re-generating the model with latest exporter/converter or with the tool onnxruntime/tools/python/remove_initializer_from_input.py.
2020-09-23 17:26:46.6935089 [W:onnxruntime:, graph.cc:814 onnxruntime::Graph::Graph] Initializer fixed_zero appears in graph inputs and will not be treated as constant value/weight. This may fail some of the graph optimizations, like const folding. Move it out of graph inputs if there is no need to override it, by either re-generating the model with latest exporter/converter or with the tool onnxruntime/tools/python/remove_initializer_from_input.py.
[Info  - 6:26:46 PM] Initializing for C:\Python38\python.exe
[Info  - 6:26:46 PM] Analysis caching mode: None.

Expected behaviour

No error

Actual behaviour

IMPORTANT: This behavior is intermittent and I'm unable to reliably reproduce it, but it does happen frequent enough that it should come up in testing.

Receiving error Error: TypeError: message must be set in Python Test Log when debugging tests (pytest) with accompanying error toast (see image)

Also occasionally receiving error:

Error: Error: Illegal value for `line`

Steps to reproduce:

[NOTE: Self-contained, minimal reproducing code samples are extremely helpful and will expedite addressing your issue]

1. Create the following files

main.py

def main():
    raise Exception("Unhandled Exception")

if __name__ == "__main__":
    main()

main_test.py

from main import main

def test_main():
    main()
    assert True

2. Discover tests
3. Debug test (individually or all seems to have no impact)

Logs

============================= test session starts =============================
platform win32 -- Python 3.8.5, pytest-6.0.2, py-1.9.0, pluggy-0.13.1
rootdir: c:\Users\Chad\Documents\code\VSCode_Debug
collected 2 items

test_main.py F.                                                        [100%]

================================== FAILURES ===================================
__________________________________ test_main __________________________________

    def test_main():
>       main()

test_main.py:4:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

    def main():
>       raise Exception("Unhandled Exception")
E       Exception: Unhandled Exception

main.py:2: Exception
- generated xml file: C:\Users\Chad\AppData\Local\Temp\tmp-1176wqdg9vDcmu6N.xml -
=========================== short test summary info ===========================
FAILED test_main.py::test_main - Exception: Unhandled Exception
========================= 1 failed, 1 passed in 6.85s =========================
Error: TypeError: message must be set

[lazarillo]

I had the same error:

Error: TypeError: message must be set

I did nothing else except try to run a particular unit test within a test file via the test debugger.

Details:

Version: 1.49.1
Commit: 58bb7b2331731bf72587010e943852e13e6fd3cf
Date: 2020-09-16T23:21:17.533Z
Electron: 9.2.1
Chrome: 83.0.4103.122
Node.js: 12.14.1
V8: 8.3.110.13-electron.0
OS: Darwin x64 19.6.0

[kimadeline]

Thank you for reaching out @ChadBailey and @lazarillo! Can you reproduce this issue with multiple versions of pytest, or is it happening only when using 6.0.2?

As a side note, if other users come forward and leave a comment demonstrating they are seeing/reproducing the problem, we will raise this issue's priority.

Thanks!

[seanwsmithm3]

I am having the exact same problem using:

[04:47 PM] >pytest --version
pytest 6.0.2

[04:47 PM] >python --version
Python 3.6.9

It runs my tests and then errors after pytest completes....

========================= 4 passed, 3 xfailed in 0.32s =========================
============================= test session starts ==============================
platform linux -- Python 3.6.9, pytest-6.0.2, py-1.9.0, pluggy-0.13.1
rootdir: /home/ssmith/work/repos/ci-tools, configfile: tests/pytest.ini
plugins: xdist-2.1.0, metadata-1.10.0, forked-1.3.0, html-2.1.1
collected 7 items

tests/test_base_executor.py ....xxx [100%]

-------------- generated xml file: /tmp/tmp-2792E648AgEDEUge.xml ---------------
========================= 4 passed, 3 xfailed in 0.33s =========================
Error: TypeError: message must be set

[kimadeline]

Interestingly, I could reproduce the issue if I pick Debug All Tests, then after that any debug command will show the error message (Debug Test Method or a Debug Test code lens). I cannot repro the issue when I only run the tests (via the Run All Tests command or a Run Test code lens), or when I run tests after using Debug All Tests.

@seanwsmithm3 and @lazarillo do you have a sample repo I could use to try and replicate your issue when running the test?

For everyone: What is the output of running in the terminal the equivalent to the command ran by the extension:

path/to/python -m pytest --override-ini junit_family=xunit1 --rootdir path/to/workspace/folder --junit-xml=/path/to/a/random/xmlfile.xml .

[int19h]

Take a look at microsoft/vscode-cmake-tools#1432, and the associated PR.

[kimadeline]

That's excellent timing @int19h, because I was doing some digging and I was reaching the same conclusion: diagnostics must have a message. In our current code the message (on line 500) can sometimes be '', which is not enough, as evidenced by microsoft/vscode-cmake-tools#1433, many thanks for the link 🙂

vscode-python/src/client/testing/common/managers/baseTestManager.ts

Lines 498 to 505 in 61b179b

	const diagPrefix = this.unitTestDiagnosticService.getMessagePrefix(message.status!);
	const severity = this.unitTestDiagnosticService.getSeverity(message.severity)!;
	const diagMsg = message.message ? message.message.split('\n')[0] : '';
	const diagnostic = new Diagnostic(
	stackStart.location.range,
	`${diagPrefix ? `${diagPrefix}: ` : ''}${diagMsg}`,
	severity
	);

Alright everyone, no need for info anymore, the problematic area has been narrowed down 🔍

---

@ChadBailey (sorry for the many pings), if you have a reliable repro case for the Error: Error: Illegal value for 'line' that would be really helpful 🙂

Thank you everyone!

[ChadBailey]

@kimadeline no worries on the pings, glad I've been able to contribute in some way...

That said, exact same IDE and exact same code on exact same machine as bug report was filed on and I can no longer reproduce. I also am no longer experiencing the [desirable] behavior of breaking on unhandled exceptions [see #14056].

[ChadBailey]

I've been unable to reproduce the Error: Error: Illegal value for 'line' error, however, I did find a way to reliably reproduce the original error message. I suspect this may not be needed, but posting here just in case.

My config:
.vscode/settings.json

{
  "python.pythonPath": "/usr/local/bin/python3",
  "python.testing.pytestArgs": ["."],
  "python.testing.unittestEnabled": false,
  "python.testing.nosetestsEnabled": false,
  "python.testing.pytestEnabled": true
}

1. Create test_main.py

2. Create two simple test cases for pass & fail. This is what I've been using:

def test_this():
  assert True

def test_that():
  assert False

3. Do any of the following to ensure at least one test shows passing: (no error will be shown)
   a. Visit test explorer and click >> to run all tests
   b. Click > by test_main.py or test_this
   c. Click Run Test directly above def test_this():

NOTE: It's worth mentioning that attempting to run only a failing test will not cause this behavior.

4. Do any of the following to debug the passing test:
   a. Visit test explorer and click debug button to debug all tests
   b. Click debug by test_main.py or test_this
   c. Click Debug Test directly above def test_this():

5. Observe error message must be set now begins showing each time any test is debugged, and when the failing test is ran.

NOTE: Running (but not debugging) the passing test does not generate this error.

6. All subsequent run (failed) and debug (pass or fail) attempts will result in this error, however, issuing the command > Developer: Reload Window always fully resolves until conditions are met to cause the error to resurface.

[chs-sapentintia]

@ChadBailey Just wanted to add that I am seeing this bug as well, and it's affecting the outcome of my tests. Although the unit tests pass if it's in debug mode or if they are just running normally, I decided to write the output of my tests to a JSON file. The two are producing different files despite both passing the tests. I don't know if this has any implications as to what the issue could be.

Forgive the poor quality of the gif, but this should show what I'm talking about better than I can explain it:

[kimadeline]

Hi @chs-sapentintia, if you havne't done so yet please upvote (👍) the initial post to help us prioritize it. Thanks!

[GringoDotDev]

In case it helps anyone else, I was experiencing this issue (inside an attached Docker container) and resolved it by setting justMyCode: false and restarting the window.

[cuent]

I found the same problem.

I just created a test that compares the resulting output from a file.

[joelricci]

In case it helps anyone else, I was experiencing this issue (inside an attached Docker container) and resolved it by setting justMyCode: false and restarting the window.

I'm seeing this in Debug Console for every test:

Frame skipped from debugging during step-in.
Note: may have been skipped because of "justMyCode" option (default == true). 
Try setting "justMyCode": false in the debug configuration (e.g., launch.json).

Despite having set justMyCode: false in launch.json as well as in the general config (user and workspace). Don't know if it's related but seems like an error.

[itewk]

for me the issue was having set in python.testing.pytestargs

--cov=mypck
--cov-report=xml:cov.xml
tests/

after i removed all the params debuging was working

[VladimirDomnich]

I am observing the same issue too.

[StudyExchange]

I am observing the same issue too. And my local dev environment is not work.

[briangrahamww]

I had the same issue and fixed it by adding "--no-cov" to python.testing.pytestargs. For me the issue is that the debugger is using the settings in my pytest.ini file in my project root directory where I have some pytest-cov settings. I needed to override this with --no-cov as mentioned in the vscode docs here. Perhaps other settings in pytests.ini could be the culprit if adding --no-cov does not resolve the issue.

[singleheart]

Thanks, briangrahamww! Adding "--no-cov" removed the warning "TypeError: message must be set".

[luabud]

This should be fixed in the insiders version of the Python extension (View > Command Palette... and run "Python: Switch to Insiders Weekly Channel" command). Please do try it out, we will release this fix soon!

[luabud]

This should have been out on the December release. If you still see this problem with the newest version let us know.