Port GDScript test/debugging tools #41355

Xrayez · 2020-08-18T16:34:45Z

Also includes a disassembler for the current VM which I used to debug the code generation. It can't be accessed currently since the command for the old tests isn't available anymore, but this is useful to have for when we have a new way to trigger those.

You can now run those debug/test tools via command-line with this PR:

godot --test gdscript-tokenizer /path/to/test.gd
godot --test gdscript-parser /path/to/test.gd
godot --test gdscript-compiler /path/to/test.gd

You have to compile the engine with scons tests=yes in order to be able to use this (if you use dev=yes, you're already compiling the engine with tests).

Implementation details

I haven't modified the inner workings of those existing tools, except for the removed MainLoop since it doesn't require it for testing. This should work now as core functionality is initialized with #40980. It's also much faster to iterate because most non-core subsystems are not initialized (rendering, audio etc.), so it kind of works like --no-window option on Windows, visually.

doctest tests are skipped entirely if any such test tool is detected. I'm also reusing the dynamic initialization technique as seen in doctest, with a similar approach.

Example output so you can see it actually works:

PS D:\src\godot\4> .\bin\godot.windows.tools.64.exe --test gdscript-tokenizer D:/src/godot/4/test.gd
0001 extends Node
     ^^^^^^^
 --> extends
-------------------------------------------------------
0001 extends Node
             ^^^^
 --> Identifier(StringName) Node
-------------------------------------------------------
0001 extends Node
                 ^
 --> Newline
-------------------------------------------------------
0003 func _ready():
     ^^^^
 --> func

...and so on...

GDScript compiler will be provided by #41338. I moved those tools to modules/gdscript/tests so will need rebase in either case.

Xrayez · 2020-08-18T16:38:39Z

tests/SCsub

-# Enable test framework and inform it of configuration method.
-env_tests.Append(CPPDEFINES=["DOCTEST_CONFIG_IMPLEMENT"])
-


I removed this and define DOCTEST_CONFIG_IMPLEMENT in test_macros.cpp once. Including doctest header twice in source files doesn't protect the implementation to be included once, and leads to linker errors with duplicate symbols.

Xrayez · 2020-08-18T16:41:53Z

tests/test_main.cpp

+	for (int i = 0; i < argc; i++) {
+		args.push_back(String::utf8(argv[i]));
+	}
+	OS::get_singleton()->set_cmdline("", args);


The first argument is empty because there's no way to know the executable name with the existing testing interface, but I'm not sure whether that's essential in the first place.

akien-mga · 2020-09-01T21:34:03Z

Needs a rebase, then it can be merged.

akien-mga · 2020-09-01T21:35:20Z

modules/gdscript/SCsub

@@ -17,3 +17,7 @@ if env["tools"]:
        # Using a define in the disabled case, to avoid having an extra define
        # in regular builds where all modules are enabled.
        env_gdscript.Append(CPPDEFINES=["GDSCRIPT_NO_LSP"])
+
+if env["tests"]:
+    env_gdscript.Append(CPPDEFINES=["TESTS_ENABLED"])


Is this necessary? It should already be inherited from the main env which was cloned for env_gdscript no?

Yes, because we're actually outside of the main/ env here. Recall #40726, this is what I meant when saying that some modules may eventually need this. Defining TESTS_ENABLED for all modules is also an option, but GDScript may be just an exception.

Technically, those are more like debug tools the way I see it, but they still qualify as something which facilitate manual testing process.

akien-mga · 2020-09-01T21:37:09Z

tests/test_macros.cpp

+#include "test_macros.h"
+
+#define DOCTEST_CONFIG_IMPLEMENT
+#include "thirdparty/doctest/doctest.h"


Won't this include doctest.h first without the define (via test_macros.h), and then a second time with the define?

OK, removed doctest header.

Xrayez · 2020-09-01T23:27:39Z

Applied suggested changes and rebased.

I haven't reported this before (to test out compiler after rebase as well), but I've revealed some crashes by testing out a simple script, they all happen in GDScript code, so I'm not sure whether they are caused by running commands in the test environment or whether those crashes simply need a fix (CC @vnen):

# test.gd (at Godot source root)

extends Node

func _ready():
    pass

with the following commands:

--test gdscript-tokenizer test.gd

I suspect this may have something to do with off-by-one error.

godot.windows.tools.64.exe!CowData<String>::get(int p_index) Line 151 (d:\src\godot\4\core\cowdata.h:151)
godot.windows.tools.64.exe!Vector<String>::operator[](int p_index) Line 89 (d:\src\godot\4\core\vector.h:89)
godot.windows.tools.64.exe!TestGDScript::test_tokenizer(const String & p_code, const Vector<String> & p_lines) Line 67 (d:\src\godot\4\modules\gdscript\tests\test_gdscript.cpp:67)
godot.windows.tools.64.exe!TestGDScript::test(TestGDScript::TestType p_type) Line 219 (d:\src\godot\4\modules\gdscript\tests\test_gdscript.cpp:219)
godot.windows.tools.64.exe!test_tokenizer() Line 165 (d:\src\godot\4\modules\gdscript\register_types.cpp:165)
godot.windows.tools.64.exe!test_main(int argc, char * * argv) Line 74 (d:\src\godot\4\tests\test_main.cpp:74)
godot.windows.tools.64.exe!Main::test_entrypoint(int argc, char * * argv, bool & tests_need_run) Line 458 (d:\src\godot\4\main\main.cpp:458)
godot.windows.tools.64.exe!widechar_main(int argc, wchar_t * * argv) Line 149 (d:\src\godot\4\platform\windows\godot_windows.cpp:149)
godot.windows.tools.64.exe!_main() Line 185 (d:\src\godot\4\platform\windows\godot_windows.cpp:185)
godot.windows.tools.64.exe!main(int argc, char * * argv) Line 199 (d:\src\godot\4\platform\windows\godot_windows.cpp:199)
godot.windows.tools.64.exe!WinMain(HINSTANCE__ * hInstance, HINSTANCE__ * hPrevInstance, char * lpCmdLine, int nCmdShow) Line 213 (d:\src\godot\4\platform\windows\godot_windows.cpp:213)
[Inline Frame] godot.windows.tools.64.exe!invoke_main() Line 102 (d:\agent\_work\4\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl:102)
godot.windows.tools.64.exe!__scrt_common_main_seh() Line 288 (d:\agent\_work\4\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl:288)
kernel32.dll!00007ff80cd87bd4() (Unknown Source:0)
ntdll.dll!00007ff80e72ce51() (Unknown Source:0)

--test gdscript-parser test.gd

Works fine. 🙂

--test gdscript-compiler test.gd

Uninitialized map data root. If you add the following snippet, it would just prevent the crash there:

			if (GDScriptLanguage::get_singleton()->get_global_map().empty()) {
				break;
			}

godot.windows.tools.64.exe!Map<StringName,int,Comparator<StringName>,DefaultAllocator>::operator[](const StringName & p_key) Line 575 (d:\src\godot\4\core\map.h:575)
godot.windows.tools.64.exe!GDScriptCompiler::_parse_class_level(GDScript * p_script, const GDScriptParser::ClassNode * p_class, bool p_keep_state) Line 1917 (d:\src\godot\4\modules\gdscript\gdscript_compiler.cpp:1917)
godot.windows.tools.64.exe!GDScriptCompiler::compile(const GDScriptParser * p_parser, GDScript * p_script, bool p_keep_state) Line 2305 (d:\src\godot\4\modules\gdscript\gdscript_compiler.cpp:2305)
godot.windows.tools.64.exe!TestGDScript::test_compiler(const String & p_code, const String & p_script_path, const Vector<String> & p_lines) Line 156 (d:\src\godot\4\modules\gdscript\tests\test_gdscript.cpp:156)
godot.windows.tools.64.exe!TestGDScript::test(TestGDScript::TestType p_type) Line 225 (d:\src\godot\4\modules\gdscript\tests\test_gdscript.cpp:225)
godot.windows.tools.64.exe!test_compiler() Line 173 (d:\src\godot\4\modules\gdscript\register_types.cpp:173)
godot.windows.tools.64.exe!test_main(int argc, char * * argv) Line 74 (d:\src\godot\4\tests\test_main.cpp:74)
godot.windows.tools.64.exe!Main::test_entrypoint(int argc, char * * argv, bool & tests_need_run) Line 458 (d:\src\godot\4\main\main.cpp:458)
godot.windows.tools.64.exe!widechar_main(int argc, wchar_t * * argv) Line 149 (d:\src\godot\4\platform\windows\godot_windows.cpp:149)
godot.windows.tools.64.exe!_main() Line 185 (d:\src\godot\4\platform\windows\godot_windows.cpp:185)
godot.windows.tools.64.exe!main(int argc, char * * argv) Line 199 (d:\src\godot\4\platform\windows\godot_windows.cpp:199)
godot.windows.tools.64.exe!WinMain(HINSTANCE__ * hInstance, HINSTANCE__ * hPrevInstance, char * lpCmdLine, int nCmdShow) Line 213 (d:\src\godot\4\platform\windows\godot_windows.cpp:213)
[Inline Frame] godot.windows.tools.64.exe!invoke_main() Line 102 (d:\agent\_work\4\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl:102)
godot.windows.tools.64.exe!__scrt_common_main_seh() Line 288 (d:\agent\_work\4\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl:288)
kernel32.dll!00007ff80cd87bd4() (Unknown Source:0)
ntdll.dll!00007ff80e72ce51() (Unknown Source:0)

Otherwise my changes are complete.

akien-mga · 2020-09-02T07:39:26Z

Let's merge as is and then work on fixing the identified issues, which don't seem to be introduced by this PR. Could you open issues for them?

akien-mga · 2020-09-02T07:39:32Z

Thanks!

Xrayez requested review from bojidar-bg and vnen as code owners August 18, 2020 16:34

Xrayez commented Aug 18, 2020

View reviewed changes

Calinou added enhancement topic:tests labels Aug 18, 2020

Calinou added this to the 4.0 milestone Aug 18, 2020

vnen approved these changes Sep 1, 2020

View reviewed changes

akien-mga reviewed Sep 1, 2020

View reviewed changes

akien-mga requested a review from RevoluPowered September 1, 2020 21:37

Andrii Doroshenko (Xrayez) added 2 commits September 2, 2020 01:09

Move GDScript tests to respective folder under modules

6b7e50a

Register GDScript test tools as test commands to run via command-line

0cc05c5

akien-mga merged commit 3922883 into godotengine:master Sep 2, 2020

Xrayez deleted the port-gdscript-tests branch September 2, 2020 08:25

vnen mentioned this pull request Oct 20, 2020

Add initialization for the language before GDScript debug tests #42938

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Port GDScript test/debugging tools #41355

Port GDScript test/debugging tools #41355

Xrayez commented Aug 18, 2020 •

edited

Loading

Xrayez Aug 18, 2020

Xrayez Aug 18, 2020 •

edited

Loading

akien-mga commented Sep 1, 2020

akien-mga Sep 1, 2020

Xrayez Sep 1, 2020 •

edited

Loading

akien-mga Sep 1, 2020

Xrayez Sep 1, 2020

Xrayez commented Sep 1, 2020

akien-mga commented Sep 2, 2020

akien-mga commented Sep 2, 2020

		# Enable test framework and inform it of configuration method.
		env_tests.Append(CPPDEFINES=["DOCTEST_CONFIG_IMPLEMENT"])

Port GDScript test/debugging tools #41355

Port GDScript test/debugging tools #41355

Conversation

Xrayez commented Aug 18, 2020 • edited Loading

Implementation details

Xrayez Aug 18, 2020

Choose a reason for hiding this comment

Xrayez Aug 18, 2020 • edited Loading

Choose a reason for hiding this comment

akien-mga commented Sep 1, 2020

akien-mga Sep 1, 2020

Choose a reason for hiding this comment

Xrayez Sep 1, 2020 • edited Loading

Choose a reason for hiding this comment

akien-mga Sep 1, 2020

Choose a reason for hiding this comment

Xrayez Sep 1, 2020

Choose a reason for hiding this comment

Xrayez commented Sep 1, 2020

--test gdscript-tokenizer test.gd

--test gdscript-parser test.gd

--test gdscript-compiler test.gd

akien-mga commented Sep 2, 2020

akien-mga commented Sep 2, 2020

Xrayez commented Aug 18, 2020 •

edited

Loading

Xrayez Aug 18, 2020 •

edited

Loading

Xrayez Sep 1, 2020 •

edited

Loading