Fabric Game Test API

[modmuss50]

// optional to impl FabricGameTest
public class ExampleFabricTestSuite implements FabricGameTest {
	/**
	 * By overriding invokeTestMethod you can wrap the method call.
	 * This can be used as shown to run code before and after each test.
	 */
	@Override
	public void invokeTestMethod(TestContext context, Method method) {
		beforeEach(context);

		FabricGameTest.super.invokeTestMethod(context, method);

		afterEach(context);
	}

	private void beforeEach(TestContext context) {
		System.out.println("Hello beforeEach");
		context.setBlockState(0, 5, 0, Blocks.GOLD_BLOCK);
	}

	private void afterEach(TestContext context) {
		context.addInstantFinalTask(() ->
				context.checkBlock(new BlockPos(0, 5, 0), (block) -> block == Blocks.GOLD_BLOCK, "Expect block to be gold")
		);
	}

	@GameTest(structureName = "fabric-game-test-api-v1-testmod:exampletestsuite.diamond")
	public void diamond(TestContext context) {
		context.addInstantFinalTask(() ->
				context.checkBlock(new BlockPos(0, 2, 0), (block) -> block == Blocks.DIAMOND_BLOCK, "Expect block to be diamond")
		);
	}

	@GameTest(structureName = EMPTY_STRUCTURE)
	public void noStructure(TestContext context) {
		context.setBlockState(0, 2, 0, Blocks.DIAMOND_BLOCK);

		context.addInstantFinalTask(() ->
				context.checkBlock(new BlockPos(0, 2, 0), (block) -> block == Blocks.DIAMOND_BLOCK, "Expect block to be diamond")
		);
	}
}

This example has 2 tests, the first one uses a structure that can be manually built and exported using the test command. The 2nd test uses an empty structure provided via the api, this allows you to write code only tests. invokeTestMethod can be used to customise and extend the test method invocation, useful for before/after methods or having a custom context.

These tests can easily be ran headless via gradle:

loom {
	runs {
		gametest {
			server()
			name = "Game Test"
			vmArg "-Dfabric-api.gametest.server=true"
		}
	}
}
test.dependsOn runGametest

Registered via entrypoints like so:

  "entrypoints": {
    "fabric-game-test" : [
      "net.fabricmc.fabric.test.test.ExampleFabricTestSuite",
      "net.fabricmc.fabric.test.test.ExampleTestSuite"
    ]
  }
``

[modmuss50]

Pushed what ive done for now, it should build and run.

I think im going to move over to using custom entrypoints to register the test suites, thats for another day though 👍

[liach]

On a side note, we should explore using custom game providers in the loader in the future for such purposes

[modmuss50]

Moving to another game provider opens a huge rabbit hole that I dont want to go into, this is a much neater solution imo.

[modmuss50]

Thinking about it, it might not be as much as I initally thought but loader needs quite a bit of work to allow extending the minecraft game provider from a mod and with a stable api that we could trust to depend on in fab api.

The current way is basically the same for modders but doesnt require massive amounts of work to create stable apis in loader. It will also be easier to run in a prod server. Doing it that was has little to nothing to gain, but a huge amount of added complexitly.

[liach]

Marking as low priority as this doesn't affect most game users or modders and there are other companion tasks as well, namely moving the test invocation from mixin on server main to a custom loader game provider.

[liach]

Why is this needed?

[modmuss50]

I forget as I wrote this months ago :D I will check and get back to you.

[modmuss50]

I think this is used to allow dedicated server commands to be registered. I think when I first wrote this I was running with all the fabric test mods and it would cause the server test to fail. Seems reasonable to leave enabled, I will add a comment.

[modmuss50]

Humm, just noticed this is using the internal loader api to do this, might need checking with 0.12 first.

[modmuss50]

This is going to break, going to back this out of last call for now untill a solution is found.

[Juuxel]

Out of curiosity, does this support the vanilla BeforeEach and AfterEach annotations?

[modmuss50]

Yes, those work just fine.

[Juuxel]

Using the normal structure path here would allow modders to use normal structures (eg. from their mods) in game tests. I wonder if that's something we want?

[modmuss50]

The test structures are all loaded from snbt files, and not the normal structure files. Personally I dont see the need to support that.

[FoxSamu]

GameTest automatically attempts to load data structures from data packs when it can't find a snbt structure in the gametest structures folder. This folder is just to make the custom test structures more easily version-controllable.

[FoxSamu]

Oh, realized I overlooked something here, ignore my comment. I thought this was changing the default gameteststructures folder but apparently not.

[Technici4n]

Isn't assert useless by default? It should probably be if (FabricLoader.getInstance().getGameInstance() != null) throw new AssertionError();

[modmuss50]

I might just remove this, was just a quick test tbh;

[liach]

Maybe we can just add -ea for vm args for test mod runs.

[Juuxel]

Yeah, that's enabled for the regular test task too.

[FoxSamu]

This XmlReportingTestCompletionListener generates a JUnit-like XML file, but it's seen as invalid by the EnricoMi/publish-unit-test-result-action@v1 GitHub Action (the easiest action to upload test results). Noticed this when testing with the GameTest framework in a custom mod (that adds a similar API), and wrote this workaround (in Mojang mappings):
https://github.com/FoxShadew/JustEnoughDebugTools/blob/master/src/main/java/net/shadew/debug/test/ProperJUnitLikeTestReporter.java

[FoxSamu]

As basic test structure I used a sand block that fell down, and asserted it being on the ground. Might be an inspiration for some more tests, but I see that for the sake of testing the testing api the tests themself don't really matter anyway. Just visually, the sand block test shows a visual feedback as soon as the block has landed.

[FoxSamu]

This API is more or less the same as what I implemented as part of a debugging mod/API I work on: Just Enough Debug Tools (JEDT), see most certainly this package.
This API appears to be significantly differently designed and I can't entirely get a grip on why things are as they are, but that's just me.

I wanna propose a couple of features:

• Running tests in a certain dimension: this might be hard to implement though, but it might be possible by making a custom TestServer class. It's worth it for the dimension mods that have certain features only work in their custom dimensions (because those can't be tested otherwise).
• A fix for the JUnit XML generator so that it works with GitHub Actions. I'm not sure what exactly I changed anymore, but I managed to fix it in a custom variant: https://github.com/FoxShadew/JustEnoughDebugTools/blob/master/src/main/java/net/shadew/debug/test/ProperJUnitLikeTestReporter.java.
• Support for custom test loggers, not only JUnit.

---

Not intending to advertise anything, but I recommend looking at the test part of JEDT for some comparison, which implements a fairly similar API for using the GameTest system. See these packages:

• https://github.com/FoxShadew/JustEnoughDebugTools/tree/master/src/main/java/net/shadew/debug/test (implementation)
• https://github.com/FoxShadew/JustEnoughDebugTools/tree/master/src/main/java/net/shadew/debug/mixin (mixins)

And some usages:

• https://github.com/FoxShadew/JustEnoughDebugTools/tree/master/gametest
• https://github.com/FoxShadew/JustEnoughDebugTools/tree/master/src/test/java/net/shadew/debug

I'll likely keep my testing API in JEDT for a while even after this PR gets merged, just to evaluate the differences.

[modmuss50]

I can't entirely get a grip on why things are as they are, but that's just me.

Please feel free to ask questions, ill be happy to try and explain my approach, might even find a better way of doing something 👍

Running tests in a certain dimension

Yes, I can see that beign useful, I think its out of scope for this first version but something we can look at doing later.

Expanding the intergration with existing test tools such gradle, junit or IDE's is definally something that I want to look into more, again this seems like something after this first version. I have had a quick look at your JEDT project, it looks good. I think it will still be useful as a few parts seem beyond fabric's scope.

[Ecorous]

Sorry, but I don't understand what this PR is for. Could someone explain it?

[modmuss50]

@ExoPlant this PR exposes the new* game testing framework to modders, this will allow modders to write automated tests for their mods.

https://www.youtube.com/watch?v=vXaWOJTCYNg is a good primer on what this is, and why and how mojang use it.