LWJGL_BUILD_OFFLINE=true results in releases being build with incorrect native binaries

[theofficialgman]

Version

3.3.0 (nightly), 3.2.3

Platform

Linux x64, Linux arm64, Linux arm32

JDK

OpenJDK 11

Module

All

Bug description

following these buildsteps:

git clone -b 3.2.3 https://github.com/LWJGL/lwjgl3.git
cd lwjgl3
export LWJGL_BUILD_TYPE=release/3.2.3

# use java 11 preferably
export JAVA_HOME=/usr/lib/jvm/java-11-openjdk-arm64
export LWJGL_BUILD_ARCH=arm64
ant compile-templates
ant compile
ant compile-native
export JAVA8_HOME=/usr/lib/jvm/java-11-openjdk-arm64
export LWJGL_BUILD_OFFLINE=true
ant release -Dbuild.revision=1 -Dbuild.version=3.2.3

the bin/RELEASE/modulename folders contain lwjgl-modulename-natives-linux.jar, lwjgl-modulename-natives-linux-arm32.jar, lwjgl-modulename-natives-linux-arm64.jar files. NOTE, this was on offline release, no binaries were downloaded (verified that), and all of those jars contain the ARM64 binaries. I believe this must be a problem with the generateNativeModuleInfoClasses that is causing the creation of all three architectures and the copying of the available arm64 binaries to all 3 folders which get packed into jars.

lwjgl3/modules/generator/src/main/java/org/lwjgl/generator/util/ModuleInfoGen.java

Lines 164 to 224 in 18ef45b

	private static void generateNativeModuleInfoClasses() throws IOException {
	String moduleNameSource = System.getProperty("module.name.source");
	String moduleNameRelease = System.getProperty("module.name.release");
	if (moduleNameSource == null || moduleNameRelease == null) {
	throw new IllegalStateException("Module source & release names must be specified.");
	}
	Path root = Paths.get("bin", "RELEASE", moduleNameRelease, "native");
	Module module = parseModuleInfo(Files.readAllBytes(Paths.get("modules", "lwjgl", moduleNameSource, "src", "main", "resources", "module-info.java")));
	String moduleNative = module.nameJava + ".natives";
	String moduleVersion = System.getProperty("module.version");
	try (ModuleInfoGen gen = new ModuleInfoGen()) {
	try (Stream<Path> platforms = Files.list(root)) {
	platforms
	.filter(Files::isDirectory)
	.forEach(platform -> {
	try {
	try (Stream<Path> architectures = Files.list(platform)) {
	architectures
	.filter(Files::isDirectory)
	.forEach(architecture -> {
	String nativePackage = platform.getFileName().toString() + '.' + architecture.getFileName() + '.' + module.nameJava;
	Path outputPath = architecture.resolve(METAINF);
	gen.compile(
	moduleNative,
	moduleVersion,
	"module " + moduleNative + " {\n" +
	" requires transitive " + module.nameJava + ";\n" +
	"\n" +
	" opens " + nativePackage + ";\n" +
	"}",
	Stream.concat(Stream.of(module.name), module.dependencies.stream().map(it -> it.name))
	.map(it -> "bin/classes/lwjgl/" + it + "/META-INF/versions/9")
	.collect(Collectors.joining(File.pathSeparator)),
	architecture,
	outputPath,
	nativePackage
	);
	try (Stream<Path> dummy = Files.walk(outputPath.resolve(platform.getFileName()))) {
	dummy
	.sorted(Comparator.reverseOrder())
	.map(Path::toFile)
	.forEach(File::delete);
	} catch (IOException e) {
	e.printStackTrace();
	}
	});
	}
	} catch (IOException e) {
	throw new RuntimeException(e);
	}
	});
	}
	}
	}

bin/linux/arm64 and bin/libs/native/linux/arm64 are the only folders made from the ant compile-native command so that is working correctly, it is just an issue with the ant release

this issue probably also happens on windows/macos but is untested

Stacktrace or crash log output

No response

[theofficialgman]

actually here is the issue:
lib.native is static and will always contain the ARM64 path as set in the LWJGL_BUILD_ARCH command:

lwjgl3/config/build-definitions.xml

Line 135 in ad60275

<property name="lib.native" location="${lib}/native/${platform}/${build.arch}" relative="true"/>

however, the copy command uses this static definition and copies the ARM64 file over to the changing @{arch} variable depending on the input to the get-release-offline macro

lwjgl3/build.xml

Lines 1043 to 1045 in ad60275

	<echo message="release: ${lib.native}/${module.path}/@{file} is missing" level="warning" unless:set="available"/>
	<quiet if:set="available">
	<copy file="${lib.native}/${module.path}/@{file}" todir="${release}/${module}/native/${platform}/@{arch}/${module.path}"/>

hence why arm64 binaries end up in x86_64 and arm32 jars

[theofficialgman]

moving the definition should solve the issue

    <macrodef name="get-release-offline">
        <attribute name="platform"/>
        <attribute name="arch"/>
        <attribute name="file"/>

        <sequential>
            <local name="available"/>
            <available file="${lib}/native/${platform}/@{arch}/${module.path}/@{file}" property="available"/>

            <echo message="release: ${lib}/native/@{platform}/@{arch}/${module.path}/@{file} is missing" level="warning" unless:set="available"/>
            <quiet if:set="available">
                <copy file="${lib}/native/${platform}/@{arch}/${module.path}/@{file}" todir="${release}/${module}/native/@{platform}/@{arch}/${module.path}"/>
            </quiet>
        </sequential>
    </macrodef>

    <macrodef name="get-release">
        <attribute name="platform"/>
        <attribute name="arch"/>
        <attribute name="file"/>

        <sequential>
            <quiet>
                <mkdir dir="${release}/${module}/native/@{platform}/@{arch}/${module.path}"/>
            </quiet>
            <get
                src="https://build.lwjgl.org/${build.type}/${platform.@{platform}.remote}/@{arch}/@{file}"
                dest="${release}/${module}/native/@{platform}/@{arch}/${module.path}"
                usetimestamp="true"
                unless:true="${build.offline}"
            />
            <get-release-offline platform="@{platform}" arch="@{arch}" file="@{file}" if:true="${build.offline}"/>
        </sequential>
    </macrodef>

note there may be better ways to solve this and there may be other places where lib.native is used incorrectly such as here

[theofficialgman]

theofficialgman@e04a910
(the above commit applies to 3.2.3+ on this repo but I made it on a 3.2.2 backport branch needed for another project)

[Spasi]

@theofficialgman Thanks, offline mode should produce correct artifacts now!