UKlibs and KMP Publication Scheme

[abdulowork]

This is an issue to discuss two KEEP proposals regarding the:

• New serialization format for KMP libraries
• Corresponding changes to the KMP publication scheme

[CLOVIS-AI]

Integration with variant-aware resolution

I think it is important to ensure that Gradle can still understand the different platforms as variants. This doesn't mean the UKLib format must be changed, but the necessary artifact transforms should be declared on the KGP side such that plugins and the ecosystem can continue dealing with variants.

An example of such usage is my plugin kotlin-js-resources (https://kotlin-resources.opensavvy.dev/api-docs/) which hacks into KGP to add a new resources variant for each platform which contains its src/xxx/resources directories and allows restoring them on the consumer side. While this specific functionality (accessing static resources from libraries in consumer projects) should be available through the UKLib format itself, I think it will happen again that users want to embed more data in the libraries, and the standard for that in Gradle world is through variant-aware resolution.

Multiple other projects have had similar needs. If I recall correctly, Kobweb needed to add some metadata to know if a library was specifically Kobweb-aware or not.

I can see this being useful to store other kinds of metadata, e.g. a list of services implemented by the module, service-loader style.

I don't think this needs to be within the UKLib file itself, I think it is perfectly acceptable that these other files are separate files behind the same Maven coordinates. However, I think care must be taken that KGP acts as if the contents of the UKLib were using dependency-aware resolution, such that all of these things can integrate together.

[CLOVIS-AI]

Another variant-aware Gradle feature is the avoidance of compression between local projects. If a project depends on another in the same build, Gradle skips generation of the JAR/KLib/whatever in favor of directly adding the generated binaries to the classpath. If this feature can work with UKLib as well, then users will only really need to produce UKLibs during publication to a registry and not during the regular development loop, making more expensive compression algorithms (in compression time) much more affordable.

[martinbonnin]

users want to embed more data in the libraries, and the standard for that in Gradle world is through variant-aware resolution.

I'd argue those users also want compatibility with non-Gradle resolvers. I would be OK to embedd that in the .uklib files themselves, not as separate maven artifacts. Just "allocate" a directory for third-party tools in the format? extra/kotlin-js-resources is owned by kotlin-js-resources, extra/kobweb by Kobweb, etc... And you get amper/maven/ant/bazel/jpm/etc... support for free.

[antohaby]

Hi @CLOVIS-AI, Thank you very much for such quick reaction! We will be happy to hear your future feedback about your experience with UKlibs prototype!

I think it is important to ensure that Gradle can still understand the different platforms as variants. This doesn't mean the UKLib format must be changed, but the necessary artifact transforms should be declared on the KGP side such that plugins and the ecosystem can continue dealing with variants.

It is true that existing JVM ecosystem should be still able to consume KMP libraries, as they were published by regular Maven or Gradle. And in UKlibs proposal we cover that. With Android the story is similar, and even simpler as data shows that Maven is not widely used in Android application development. So publishing AAR as additional gradle variant should be enough.

As for other platform, we don't have strong evidence that there are other ecosystems that live in Maven/Gradle world and that don't rely on Kotlin Multiplatform. And KMP publishes only regular platform-specific klibs for other targets: js, wasm, native.
And it is intended for consumption by KMP only. With this frame, KMP publication is just an implementation detail:

Jetbrains, as owner of KMP ecosystem will provide KMP Publication/Consumption API for KMP libraries (UKlibs).
Community and other companies interested in KMP Integration/extension will be using the same Publication/Consumption API as KMP Platforms managed by Jetbrains. But they can still communicate with other ecosystems, publish extra variants, artifacts, metadata etc.
For example your project: js-resources should be implemented as KMP Ecosystem Plugin, then it can use KMP Publication API to include your variants, artifacts as a companions to main KMP Publication. But on consumptions side users should use js-resources plugin to consume these extra artifacts back. Or js-resources can publish its data to other repositories, but still vanilla KMP on consumer side will be blind, until user enables js-resources plugins that can consume its data.

The picture I showed is more like my vision to the problem, and I'm happy to discuss it with you guys.

[antohaby]

w.r.t to optimizations for local projects. I don't see a problem there. its a matter of a build system and plugins that implement KMP there. UKlibs are mainly about Publication and Distribution within KMP Ecosystem.

[CLOVIS-AI]

All platforms are in a single artifact

The whole level of separate fragments is hidden inside an UKlib and entirely opaque to any external client. As such, a non-Gradle build tool can just download the UKlib and pass it to the Kotlin toolchain without tinkering with fragments, matching attributes, discovering variants, etc.

One downside is that projects using a few numbers of projects will still need to download and extract all platforms. I expect that the ecosystem will follow this pattern:

• Libraries support almost (if not all) platforms
• Applications support a few platforms

For example, a Kobweb or Kilua codebase will likely only support JVM/JS/Wasm, but they will have to download all flavors of Native for all the libraries they depend on.

Today, close to half of Kotlin users are server-side developers, who only need the JVM. How much will these changes impact them? Even if the compression algorithm is quite good, that will still increase the overall download size. (though having everything in a single file could decrease download time as Gradle can't download in parallel).

[abdulowork]

Today, close to half of Kotlin users are server-side developers, who only need the JVM. How much will these changes impact them?

We will still emit a .jar artifact for compatibility with Gradle/Maven JVM resolvers, so UKlibs don't directly impact JVM server-side developers

[martinbonnin]

We will still emit a .jar artifact for compatibility with Gradle/Maven JVM resolvers

@abdulowork Given a lib that publishes both a .jar and a .uklib, which one is KGP going to use?

My understanding is that using this new scheme, KGP would consume .uklib preferably? In that case, Kotlin backend developers will now download the (probably bigger) .uklib file. I'm not saying this is bad but I think this is still a direct impact?

[abdulowork]

which one is KGP going to use?

We have 2 main Gradle plugins in Kotlin Gradle Plugin: kotlin("jvm") and kotlin("multiplatform"). The jvm plugin will use the .jar, while the multiplatform plugin with JVM compilation will use the UKlib. So if the server-side project uses the multiplatform, they will indeed be impacted.

[martinbonnin]

The jvm plugin will use the .jar, while the multiplatform plugin with JVM compilation will use the UKlib.

Oh ok, I thought they would both use .uklib.

I kind of liked the idea that Kotlin would get its own (preferred) binary format ultimately, with a fallback to .jar only for compatibility.

As I think was mentioned previously, this would allow every Kotlin consumers (even org.jetbrains.kotlin.jvm) to work with IR. Might be easier than having to store @kotlin.Metadata annotation into JVM bytecode. This is just one example but there may be others.

[CLOVIS-AI]

Consequence №1. It's not possible to "restrict" visibility of a KMP library to only some fragments:

It is fairly common that I declare a dependency only for a single platform. For example, kotlinx-coroutines-debug, logback, etc. This is usually because they are only supported for that platform, but if they started supporting more platforms, I would want to be aware of that before imposing a new dependency to users of that platform.

What will be the developer experience when adding a dependency on a library that only provides a subset of the platforms supported by the project? Today, adding such a library in :commonMain leads to the (…improvable…) Gradle variant-aware resolution error, which states which platforms are not supported.

If I understand this KEEP correctly, users who declare a dependency on a library that doesn't support all platforms will simply not see the symbols from the library in their commonMain, and will have to create a new intermediary source set for the supported platforms. New users are sometimes confused by some libraries not being available in some platforms, this seems to be making the UX even less explicit.

[abdulowork]

users who declare a dependency on a library that doesn't support all platforms will simply not see the symbols from the library in their commonMain

That is correct.

New users are sometimes confused by some libraries not being available in some platforms

As you mentioned, this issue already exists in KMP. For example, the usage of runBlocking from kotlinx.coroutines in commonMain in a project with js target shows up as an unresolved reference. Although this is not part of the current design, the plan is to consider IDE/compiler UX improvements before deploying the new dependency scheme.