Improve compatibility with the Gradle API, and follow best practices

[aSemy]

Is your feature request related to a problem? Please describe

Hi, this is related to the Dokka Gradle plugin, which I would like to help improve.

Improving usage of Gradle would help with the following issues:

• Support Gradle Configuration cache #2231
• Remove double negation in configuration option names #2515
• Gradle 7 incompatibility due to implicit dependency. #1833
• Merge multiproject (composite build) docs #1864
• dokka should provide gradle-idiomatic way to aggregate documentation #1752
• Error: Could not load the value of field values ... found in field dokkaSourceSets of task :dokkaHtml #2612
• Consider supporting Gradle's configuration cache #1217
• dokkaJavadocMultiModule? #1382
• Changes in custom CSS are ignored on incremental documentation rebuild #2286
• Make dokka Gradle task cacheable #830
• Remove the dependency on internal Gradle API #2822
• Bump Gradle to 7.6 #2756
• Use Gradle Worker API #2903

These PRs are related:

• Fix compatibility with Gradle's configuration cache #2499
• Fix configure version eagerly #2799
• Avoid snapshotting sourceLink.localDirectory input #2807
• Fix NoSuchMethodError with Gradle 8.0 RC1 #2816

Describe the solution you'd like

• use Kotlin DSL helpers to make the code more Kotlin-esque, for example
  

  dokka/runners/gradle-plugin/src/main/kotlin/org/jetbrains/dokka/gradle/gradleConfigurations.kt

  Line 10 in 8317436

  attributes.attribute(Usage.USAGE_ATTRIBUTE, project.objects.named(Usage::class.java, "java-runtime"))

• Tasks that require downloading files should cache and re-use files Slow documentation generation #2695
• Use configurations for sharing outputs between projects https://docs.gradle.org/7.3/samples/sample_jvm_multi_project_with_code_coverage.html
• Remove non-idiomatic helper functions https://github.com/Kotlin/dokka/blob/master/runners/gradle-plugin/src/main/kotlin/org/jetbrains/dokka/gradle/DokkaProperty.kt
• Rename files to match the contents, e.g. https://github.com/Kotlin/dokka/blob/master/runners/gradle-plugin/src/main/kotlin/org/jetbrains/dokka/gradle/main.kt
• Remove usages of internal Gradle API classes

  dokka/runners/gradle-plugin/src/main/kotlin/org/jetbrains/dokka/gradle/AbstractDokkaLeafTask.kt

  Line 16 in 8317436

  DslObject(this).extensions.add("dokkaSourceSets", container)

• avoid using allprojects {} or subprojects {}
  

  dokka/runners/gradle-plugin/src/test/kotlin/org/jetbrains/dokka/gradle/AbstractDokkaParentTaskTest.kt

  Lines 21 to 23 in 8317436

  	rootProject.subprojects { project ->
  	project.tasks.create<DokkaTask>("dokkaTask")
  	}

  dokka/runners/gradle-plugin/src/main/kotlin/org/jetbrains/dokka/gradle/AbstractDokkaParentTask.kt

  Lines 54 to 60 in 8317436

  	fun addSubprojectChildTasks(childTasksName: String) {
  	addChildTasks(project.subprojects, childTasksName)
  	}
  	fun removeSubprojectChildTasks(childTasksName: String) {
  	removeChildTasks(project.subprojects, childTasksName)
  	}

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered

Additional context

Perhaps the dokka team could create a 'gradle-plugin' tag for all relevant GitHub issues? https://github.com/Kotlin/dokka/issues?q=is%3Aissue+is%3Aopen++gradle

Are you willing to provide a PR?

Yes. I will try and split this up into smaller PRs.

[IgnatBeresnev]

Wow... thanks! Our Gradle plugin and everything around it definitely needs work. I'll take us some time to go through PRs, so please bear with us 😅

[aSemy]

So I've submitted a few PRs and they make some progress towards improving the plugin, but actually started re-implementing the plugin mostly to get my head around it, but also I think it would be nice to have a fresh start. I see there's code in the plugin from 2015, and Gradle has changed a lot since then.

Here's what I've got so far:

• Use the dokka-cli project to execute Dokka
• use javaexec {} to run the cli, not reflection magic.
• (Aside: because of this the Gradle plugin could be made completely independent from the Dokka code base, with the dokka-cli version being customisable, perhaps with a runtime check for the latest version. This decoupling is used in other plugins, like JaCoCo, gradle-intellij-plugin.)
• Simplify the Dokka tasks to three: 2 for configuration, 1 for executing. These use file-based outputs to share configuration between projects, ensuring tasks can be cached properly:
  1. DokkaConfigurationTask - takes user/project input and creates a Dokka JSON config file, to pass to dokka-cli. It also consumes DokkaModuleDescriptions from subprojects (using Gradle configurations)
  2. DokkaModuleConfigurationTask - creates a DokkaModuleDescription subprojects, and encodes it as a JSON file, which can be shared properly with Gradle configurations.
  3. DokkaExecutorTask - consumes the configuration file produced by DokkaConfigurationTask, uses the file to correctly mark task inputs/outputs, and runs Dokka using dokka-cli + javaexec{}
• Create Dokka specific Gradle Configurations to share the JSON config files between subprojects (based on this Gradle example: https://docs.gradle.org/7.3/samples/sample_jvm_multi_project_with_code_coverage.html)

I'm not sure how much impact this will have. Perhaps it's possible to 'backport' it to the current plugin, or maybe it needs a clean break. It's still a work-in-progress. I can make a PR if you want to have a very early look?

[Goooler]

#2652 would be a part of this?

[Goooler]

And CC is WIP in #2499.

[aSemy]

#2652 would be a part of this?

#2652 is about improving the Gradle build scripts for the Dokka project, not the Dokka Gradle plugin. They both use Gradle, but they are independent from one another.

And CC is WIP in #2499.

Oh good catch, I missed that PR. That PR is indeed related, and should help.

[IgnatBeresnev]

Hi! Sorry it's taking so long, I've only now finally got time after re-writing documentation for stable release.

I'll go over all Gradle PRs in the upcoming weeks 👍

[IgnatBeresnev]

If you don't mind, I'll use this issue as an umbrella issue to cover all Gradle-related PRs from you and @Goooler. I'll include "infrastructure" tagged issues in my comment as well as they're about following Gradle's best practices and there aren't that many of them.

Below I'll write down the order in which the PRs should be reviewed and merged, feel free to correct me.

First wave

Can be reviewed now, but merged after we branch off for 1.8.10:
✔️ * #2833
✔️ * #2707
✔️ * #2705
✔️ * (infra) #2704

✔️ Blocked by #2841 (ETA: right after 1.8.10):
✔️ * (approved) #2702

✔️ Blocked by #2704 (ETA: right after 1.8.10)
✔️ * (infra) #2912

✔️ Blocked by #2912
✔️ * (infra) #2884

Second wave

• (needs a solution) Fix compatibility with Gradle's configuration cache #2499
• (we have some concerns to discuss) Dokka Gradle Plugin: Implement Gradle Worker Daemon for Dokka generation #2740

Third wave:

• Dokka Gradle Plugin 2 #2839

To be discussed further / postponed:

• (might not be needed) Gradle Plugin: Expose finalizeCoroutines as Gradle task option #2759

[aajtodd]

What's the status of this? It looks like a lot of great improvements that we would like to see/use (gradle worker API, better cache support, fix inter project task dependency relationships, etc).

The AWS SDK for Kotlin is an abnormally large project (300+ modules). It takes several hours to build everything and we are weighing our options on how to get this time down (including possibly investigating moving to https://github.com/adamko-dev/dokkatoo).

There are also some issues with upgrading to Gradle 8.2+ w/Dokka 1.8+ that we are wondering might be fixed by some of these changes.

e.g.

    Reason: Task ':services:dokkaHtmlMultiModule' uses this output of task ':dokkaHtmlMultiModule' without declaring an explicit or implicit dependency. This can lead
 to incorrect results being produced, depending on what order the tasks are executed.  

[IgnatBeresnev]

@aajtodd I definitely agree, and we honestly can't wait ourselves to address Gradle problems, but our resources have been spread thin, and there have been multiple things falling apart at the same time (K2/analysis-related), so we couldn't focus on Gradle much.

Rest assured, we understand that our Gradle plugin is a severe problem for Dokka's users, and it is very high on our list. We should hopefully get to it quite soon - likely within the next month.

In the meantime, if migrating to github.com/adamko-dev/dokkatoo resolves your problems - please, do use it :) Adam did an amazing job, and we plan to merge dokkatoo into Dokka, making it Dokka's new Gradle plugin - albeit, with some configuration DSL changes, so expect some migration work up ahead. It would also help us with finding bugs early, so you'd be helping us with testing.

P.S., Dokka 1.9.0 should work with aws-sdk-kotlin, we've somewhat "fixed" the single-target multiplatform support (with your help in debugging), and I'm still trying to get to providing more details on this in #2679 - will try hard to do it this friday or next week.