Allow custom sources for Javadoc and Scaladoc

[Iltotore]

This pull request allows the user to define custom sources for Javadoc/Scaladoc though processedDocSources:

• JavaModule#docJar now uses processedDocSources instead of allSources
• ScalaModule#docJar now uses processedDocSources instead of os.walk(compile().classes.path).map(PathRef(_))

Fixes #1384 .

[lefou]

I'm not that happy with the name processedDocSources, as I immediately asked myself what "processed" could mean. Why not just docSources? Some scaladoc should indicate the purpose of the new target.

Also, one motivation to customize the docSources could be to adapt the set of sources files, so it might be probably worth a though, if we could feed files instead of dirs.

It looks like some unrelated formatting changes went into this PR.

[Iltotore]

It looks like some unrelated formatting changes went into this PR.

Yes, I just noticed my IDE reformatted the code. If needed, I can undo this reformat.

I'm not that happy with the name processedDocSources, as I immediately asked myself what "processed" could mean. Why not just docSources?

I also thook about docSources but this method already exists and it isn't the same feature. According to scaladocs, docSources is for static (not processed by scaladoc) files.

What if we have, for some obscure reason, a tasty file and we just want to have it in our doc without processing it ?

[lefou]

It looks like some unrelated formatting changes went into this PR.

Yes, I just noticed my IDE reformatted the code. If needed, I can undo this reformat.

That would be nice. If you enable scalafmt in your IDE, it should automatically use the .scalafmt.conf from the repo.

I'm not that happy with the name processedDocSources, as I immediately asked myself what "processed" could mean. Why not just docSources?

I also thook about docSources but this method already exists and it isn't the same feature. According to scaladocs, docSources is for static (not processed by scaladoc) files.

What if we have, for some obscure reason, a tasty file and we just want to have it in our doc without processing it ?

I must have overlooked the existing but quite new docSources. Unfortunately the existing docSources has also misleading name, as it does not contain sources (to be process by the doc tool) but static resources to be bundled as-is. I guess we should rename to docResources and deprecate the old name for the long-term benefit. The whole docSources seems to be unsupported outside of Scala 3, so we might as well clear it up without deprecation cycles, which also allows us to re-use it's name now.

Besides, the "processed" prefix is misleading, as these sources are going to be processed by the doc tool.

[Iltotore]

That would be nice. If you enable scalafmt in your IDE, it should automatically use the .scalafmt.conf from the repo.

That's what I did. I will push the code formatted using scalafmt

I guess we should rename to docResources

Do you want me to commit this change then, use docSources instead of processed ?

[lefou]

I guess we should rename to docResources

Do you want me to commit this change then, use docSources instead of processed ?

Yes, please!

To avoid conflicts with users who used that target to provide static resources, we should probable output a warning in case the outcome of docSources isn't empty but does not contain any scala or java source files. Something like: "Since mill after 0.9.8 docSources should only be used to provide source files. To provide static resources, please use docResources."

[Iltotore]

Earlier we talked about reformatting modified files using ScalaFMT but it indicated that they're already formatted correctly.

[Iltotore]

Once this change is approved, I will add the suggested warning.

[lefou]

Earlier we talked about reformatting modified files using ScalaFMT but it indicated that they're already formatted correctly.

Scalafmt isn't going to remove new lines which you introduced earlier.

[Iltotore]

This last change works for Scala 3 and Scala 3 Milestone. Haven't tested with Scala 2 yet but it should as I simply applied the same changes (except for tasty files filter) and it also uses packageWithZinc. I'll test it when available.

EDIT: Seems to work in Scala 2.x.

[lefou]

There are still failing tests related to scaladoc.

[Iltotore]

Actually, I don't really understand where this error comes from:

-- Error: C:\Users\rapha\IdeaProjects\mill\target\workspace\mill\scalalib\DottyDocTests\eval\multiple\multidocs\compile\dest\classes\pkg\SomeClass.tasty:2:407 
2 |??u?6??p?V3☻?@???o?t☻?3???p?V2☻?‼↨ ?☺??☺???p?V3???u?3?3?q?p?3?u?Pu?Pu?3?v?\r☻?3????)3????)u?3???p?Vu?3?R3♥?♠↨???)3???p?V3?↨?????3?↨\♦♦↨????3?↨?????u?3?t♥?P3☻??p?V3?↨‼↨??3☻???p?V3☻?@??☺?????
??☺????~???????????????????????????????????????????????????????☺?????☺?????☻?}???????????????????????????????????????/** A doc comment. */▬?☻?/** A doc comment. */▬?☻?/** A doc comment. */▬?
  |

                          ^
  |                                                illegal character '\u0000'

Seems like an encoding error but if I'm not wrong, I didn't change something about file encoding.

Failing tests are related to pre-release Dotty. I'll check the code tomorrow.

EDIT: After some researches, the compiled classes already have illegal characters once compiled. I'm not sure this issue comes from this PR the master branch doesn't have this issue so it comes from my fork. I don't know if it is an old bug patched in master or if it is caused by my changes. I'll update my fork with the upstream.

[lefou]

This seem not correct for early dotty / scala 3 versions (isDotty(scalaVersion()) || isScala3Milestone(scalaVersion())) which use dottydoc and consumes scala source files.
Only Scala 3 releases (isScala3(scalaVersion())) use scaladoc 3 tool, which consumes tasty files.

[Iltotore]

The last commit should fix this issue. Tests passed in local.

[lefou]

Your PR looks good now. There are still some newly introduced but unwanted newlines. Can you please remove these, and we are good to go.

[lefou]

Looks good to me. Thank you!