Omit sources from generated Javadoc

[cpovirk]

When we build Javadoc, we set linksource. That causes the Javadoc tool to generate links from text like public interface Multimap in https://guava.dev/Multimap. Those links go to pages like https://guava.dev/releases/snapshot-jre/api/docs/src-html/com/google/common/collect/Multimap.html#line-164.

I somewhat suspect that no one cares about these pages: If they want source, they'll look in GitHub or in the source jar for a release or snapshot.

Additionally, these pages force the generated Javadoc to change even for implementation changes (example output changes from input changes).

And any additional page presumably slows down downloading history, switching branches, and deploying the guava.dev. site.

I'm not sure I'll even bother to remove linksource, but I'm thinking about it, so let me know if that would cause a problem for you. If I do try it, and if it goes well, I might even consider removing some of the existing sources....

[cpovirk]

(If we cared enough, I'm sure we could rewrite the links to point to GitHub.)

[Brijeshthummar02]

@cpovirk may be we can rewrite the links what are your thoughts on this also if we can fix it provide me path to the file.

[cpovirk]

The idea would be that the Javadoc for a release (e.g., https://guava.dev/releases/33.4.5-jre/api/docs/com/google/common/annotations/Beta.html) would link not to https://guava.dev/releases/33.4.5-jre/api/docs/src-html/com/google/common/annotations/Beta.html#line-36 but to https://github.com/google/guava/blob/v33.4.5/guava/src/com/google/common/annotations/Beta.java#L36 (and similarly for the Android link, which would go to https://github.com/google/guava/blob/v33.4.5/android/guava/src/com/google/common/annotations/Beta.java#L36).

Then the snapshot Javadoc (e.g., https://guava.dev/releases/snapshot-jre/api/docs/com/google/common/annotations/Beta.html) would link not to https://guava.dev/releases/snapshot-jre/api/docs/src-html/com/google/common/annotations/Beta.html#line-36 but to https://github.com/google/guava/blob/master/guava/src/com/google/common/annotations/Beta.java#L36 (and again, a similar variant for Android).

The dream would be that we can then wipe out all the src-html files that slow down everything. To really get the benefits, we might even want to rewrite the entire history of gh-pages so that the files don't exist anywhere in history.

(I just ran into trouble with this yesterday when I was trying to perform multiple Guava releases: I ended up with multiple copies of the Guava repo in my /tmp, and my machine is (for whatever reason) partitioned to put /tmp on a separate (small!) partition. That filled up /tmp, and most of the releases failed. I ended up rerunning with an environment variable set to place the temporary directories elsewhere.)

[Brijeshthummar02]

@cpovirk after checking and rebuilding the project now there is no src-html.

i need to know more about it like how it can be done

To really get the benefits, we might even want to rewrite the entire history of gh-pages so that the files don't exist anywhere in history.

just general question what does the label P3 mean on this issue.

[cpovirk]

As I mentioned last week, I filled up my /tmp partition as a result of a few Guava releases. Today, I discovered that I'd almost filled up my main partition, too. There are much bigger culprits than Guava (e.g., three separate variants of IntelliJ that I still have installed...), but Guava is very much Not Helping.

I ran some n=1 experiments to see how slow Guava currently makes various operations:

• git clone git@github.com:google/guava.git, SSD filesystem: 27s
• git clone git@github.com:google/guava.git, tmpfs filesystem: 24s
• git checkout gh-pages, SSD filesystem: 52s
• git checkout gh-pages, tmpfs filesystem: 19s

So I am going to see what I can do about that. Over the weekend, I am going to run:

git filter-branch --tree-filter ~/rewrite-gh-pages.sh gh-pages

(Yes, I am aware that filter-branch is heavily discouraged. But filter-repo is more complex (and I don't want to hack up lint-history), rebase --strategy-option=theirs --exec ... may lead to merge trouble [edit: though this workaround looks quite clever], and anything I work out with plumbing commands would amount to reinventing filter-branch. [edit: Maybe I should try git test fix from git-branchless?] I have backups, including both google/guava and cpovirk/guava on GitHub for now [edit: including an additional line of defense, a tag in my fork].)

As I write this, the contents of rewrite-gh-pages.sh are:

#!/bin/bash

set -u

if [[ ! -e releases ]]; then
  exit
fi

if [[ -e releases/snapshot-jre/api/docs/com/google/common/util/concurrent/ForwardingBlockingDeque.html && -e javadocshortcuts/forwardingblockingdeque/index.md ]]; then
  perl -pi -e 's"collect"util/concurrent"g' javadocshortcuts/forwardingblockingdeque/index.md javadocshortcuts/ForwardingBlockingDeque/index.md
fi

find releases -name src-html | xargs -r rm -r

( cd releases; for R in [0-9]*; do ( cd $R && shopt -s extglob && T=v${R%%?(-jre|-android)} && if [[ $R == *-android ]]; then P=android/; else P=""; fi && case $R in 1.0 | 2.0 | 3.0 | 4.0 | 5.0 | 6.0 | 7.0 | 8.0 | 9.0 ) G="" ;; * ) G=guava/ ;; esac && find -name '*.html' | xargs -r perl -pi -e "s!../../../../src-html/([^#]*)html#line[-.](\d+)!https://github.com/google/guava/blob/$T/${P}${G}src/\${1}java#L\$2!g" ); done )

( for R in releases/[0-9]*; do ( cd $R && for F in $(find api/docs -name '*.html'); do perl -lpi -e 's!^<title>!<link rel="canonical" href="https://guava.dev/releases/snapshot-jre/'$F'"><title>!' $F; done ); done )

if [[ ! -e releases/snapshot ]]; then
  exit
fi

find $(find releases/snapshot* -maxdepth 0 -type d) -name '*.html' | xargs -r perl -pi -e 's!<a href="../../../../src-html/[^>]*>(.*?)</a>!$1!g'

That will:

• delete src-html
• replace existing links to src-html with links to GitHub for releases, and delete them entirely from snapshots (since changes in line numbers in the links are a source of churn there even with the actual sources removed)
• standardize guava.dev/forwardingblockingdeque to point to consistently point to the util.concurrent copy after that copy was introduced
• insert a <link> element to attempt to make search engine prefer the head copy of Guava's Javadoc over random old versions. (e.g., a Google search for guava ascii currently has release 19.0 as its top result)
  • I should really have a plan to insert this into future releases' Javadoc.
  • Incidentally, if you're reading this and you haven't already heard, note that you can get to Guava's Javadoc by typing "guava.dev/api" and that you can get to the Javadoc for a type like ImmutableList by typing "guava.dev/immutablelist."

[cpovirk]

(I think things are going a little slower than Git projected, perhaps because the size of the files in the branch grows over the life of the repo as we publish more releases? But I had already seen on Friday that "over the weekend" was not going to be enough, so we're probably talking something like the difference between Monday afternoon and Wednesday morning. Anyway, I hope that the results turn out to look as expected, given how slow this is to run :))