Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

x/pkgsite: generate static docs #2381

Open
moraes opened this issue Oct 18, 2011 · 52 comments
Open

x/pkgsite: generate static docs #2381

moraes opened this issue Oct 18, 2011 · 52 comments
Labels
FeatureRequest NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. pkgsite
Milestone

Comments

@moraes
Copy link
Contributor

moraes commented Oct 18, 2011

It is fairly trivial to compile a given Go version and run the doc server locally to
search for an older API.

But it would be nice if golang.org provided access to docs per release. One simple
solution to this is to add an option to godoc to generate static docs on disk -- just a
tree of static files (search would be disabled of course). Then docs for old versions
could be made available somewhere in golang.org.

One additional benefit is that projects that want to ship docs could generate the static
files and provide the html files for download, or create pdfs, chms etc based on them.

PS: I tried searching for a similar issue; sorry if this is a duplicate.
@bradfitz
Copy link
Contributor

Comment 1:

This is being worked on, kinda.
Now that godoc can run on App Engine, we'll be doing something like running different
subdomains of golang.org for old releases.
Andrew knows the details.

Owner changed to @adg.

Status changed to Accepted.

@adg
Copy link
Contributor

adg commented Oct 26, 2011

Comment 2:

To generate static docs, just run this command:
godoc -http=localhost:8080 &
wget -r -np http://localhost:8080/pkg/
We're working on providing online docs for older versions of Go.

@moraes
Copy link
Contributor Author

moraes commented Oct 31, 2011

Comment 3:

I haven't thought about using wget. Nice. For those interested in this trick, here are
some notes.
First, edit robots.txt in the go root directory, and remove "Disallow: /". Otherwise
only the index will be downloaded because wget respects robots.txt.
Then start godoc pointing to the project path:
godoc -path="/path/to/project" -http=:8080
And finally this a more complete command to get a working static docs, including static
files and with proper links:
wget -r -np -N -E -p -k http://localhost:8080/pkg/
-r  : download recursive
-np : don't ascend to the parent directory
-N  : don't retrieve files unless newer than local
-E  : add extension .html to html files (if they don't have)
-p  : download all necessary files for each page (css, js, images)
-k  : convert links to relative

@rsc
Copy link
Contributor

rsc commented Dec 9, 2011

Comment 4:

Labels changed: added priority-later.

@adg
Copy link
Contributor

adg commented Dec 19, 2011

Comment 5:

Owner changed to ---.

Status changed to HelpWanted.

@adg
Copy link
Contributor

adg commented Mar 18, 2013

Comment 7:

Labels changed: added godoc.

@gopherbot
Copy link
Contributor

Comment 8 by pedromorgan:

Yes please

@josharian
Copy link
Contributor

Comment 9:

You can ask wget to ignore robots.txt by adding the flag `-e robots=off`.

@rsc
Copy link
Contributor

rsc commented Nov 27, 2013

Comment 10:

Labels changed: added go1.3maybe.

@rsc
Copy link
Contributor

rsc commented Dec 4, 2013

Comment 11:

Labels changed: added release-none, removed go1.3maybe.

@rsc
Copy link
Contributor

rsc commented Dec 4, 2013

Comment 12:

Labels changed: added repo-tools.

@rsc rsc added this to the Unplanned milestone Apr 10, 2015
@rsc rsc changed the title cmd/godoc: generate static docs x/tools/cmd/godoc: generate static docs Apr 14, 2015
@rsc rsc removed the repo-tools label Apr 14, 2015
@kegsay
Copy link

kegsay commented Nov 2, 2016

I would also like godoc to be able to generate static docs. The proposed workaround (hosting all the files and wgeting it) works but it crawls the entire contents of your GOPATH, which is less than ideal because that then bloats the static docs incredibly.

I still want things like the builtins to link correctly to docs, so some incantation of go list to pull out the dependencies of the project would be ideal. Unfortunately, I can't just write a bash script to do this because using godoc -html path/to/package just returns the raw HTML without any CSS, and no JS to handle section expansion.

EDIT: Managed to do this in the end with a fairly noddy script.

@gopherbot
Copy link
Contributor

Change https://golang.org/cl/72890 mentions this issue: godoc/internal/render: add render package for text formatting

@dsnet
Copy link
Member

dsnet commented Oct 24, 2017

As part of my work on #18342, I wrote simple static generator for godoc, that I realized is really well suited for resolving this issue. Is there any interest in including it's generated output in each release?

You can see statically generated docs at: https://static-hotlinks.digitalstatic.net/

@dsnet dsnet self-assigned this Oct 31, 2017
@dsnet dsnet removed the help wanted label Oct 31, 2017
@ghost
Copy link

ghost commented Nov 6, 2017

As part of my work on #18342, I wrote simple static generator for godoc, that I realized is really well suited for resolving this issue. Is there any interest in including it's generated output in each release?

@dsnet Is this static generator available somewhere? I'd like to give it a try for hosting docs for a few packages.

@rsc rsc unassigned dmitshur and dsnet Jun 22, 2022
@HRogge
Copy link

HRogge commented Nov 20, 2023

This is really annoying, especially because the anchors of links to source files are too the wrong line number... which is normally not recognized because godoc uses javascript to jump to the line too.

@3052
Copy link

3052 commented Jan 20, 2024

you can generate with Doc2Go:

https://github.com/abhinav/doc2go

it also supports generating different versions of a module

@adonovan
Copy link
Member

Is this feature (browsing old versions) essentially redundant with the "Added in go1.x" caption that pkg.go.dev displays next to each symbol? See:

pkg.go.dev should also display such a caption for the package as a whole, but does not do so yet. See:

@adonovan
Copy link
Member

the title is "generate static docs", so I dont see what your comment has to do with anything. currently the go tool cannot generate static docs, or even serve them for that matter, to my knowledge

I realize that's the title, but if you read the first note of the issue it is merely proposed as "one simple solution"; the actual problem described is

it would be nice if golang.org provided access to docs per release.

@adonovan
Copy link
Member

could you explain why you seem to be steering the conversation away from static docs and towards a server setup, across multiple issues? I get it, it seems you prefer that approach, but thats not the only approach.

x/tools/cmd/godoc is deprecated and unmaintained, and doesn't work with modules, and for those reasons I would like to delete it and close all open issues associated with it. However, there are a number of remaining obstacles to doing so, such as features or modes of operation that godoc supports but its proposed replacements do not. In addition, there are requests such as this one for new features to be added to godoc, which is of course not going to happen.

My goal is to make sure that the needs or problems motivating these issues are properly addressed by the newer tools. As usual, addressing the need or problem does not always mean implementing the feature exactly as proposed. In this case, the needs appear to be ways for readers to (a) understand which version a symbol appeared in the Go std API, which could be addressed by "Added in go1.x" captions as in pkg.go.dev; and (b) view specific versions of an arbitrary package's docs, which is already addressed by pkg.go.dev (for public packages) or a local pkgsite instance (for private ones).

Is there some other need addressed by static docs here that I have missed?

@adonovan
Copy link
Member

people want to be able to generate HTML files for a given module or package.

But why? That's my question, and none of the lines you quote contains an answer.

"Generate static docs" is a solution, not a problem. So what is the problem? I identified two problems, both of which (I claim) are solved by other means. What others remain?

is your plan to just get the serve option working, then close the static requests as wont fix? if so it would be nice if you are at least up front about it, so people can pursue other options.

As I said in my previous note, cmd/godoc will not be getting new features, so this issue as narrowly defined by its first comment is already "won't fix". Far from "avoiding the issue", I'm trying to understand what problems remain so that we can solve them by other means. No-one is stopping anyone else from pursuing other options.

I would appreciate it if you take your rhetoric down a notch in future.

@edaniels
Copy link
Contributor

Is there any plan then to have the go doc incantation become pkg site? For modern languages it seems imperative to have a documentation tool/command. It would be disappointing to see this go.

I had no clue this was a deprecated tool (I guess the source code says so?)

also @alandonovan, if no more work will be done on this, why not just close the issue and point people to pkgsite and its corresponding github issues since you mentioned it's missing some features. The back and forth seems silly for something that the Go team has committed to abandoning

@maxatome
Copy link

people want to be able to generate HTML files for a given module or package.

But why? That's my question, and none of the lines you quote contains an answer.

Just serving godoc in an already running web server accepting only static files? No need to run a host or a specific program just to dynamically serve files that almost never change, for example.

The first message mentioned the creation of PDF file. Why not? Why the user should launch a web server + a web crawler to do this conversion?

I am sure we can find plenty of other examples of static files uses...

++

@adonovan
Copy link
Member

adonovan commented Jun 21, 2024

[@maxatome] Just serving godoc in an already running web server accepting only static files? No need to run a host or a specific program just to dynamically serve files that almost never change, for example. The first message mentioned the creation of PDF file. Why not? Why the user should launch a web server + a web crawler to do this conversion?

These all seem rather hypothetical: for example, PDFs from godoc or pkgsite are not very pleasant to read, and you would still need to run a browser to render the HTML. But at least you are addressing my question.

OK, I will leave this issue open, and we can interpret "godoc" as "pkgsite".

[@edaniels ] Is there any plan then to have the go doc incantation become pkg site?

Yes, there is a plan to make go doc -http fmt.Println start an instance of pkgsite (or rather, a version of it shorn of heavyweight dependencies), and open a browser to the documentation for the chosen package or symbol. Oddly there isn't an issue for the feature itself, but the preparatory work is #61399.

[Update: I filed this feature request:]

@edaniels
Copy link
Contributor

Sounds great!

@3052
Copy link

3052 commented Jun 21, 2024

people want to be able to generate HTML files for a given module or package.

But why? That's my question, and none of the lines you quote contains an answer.

judging by your comment here and others, your position seems to be "I don't think static generation is useful" is that really your position? do you really not understand why people would want to do this? personally I think the responsive design of https://pkg.go.dev is awful, and want to get away from it. I much prefer the older style, which https://godocs.io seems to be similar to. so my goal is to have a tool that can do

SOME-TOOL SOME-MODULE

and I get a folder of HTML files, that I can serve on GitHub or where I want. this is an extremely common scenario/request, so why you are having a hard time grasping it, or why it could be useful is beyond me. with the current approach, user would need to literally rent a VPS just to host their own docs, since its forcing a server approach instead of static generation.

@torbenschinke
Copy link
Contributor

But why? That's my question, and none of the lines you quote contains an answer.

Our use case is to hand out a (static) documentation for our customers which use one or more of our private go modules in their products.

@seankhliao
Copy link
Member

OK, I will leave this issue open, and we can interpret "godoc" as "pkgsite".

Let's just retitle this.

@seankhliao seankhliao changed the title x/tools/cmd/godoc: generate static docs x/pkgsite: generate static docs Jun 21, 2024
@adonovan
Copy link
Member

adonovan commented Jun 21, 2024

judging by your comment here and others, your position seems to be "I don't think static generation is useful" is that really your position? do you really not understand why people would want to do this?

No, I don't have a position on the issue yet. I can imagine lots of ways it might be useful, but that's just my imagination (or @maxatome's, three comments back). I wanted to hear about actual concrete problems before deciding on a solution.

personally I think the responsive design of https://pkg.go.dev is awful, and want to get away from it. I much prefer the older style, which https://godocs.io seems to be similar to.

Well, that's really a separate question of what HTML is generated for a page. If pkgsite were to support static generation, perhaps it should use a print-friendly stylesheet.

@micahhausler
Copy link

I can imagine lots of ways it might be useful, but that's just my imagination. I wanted to hear about actual concrete problems before deciding on a solution.

At Amazon, I can safely say we have thousands of developers writing and reading Go. Our source code is stored in an internal Git hosting, and our home-grown build system has a build tool that leverages an internal Go module proxy. One of the language-agnostic outputs of our build system for a particular package (along with sources or binaries) is a documentation folder. Our build viewer can serve HTML files for a built release, which is a common pattern from other languages like Java (javadoc), Python (pydoc), Ruby (rdoc or 3rd party yard), Rust (rustdoc), C/C++ (3rd party doxygen). For Java, Python, Ruby, and Rust those are all first-party tools that our build tooling automatically calls to generates docs.

Today, in order to view Go documentation for a package, I have to check out source code on a machine (assuming I have source viewing/checkout permissions, which are separate from release consumption/viewing. For Go thats roughly equivalent, but not for all languages/builds), build it, spawn a process, possibly set up a port-forward (say, if I'm building on a remote Linux box), and then I can finally view the documentation. I can't tell you how many times I've had to repeat this process when I just want to browse another team's Go package.

The ability to invoke a maintained tool with roughly the arguments (to @3052's point)

TOOL -output-directory ./docs PACKAGE... 

would be in line with what other programming languages already support and would dramatically simplify the developer experience at Amazon when viewing docs for internal Go packages.

Because there is not one single filesystem that contains all versions of all packages' source code, and packages have differing permissions, running a fleet of pkgsite processes for all internal code, or spawning a per-user pkgsite process is not going to work for us.

If pkgsite were to support static generation, perhaps it should use a print-friendly stylesheet.

Our use case does not need a print-friendly version, as the vast majority of use cases would be online.

While I've described the Amazon internal use case here, but I think the general contours of this use case probably applicable elsewhere. I can also speak to previous places I've worked where I fronted statically generated Python and JavaScript HTML docs with an authenticated webserver, and Go was the odd one out.

@jimmyfrasche
Copy link
Member

(A print friendly stylesheet should be a separate issue. That would be equally as useful for someone wanting to print from/save a pdf of a page on go.dev and nothing about being dynamic or static factors into adding print styles.)

I agree that static documentation is important.

I worry about creating issues where things have to be done twice for the static vs. dynamic presentation or skew when someone adds a feature but no one checks to make sure how it works in the static version or scope creep because someone has a use case where they need to have all packages in a module go on one page or have each type get its own page and so on.

Maybe it would suffice to have a -json flag that output all the raw data. (And when I say raw I mean even the doc comments should be the go/doc/comment AST instead of html fragments).

Then there just needs to be a frontend process that takes that and applies the data to templates, but, importantly, there could be many different such frontends depending on taste and need.

Essentially, instead of

$ pkgsite -static -output dir pattern

you'd have

$ pkgsite -json pattern | static-pkgsite dir

but you could have many such static-pkgsite commands.

(pkgsite could also serve this same json dynamically for headless operation, but that would also be another issue).

JustinKuli added a commit to JustinKuli/governance-policy-nucleus that referenced this issue Jun 30, 2024
Previously, the coverage reports were pushed to the wiki, but since the
wiki is not meant for serving arbitrary HTML, viewing them properly
required going through `raw.githack.com`.

Also, the main `pkg.go.dev` site does not handle this repository well,
possibly because it is a fork (also complicated by the archiving of the
original) or because downloads of the package are very rare. Based on a
suggestion in github (dot) com/golang/go/issues/2381, an updated GH
action now downloads the contents of a local documentation server with
`wget`, and modifies some local links with `sed` to get something which
can be statically served. This mostly works, the main downside is that
JS files seem to be missing.

This adds on to the existing test step in the action, because that step
is necessary for the coverage information. If additional resources are
to be served on the GH Pages site, it might be a good idea to move to
a separate workflow, even though it would require re-running the tests.

Signed-off-by: Justin Kulikauskas <[email protected]>
@cespare
Copy link
Contributor

cespare commented Sep 22, 2024

My goal is to make sure that the needs or problems motivating these issues are properly addressed by the newer tools. As usual, addressing the need or problem does not always mean implementing the feature exactly as proposed. In this case, the needs appear to be ways for readers to (a) understand which version a symbol appeared in the Go std API, which could be addressed by "Added in go1.x" captions as in pkg.go.dev; and (b) view specific versions of an arbitrary package's docs, which is already addressed by pkg.go.dev (for public packages) or a local pkgsite instance (for private ones).

Is there some other need addressed by static docs here that I have missed?

@adonovan I also want to pitch in some with needs that you have missed. (Though @micahhausler did a good job already.)

At my job, I want to serve docs of different types -- Go docs, other language docs, web pages generated from protobuf definitions and markdown files, and more -- and the paradigm that makes it easiest to get all these working in a uniform way is if some job can generate some static files from the code in our monorepo at a particular commit and stick them somewhere to be served.

Also, for our monorepo the normal notions of "module versions" are meaningless. What we care about is what particular git commit (or maybe a branch or a tag) the docs you are looking at were generated from.

A tool that does a straightforward code -> HTML transformation is a primitive upon which it is easy to build this kind of stuff.

@chocolacula
Copy link

So, guys, you inspired me to develop gdmd. You can just generate README.md with same documentation and keep it in your repo. GitHub/GitLab render it by default during directories(packages) navigation. Simple but very useful tool.

@3052
Copy link

3052 commented Dec 6, 2024

another good option

https://codeberg.org/pfad.fr/vanitydoc

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
FeatureRequest NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. pkgsite
Projects
None yet
Development

No branches or pull requests