x/pkgsite, x/tools/cmd/godoc: some code examples are not displayed (but work at godoc.org)

[vmihailenco]

For example

• https://pkg.go.dev/github.com/go-pg/pg/v10?tab=doc#example-DB-Model-ForEach is missing
• https://godoc.org/github.com/go-pg/pg?tab=doc#example-DB-Model-ForEach works fine

[dmitshur]

Thanks for reporting.

I tested this package at commit go-pg/pg@6fe164e with pkg.go.dev, godoc.org, and x/tools/cmd/godoc, and can confirm the issue is present in pkg.go.dev and x/tools/cmd/godoc, but not godoc.org.

It's also helpful to look at the complete listing of examples at https://godoc.org/github.com/go-pg/pg?tab=doc#pkg-examples, https://pkg.go.dev/github.com/go-pg/pg/v10?tab=doc#pkg-examples, and the output of x/tools/cmd/godoc:

This looks like a valid issue. It needs investigation to understand why these examples, which appear to be following the example naming convention correctly, are not showing up in 2 out of 3 documentation viewers. It might be a problem in the example matching logic in go/doc package, or at a higher level. Once we understand why this is happening, it'll be easier to figure out how to fix this.

/cc @julieqiu @shaqque

[dmitshur]

One lead to investigate is that the exported DB type is defined as:

type DB struct {
	*baseDB
	ctx context.Context
}

Notably, baseDB is an embedded field. The DB type gets many of its methods indirectly via that embedded baseDB type:

// Begin starts a transaction. Most callers should use RunInTransaction instead.
func (db *baseDB) Begin() (*Tx, error)

The example classification logic in go/doc.NewFromFiles needs to know the complete API to determine if an example should be attached to a given identifier, so if it's not able to handle methods via embedded types, that could be the cause for this.

I haven't confirmed further, just wanted to share this.

[shaqque]

@dmitshur you're right that the issue is related to the embedded field. However, this appears to be expected behavior and is documented in https://github.com/golang/go/blob/master/src/go/doc/example.go#L489-L491

if !token.IsExported(m.Name) || m.Level != 0 { // avoid forwarded methods from embedding
	continue
}

Removing

|| m.Level != 0

appears to fix this issue.

The question now is whether we should consider embedded field methods for examples. One problem I can think of is that the embedded field type could also be exported, resulting in duplicate examples. However, we can check for this case easily (i.e. whether the embedded field is exported) and choose not to show the example for either the original struct type or the embedded field type.

/cc @dsnet @griesemer

[vmihailenco]

Gentle ping.

It seems to be better having duplicated examples rather than not having them at all. Also if duplicates are problematic - authors can rework examples and place them on different struct.

[jba]

choose not to show the example for either the original struct type or the embedded field type.

The type for which to show the example is embedded in its name: ExampleT_M should be shown on type T.

In determining whether ExampleT_M applies to T, we should consider all methods of T and *T, including those arising from embedding.

[jba]

The fundamental problem is that we don't have full type information when building the doc, so we can't really compute the full method set in all cases.

However, we do know about types in the same package. So I can't think of a reason we shouldn't use that information. So we should remove the m.Level != 0 check in pkgsite's godoc.

It is out of scope for this issue to make the change in go/doc itself.

[dmitshur]

So we should remove the m.Level != 0 check in pkgsite's godoc.

It is out of scope for this issue to make the change in go/doc itself.

@jba The original plan for pkgsite's copied go/doc package was to delete it as soon as possible, when pkgsite could start using Go 1.14 in deployment. If you're considering changing that plan, please take into account and update the package comment here.

[jba]

@dmitshur I see, I hadn't realized that. In that case, I think we should make the change in go/doc as well.

[jba]

I sent https://golang.org/cl/249557 to change go/doc.

[gopherbot]

Change https://golang.org/cl/249638 mentions this issue: internal/fetch/internal/doc: support examples on methods from embedded unexported types

[gopherbot]

Change https://golang.org/cl/249557 mentions this issue: go/doc: support examples on methods from embedded unexported types

[dmitshur]

I believe this is fixed via CL 249557 for go/doc and CL 249638 for pkg.go.dev. The examples mentioned in the original issue report now show up.

@jba If I missed something and there's more to do here, feel free to reopen.