go/parser: line numbers of comments get reset after the package statement

[agnivade]

What version of Go are you using (go version)?

$ go version
go version devel +60d443f412 Fri Mar 8 11:56:16 2019 +0530 linux/amd64

What did you do?

Consider this code -

package main

import (
	"fmt"
	"go/parser"
	"go/token"
)

func main() {
	src := `
// Code generated by go tool dist; DO NOT EDIT.

//line /home/agniva/play/gosource/go/src/cmd/link/link_test.go:1
package main

import (
	/* comment */ io1 "io"
	"fmt" // for Printf
	/* commentlog */ "log"
	/* commentio2 */ io2 "io"
)
`

	fset := token.NewFileSet()
	f, err := parser.ParseFile(fset, "src.go", src, parser.ParseComments)
	if err != nil {
		panic(err)
	}

	for _, g := range f.Comments {
		fmt.Printf("%v: %d\n", g.Text(), fset.Position(g.Pos()).Line)
	}
}

What did you expect to see?

I expect the line numbers to be counted from the first line of the file. But they aren't. They get reset after the package statement.

What did you see instead?

Code generated by go tool dist; DO NOT EDIT.
: 2
line /home/agniva/play/gosource/go/src/cmd/link/link_test.go:1
: 4
 comment
: 4  // ----- >> This is line 4 again ! Which gets counted from the package statement.
for Printf
: 5 // --- And from then on everything is as if the code starts from the package statement
 commentlog
: 6
 commentio2
: 7

Found while working on https://go-review.googlesource.com/c/go/+/162337/.

Filing this at the request of @griesemer

[griesemer]

This is working as expected. The line numbers don't get reset by the package statement; they get set by the line comment preceding it. If you make that comment not a line comment it will not "reset" the line numbers.

If you want the unmodified, actual source file line numbers, irrespective of line comments, use token.FileSet.PositionFor rather than token.FileSet.Position to obtain the token.Position. This code works.

[agnivade]

The line numbers don't get reset by the package statement; they get set by the line comment preceding it.

If that is expected, then it certainly should be documented. It is very surprising that a block comment doesn't do anything, but a line comment sets the line numbers. And also, if I move the line comment below the package statement, everything works fine.

Is there any documentation regarding this peculiar behavior between a line comment and the package statement ?

EDIT: The documentation for PositionFor just says

If adjusted is set, the position may be adjusted by position-altering //line comments; otherwise those comments are ignored.

It does not say that the position-altering happens only if it is above the package statement. Don't we need this clarification or am I misunderstanding something ?

[griesemer]

This is documented here. It works for line and block comments, as long as they start with //line or /*line and follow the respective syntax.

Again, this has absolutely nothing to do with package clauses. This is just a mechanism of //line and /*line comments. At least //line comments are a well-known feature, originally inherited from the C world, where the equivalent are # line preprocessor directives.

If you move the respective line comment below the package clause, you're simply renumbering the lines after that comment. This example may work "fine", others may not.

[agnivade]

Sorry that's right. I somehow missed that it's a //line comment, and kept thinking it's a line comment.