gazelle/go_repository build_file_generation=on should ignore existing BUILD files

[justinsb]

I have a dependency on kubernetes apimachinery https://github.com/kubernetes/apimachinery and it has a bazel build which is invalid for that repo in isolation - it requires me to add the dependency '@io_kubernetes_build//defs:go.bzl' and then fails because of version skew:

ERROR: /home/justinsb/.cache/bazel/_bazel_justinsb/6b4e21bf5ec591ada25084ae88726b8d/external/io_kubernetes_build/defs/go.bzl:1:1: file '@io_bazel_rules_go//go:def.bzl' does not contain symbol 'GoSource'
ERROR: /home/justinsb/k8s/src/k8s.io/kops/cmd/kops/BUILD.bazel:3:1: error loading package '@io_k8s_apimachinery//pkg/util/sets': Extension file 'defs/go.bzl' has errors and referenced by '//cmd/kops:go_default_library'

So I try adding build_file_generation="on" to the import, but ... it still does the same thing - still requires the io_kubernetes_build dependency, and still gives the GoSource error after I do.

I believe this means it is still consider the existing BUILD files, but I would have expected build_file_generation=on to ignore them.

The good news is that I did get this working by vendoring the files and stripping the BUILD files, but it would be nice to be able to use go_repository so I don't have to "manually" vendor.

[jayconrod]

This is kind of an unfortunate situation. Bazel doesn't do well when you have a dependency with preexisting but incompatible build files. When build file generation is turned on in go_repository, Gazelle will update existing build files; it won't replace them or remove any rule or file it wouldn't normally touch. I'd rather not add this functionality to Gazelle, since we plan on removing go_repository (#460), and I think that's the only context where it makes sense to do this.

Vendoring is a good option for this case, since it requires local modifications to the repository.

Another option is to set build_file_name = "BUILD.bazel". This will cause Gazelle to ignore BUILD files and generate / update BUILD.bazel files instead. Bazel will prefer BUILD.bazel files if both are present. This solution is a bit of a hack, since you end up with two sets of build files in that repository, and it wouldn't work if the files had originally been named BUILD.bazel.

The root cause of this issue seems to be dependency on the GoSource provider, which existed privately for a while during some recent refactoring. We're planning to tag a release this week that includes a public, documented GoLibrary provider (that's the new name). So once everyone is updated to the new tag, this problem should go away. We'll work with Kubernetes to make that happen.

[jayconrod]

I tagged a new release on Monday. Kubernetes is working on updating to a newer Bazel and a newer rules_go. I'll have another update next week.

[astratto]

👋 I'm hitting this error, but I haven't found a clear way to solve it.

I'm using Bazel 0.10.0 and current master for rules_go.
I'm also generating the Gopkg.bzl file automatically with:

# Dependencies are managed via dep's Gopkg.lock and converted via rules_go_dep
load("@com_github_scele_rules_go_dep//dep:dep.bzl", "dep_import")

dep_import(
    name = "godeps",
    gopkg_lock = "//:Gopkg.lock",
)
load("@godeps//:Gopkg.bzl", "go_deps")
go_deps()

Ideas?

[jayconrod]

Sorry for not coming back to this sooner. We've had a lot going on.

@justinsb Did you find an adequate workaround?

I can fix this on the Gazelle / go_repository side by adding a mode that overwrites build files instead of merging rules.

I'd like to understand the problem in apimachinery though. Is there a simple way to reproduce this? Ideally the problem would be fixed there by adding a set of correctly versioned dependencies instead of blowing away build files.

[jmhodges]

I can repro this with

go_repository(
    name = "io_k8s_api",
    importpath = "k8s.io/api",
    tag = "kubernetes-1.9.3",
)

go_repository(
    name = "io_k8s_apimachinery",
    tag = "kubernetes-1.9.3",
    importpath = "k8s.io/apimachinery",
)

go_repository(
    name = "io_k8s_client_go",
    tag = "v6.0.0",
    importpath = "k8s.io/client-go",
)

and then depending on "@io_k8s_client_go//kubernetes/typed/core/v1:go_default_library" in a target.

You'll get an error like:

$  bazel test   //path/to/whatever:go_default_test 
ERROR: /private/var/tmp/_bazel_jmhodges/ee957dabdee1b1a9e1f2fe90369cf80f/external/io_k8s_api/core/v1/BUILD:80:1: no such package '@io_k8s_api//k8s.io/apimachinery/pkg/runtime/schema': BUILD file not found on package path and referenced by '@io_k8s_api//core/v1:v1_go_proto'
ERROR: Analysis of target '//path/to/whatever:go_default_test' failed; build aborted: no such package '@io_k8s_api//k8s.io/apimachinery/pkg/runtime/schema': BUILD file not found on package path
INFO: Elapsed time: 112.948s
FAILED: Build did NOT complete successfully (111 packages loaded)
    currently loading: @io_k8s_apimachinery//pkg/runtime/serializer/versioning ... (7 packages)
ERROR: Couldn't start the build. Unable to run tests

[jmhodges]

Ah, maybe this is an unrelated bug. I will take it to a new issue.

[jayconrod]

Closing this old issue. I think this was ultimately resolved in kubernetes/publishing-bot#49.

[be9]

@jayconrod I've stumbled upon this problem in 2021 when trying to import one of k8s kops packages that pulls "k8s.io/kops/upup/models". The latter has a genrule defined in the build file:

genrule(
    name = "bindata",
    srcs = glob(
        [
            "cloudup/**",
            "nodeup/**",
        ],
    ),
    outs = ["bindata.go"],
    cmd = """
$(location //vendor/github.com/go-bindata/go-bindata/v3/go-bindata:go-bindata) \
  -o "$(OUTS)" -pkg models \
  -nometadata \
  -nocompress \
  -prefix $$(pwd) \
  -prefix upup/models $(SRCS)
""",
    tools = [
        "//vendor/github.com/go-bindata/go-bindata/v3/go-bindata",
    ],
)

While it's not needed for building the package, it makes analysis fail with:

ERROR: /private/var/tmp/_bazel_odashevskii/62903a9b6d07bad98db253c528844efb/external/io_k8s_kops/upup/models/BUILD.bazel:14:8: no such package '@io_k8s_kops//vendor/github.com/go-bindata/go-bindata/go-bindata': BUILD file not found in directory 'vendor/github.com/go-bindata/go-bindata/go-bindata' of external repository @io_k8s_kops. Add a BUILD file to a directory to mark it as a package. and referenced by '@io_k8s_kops//upup/models:bindata'

The full repro is here: https://github.com/be9/kopsexample/

[jayconrod]

@be9 It looks like you're downloading that project with go_repository in module mode. Vendor directories aren't included in modules, so if their build files refer to targets in the vendor directory, it won't work. You can download in VCS mode by setting the commit and remote attributes or in HTTP mode by setting urls and sha256. You'll need to remove version and sum.