Enable transitive target CPU constraint #1264
Comments
|
ios_arm64 is not a cpu... I have no idea if what you are trying to do will work though, I think you are after toolchain transitions, which we would love to have but are not exposed to skylark right now. |
|
I'm trying to embed a It does work fine when I specifically select the Note that Indeed toolchain transitions sounds quite a bit like what's needed. However, I'm surprised it properly works when using |
|
For future record, Bazel reports this: |
|
The cpu command line argument is not the same as the cpu constraint, the former is a hodgepodge of crazy values that only make sense to a CROSSTOOL (and the list varies depending on what crosstool is loaded), the latter has cpu and os correctly separated and is designed to work with toolchains and the constraint system (@bazel_tools//platforms:cpu and @bazel_tools//platforms:os) I suspect the custom magic in the config transition logic for ios_application probably sets the effective value of the command line argument cpu for you, and thus selects based on that work, but they may have not got around to making the constraint system understand that transition, which would also mean that toolchain selection for anything you depend on would be broken. You would need to talk to the bazel team about that. |
|
Very clear, thank you. That would be kind of a hack, but it might work. |
|
We used to, but it was seriously problematic... There was no sane way to map config setting to GOOS and GOARCH values, and many things allowed for go cross compile were not valid as --cpu values. I think there were some other issues, I don't remember fully any more. Instead we describe a complete system based on constraints, and trust bazel to map the --cpu value to constraints correctly for now. Hopefully even that part will go away soon when the cc toolchain selection is switched to also obey constraints and the platform stuff stops being experimental. What we need is for the thing that does the config transition for the ios rules to correctly set the constraints in the same way it would on start up when picking them based on --cpu, and make sure toolchain resolution is run in the dependent rules correctly. |
|
Is there any way to define a constraint based on something else than internal Bazel state and platform? |
|
But actually, my only problem is because I'm trying to build a multi-arch artefact (namely an iOS app) and i'm not sure how to do that unless getting it via the Apparently Bazel does propagate things such as Command line: |
|
Actually, I'm thinking i could hack it by |
|
That might work, but it also might cause you issues with caching, depending on how the config switch for ios was written. I am really interested to know what you are using go for on ios, I did not realize anyone was using that! |
|
I am indeed using cgo! Our app (Zenly) has a common "on-device app backend" that is built almost entirely using Go and GoMobile, for both Android and iOS. We've been doing this for a year and half now with great success, but are now kind of constrained by the fact that gomobile handles the build part (which was great for a long time) by itself, limiting us in the way we can mix and match languages and libraries (for instance, using a patched/modified Go runtime for golang/go#22716). That part is responsible for network interface (via gRPC), talking to the device APIs directly (CoreLocation for instance, hence my issue about objc support and https://github.com/znly/rules_go/commit/a9a23fb08c4b35be72c488e3300d18447f173f1f) and pushing this data to the UI via ReactiveX endpoints. We use protobuf to do trans-vm marshalling and ReactiveX for concurrency unification between Java/Swift and Go, which nice features such as a ReactiveX subscription becoming a Also, finally being able to have one build tool for apps on both platforms, backend, protobuf while also properly expressing the dependency graph cross language is really, really cool (for instance part of the Swift app depending on part of the Go backend etc...). Hence all the commits we are doing in our fork of rules_go :) EDIT: and I should also point out that we are re-implementing what |
|
Bouncing on your comment, I just tested the following, on Bazel load("@io_bazel_rules_go//go:def.bzl", "go_binary", "go_library")
config_setting(
name = "ios",
values = {"cpu": "ios_arm64"},
visibility = ["//visibility:public"],
)
go_library(
name = "go_default_library",
srcs = [
"main.go",
],
cgo = True,
importpath = "github.com/znly/cmd",
visibility = ["//visibility:private"],
)
go_binary(
name = "cmd",
embed = [":go_default_library"],
goos = "darwin",
goarch = select({
":ios": "arm64",
"//conditions:default": "amd64",
}),
visibility = ["//visibility:public"],
)And Bazel fails with: |
|
I have been doing a lot of work to get to the point where the link modes will work properly, I see you are piggybacking of most of it. goarch has to be a plain string, because of the way it propagates onto an aspect. |
|
What do you think I open a PR for link modes and have that conversation there? I still have to rebase on the stdlib builder (not sure if it's really that big of a deal, though). The more we upstream, the better. |
|
The multiple target option seems to be working ok! Perhaps I messed something up in https://github.com/znly/rules_go/commit/fce5ac2d1f8a18ae77712cd392c76b54cb03e2d8 or https://github.com/znly/rules_go/commit/344d1337b0350ea5e3b9e712437d38f33054d8a7 ? |
|
You might want to update to the latest stdlib builder, it makes sure the stdlib is only ever built by aspect propagation. |
|
Thanks Ian i'm making great progress. I just rebased Now the stdlib bootstraps fine and I can even build a sample With the following BUILD: go_library(
name = "go_default_library",
srcs = [
"main.go",
],
cgo = False,
importpath = "github.com/znly/cmd",
visibility = ["//visibility:private"],
)
go_binary(
name = "cmd",
embed = [":go_default_library"],
goos = "darwin",
goarch = "arm64",
visibility = ["//visibility:public"],
)However, if I set The second weird thing is that with Instead of pure. Not that I really care for my use case, but it seems weird don't you think? Here is the subcommand: It fails because it is set to use the iPhoneOS sdk. |
|
It seems |
|
Indeed, the I tried giving it manually to |
|
The goos and goarch of the go_binary are collected onto the aspect, and then the aspect propagates across all the go rules, all the way down to the standard library. The rules also add a default output outside the aspect for the host, so you can actually build a go_library directly (it would be really annoying if you could not build a library from the command line), but if the rule is not built directly and that output is not used, it won't be built. It does always have to build a copy of the standard library in host mode in order to build the tools it uses, maybe you are seeing that? |
|
Indeed I am seeing the host stdlib being built successfully (and bazel marks it as the host, too). When building as c-archive, cgo needs to be enabled to export functions from the main package. Also, I am calling directly ios apis via the objc support. Thanks for being so fast! |
|
I tried digging into the aspect part to add it to CGo but failed to see how it propagates in the |
|
Ok so I found the reason: 2 cgo targets do not get
However, |
|
What is it that actually goes wrong and needs the goos and goarch on those targets? |
|
Well apparently since they do reference the stdlib, it triggers a build for the stdlib for the wrong target since it falls back to auto for those two rules. And because |
|
Instead of trying to bend |
|
I am not sure why the cgo_codegen would need the stdlib at all, in theory it should be happy with the bootstrap mode that uses the non target specific form. |
|
Sure, here is a sample project: https://github.com/znly/rules_go_ios_example I'm using our fork, specifically this branch: https://github.com/znly/rules_go/tree/misc/newstdlib You need a Mac with Xcode and iOS SDKs installed, of course. Basically, if I add |
|
Also, I think if this is fixed cross-compilation + cgo would work just fine (assuming a proper crosstool for cc targets). |
|
Quick update: I've tried referencing only the bootstrap toolchain (via go_rule bootstrap param), for cgo_import rule for instance, but it does fail also, because the builders (at least cgo tool) is not available in bootstrap mode. I've been thinking about a way to properly propagate mode to those rules but fail to see how. Would you have any pointers as to how would be the course of action to take? Many thanks |
This removes the hard coded list of standard libraries, and builds it per aspect being propagated. This is both cleaner and a pre-requisite for the additional link modes to work. Related to bazel-contrib#1264 and bazel-contrib#54 and bazel-contrib#539
|
Thanks for #1295, however, it seems to break c-archive link mode (stdlib gets' built twice). |
|
Also, I've managed to fix the goos/goarch issue with a hack that's unlikely to be upstreamed: |
|
It will build the stdlib once per build mode now, so it could be built twice, but that should not break anything. When you say breaks, what do you mean? |
|
Oh sorry for the wording: I meant break when doing cgo-enabled cross compilation since the mode (goarch/goos) is now wrong compared to what the CROSSTOOL is doing. However, I do believe that the fact that it does build the stdlib twice (once with the wrong mode) as something to do with it. |
|
What is the breakage? which I believe is because the standard library is built in the wrong mode, and I cannot find any way to build it in the right mode. |
|
I kind of remember seeing that building the stdlib itself as c-shared does not work. Although it's not the same as plugin, could it be related? golang/go#13234 |
|
The breakage seems to be that somewhere goos/goarch is lost, so go_context falls back to auto (which would be darwin_amd64 in my case instead of darwin_arm64+ios) In the case of |
|
In my case I'm building the binary with |
|
Yes, exactly, you cannot build the stdlib with the special link modes, because they require you to be building the main package, and there does not seem to be any invocation I can find that would build the stdlib in the same way those modes would do, short of actually compiling a binary that just happened to use the entire standard library.... The mode is lost inside cgo rules, I spent a while looking at it during the week and I don't see how to fix that because it can't get through the cc_library and cc_binary rules that we have to add in. The question is what does it actually break, and is there anything we can do about the breakage (I have not reproduced your problem, I am not set up to build ios apps). |
|
Basically, what it does break is cross-compilation using cgo via goos/goarch. In my case, I have a target: Also, Indeed if I build without cgo, the target is properly bootstrapped and built, but with CGo it will somehow try to build the stdlib for It is also important to add and I've manually added |
|
Here is the subcommand which fails because |
|
Also, I don't get why the mode would be lost in my case, as I'm actively passing goos and goarch and when I print the mode inside the rule, it seems fine: So I don't see why |
|
So the goos and goarch attribute affect the aspect that compiles go code. They cannot change the c compiler, and the aspect does not travel across cc_library and cc_binary rules, so using them to cross compile cgo is never going to work, and there is really nothing I can do about that. What you are talking about I think is a special case however, where the c compiler has been switched for you by the apple rules (and is thus not the default target platform), and you are just forcing the go aspect to match it, which is not quite the same thing. The problem in this case is that in order to compile cgo, we have to have a cc_library and cc_binary rule in the chain, which again the aspect cannot travel across. The inputs to those rules come from the cgo_codegen step (because that's how we make the c files being passed in) and thus the cgo_codegen happens without the benefit of the aspect, and thus does not know what the target goos and goarch are, so cgo_codegen always happens for the default platform as specified on the command line. I cannot think of any way to fix this until we get the ability to generate cc actions (rather than rules) so that we have control over them and can transit through them with the aspect. |
|
Agreed, however, I am manually setting |
|
I did find that removing the |
|
Ok I got it to work. I'm not sure it's upstreamable (although it shouldn't break anything), but I'm now able to fully build for iOS with it: https://github.com/znly/rules_go/commit/6481b91631d2eeeae9930baf3832835658641441 Basically I'm manually setting goos/goarch for the stdlib rule via config_setting/select. Not that great, but it gets the job done and shouldn't break anybody. |
Hi,
As part of implementing
c-archiveandc-sharedsupport, I'm trying to properly propagate CPU selection to the Go toolchain when doing things such as building an iOS app or library.See https://github.com/znly/rules_go/commit/344d1337b0350ea5e3b9e712437d38f33054d8a7 and https://github.com/znly/rules_go/commit/fce5ac2d1f8a18ae77712cd392c76b54cb03e2d8.
I'm am quite confused as to how to do that properly based on constraints.
I am able to properly
selectbased on aconfig_settingsuch as:However, when trying to turn that into a constraint, in doing either:
Or even:
And adding that constraints to the arch in
list.bzl, it fails to properly select the correct target based on theios_applicationI'm building, whereconfig_settingworks as expected.I'm using the following command line
This isn important because in the case of iOS one could have multiple versions of the
c-archivebuilt on different archs (armv7 and arm64).Thanks!
The text was updated successfully, but these errors were encountered: