test_exec_properties differentiation

[werkt]

Description of the problem / feature request:

Provide a test_exec_properties option for *_test rules.

Feature requests: what underlying problem are you trying to solve with this feature?

The test execution platform properties should be differentiated from exec_properties. Currently setting an exec_properties of a *_test will apply to the actions used to build the target, as well as the test itself. In order to apply specific runtime configurations to the platform of the test, but not, say, the compiles of a cc_test, a different rule attribute is required.

[jlisee]

This would be a really great feature to have. Some example "user stories":

• Alice wants to say this test needs 16 cores and 64GB of RAM, but allow the compilations of the 10 files in the test to use only a single core and 8GB of RAM. This greatly reduces the resources needed for the test.

• John wants to says this test needs a GPU and 128GB RAM but not consume GPU resources when build the files for that test

Without this level of control we are blocking on fully leveraging a large shared Buildfarm cluster. To do this we need to give each job just the resources it needs and those needs widely vary. With wide ranges in needed cores, memory, wall clock time, network, GPU requirements, and overall duration.

[katre]

Taking a look at this (and the PR #10866), I think this is strongly related to the proposal for Action Groups. Action groups would allow test rules to declare that the test execution actions are in a separate, named group, and then targets can use the proposed syntax:

my_test(
  ...
  exec_properties = {
    "test": {
      "cores": "16",
      "ram": "64GB",
    },
  },
)

The proposal itself doesn't mention having all tests create this action group, but I think that would be a reasonable thing to implement once the functionality is available, and doesn't require either a) a new globally-available attribute, or b) more target-specific knowledge baked into Bazel's core.

(Pinging @juliexxia to make sure we are considering this use for Action Groups).

[werkt]

I'm just stuck on not being able to do this from the target itself without the action group - exec_properties as identified here should have the same principles as those in rule-base, it could even be left backwards compatible with dict detection in values allowing the new behavior.

What is the ETA on ActionGroups? That along with the above is what matters to me.

[juliexxia]

ActionGroups is in the design review stage now and will hopefully be approved and ready for implementation by early next week. It's a P0 OKR for Q1 but since we only have a month of Q1 left I wouldn't be surprised if some work bled into Q2.

@katre are you proposing that all test rules automatically have the "test" action group? That seems reasonable to me

[juliexxia]

Added a note about this to the 'Potential Next Steps' section of the action groups doc: https://docs.google.com/document/d/1m9xLRLg09lncQTuBUMhhG_lqDQGJRYMZPejvwdBXvqo/edit#bookmark=id.nh45ww714xso

[brandjon]

If we're going to have a special action group for "test" actions, does that mean we should also have a special action group for "run" actions, used when you do a bazel run command? And would they be equivalent for test actions?

[brandjon]

Also it's not clear to me what the execution platform for a "run" or "test" action group would be. By definition, wouldn't it have to be the same as the target platform of that target? Likewise, do toolchains for "run" or "test" action groups make sense? If these concepts do not map well, it may be better to think of "run"/"test" as just things that exec properties can apply to, and not as proper action groups.

[katre]

"Run" is a special case: it's not technically an action generated by a rule, it's Bazel executing the output from a build step.

"Test" actions are a separate issue: Bazel first builds the test executable, then executes it, then examines the output and reports on it.

[brandjon]

From a user's POV is there any reason to call one an action and not the other? Or is this just an implementation difference?

If they (or just "Test") are actions, do execution platforms and toolchains make sense for them, or only execution properties?

If "Run" is not an "action", can execution properties apply to it nonetheless?

[katre]

"Run" always means "run on the host platform". Ideally we'd change the default of the "--platforms" flag to make that more true, but for now there's a chance of mismatch.

"Test" is trickier: you're right that it's not technically an action, but there are several requests to manage test execution with platforms: using a platform to describe the remote platform where a test executes, or an android emulator that runs the test process, for example. Setting specific execution properties for a test for another.

[brandjon]

In that case I suggest that "Run" and/or "Test" not be considered "action groups", even if they share some common subset of features with action groups -- "things that can have execution properties" or what not.

That said, maybe we want to make "run"/"test" reserved names from action groups, so that they can be used in execution properties dictionaries. (Of course, that complicates any code that needs to introspect the execution properties dict and treat the keys like action groups, kinda like how code that iterates over struct fields must discard to_json and to_proto.)

[werkt]

So how's ActionGroups going? is there a tracking issue proper for it, or is this it?

[katre]

@juliexxia has been working on it, I'll let her update. @juliexxia, is there a Github tracking issue?

[juliexxia]

Newly minted as of two seconds ago: #11250

I back-added some existing commits that are contributing work and I'll attach the issue to commits going forward.

[juliexxia]

@werkt Thanks so much for this PR and apologies it's taken so long to get you and update. Now that native exec groups are fully landed I'm in the process of adding an exec group for TestRunnerActions that can be used something like:

some_test_rule(
  name = "my-test",
  exec_properties = {
    "property-a": "value-a"
    "test.property-b": "value-b",
  },
  ...
)

All actions produced by the test rule will see the exec properties dict {"property-a", "value-a"} and TestRunnerActions will see {"property-a": "value-a", "property-b": "value-b"}.

I think that should cover your use case (your PR was particularly helpful for helping me understand it) but let me know if it doesn't sound like it will!

[jlisee]

@juliexxia what version of Bazel supports the test.property-b syntax for exec_properties (or what is the target release)?

Also an aside: does the exec groups system allow me to apply tags like no-remote only to the compile or link actions of a cc_compile target for example?

[juliexxia]

@jlisee, the test.property-b syntax isn't released yet. It's in development, blocked by #12006 which I'm waiting on comments from from the bazel starlark team.

re: no-remote, etc. No, not yet. But that's in the pipeline for the coming quarter or so. The plan is to deprecate that functionality from tags and add it to exec_properties. Doc is in the works.

[ulfjack]

We have a use case where we want to run tests on a different architecture from builds. Ideally, this would not require annotating every test rule individually, for example, this could be inferred from the target platform (does such a thing exist?). As far as I can tell from the source code, there is no working mechanism for this right now. I'd also be happy to file this as a separate bug - it might be sufficiently different from this proposal.

[juliexxia]

When c266ac9 is released, prefixing exec_properties entries with test. will apply the property to test actions only e.g.

exec_properties = {
    "test.mem": "16g", 
    "mem": "12g",
}

Test actions would see the exec properties dict {"mem": "16g"} and other actions would see the dict {"mem": "12g"}

[werkt]

Nice job on this. I will take a look at the capabilities, as I'm curious of the extents of this feature from the starlark side - I hope that more than just 'test' is available

[ulfjack]

It looks like this was rolled back. The linked error logs indicate a failure in curl, and I don't see how that could be related.

[EdSchouten]

This can be closed, right? You can nowadays use exec_properties = {"test.foo": "bar"} to set properties on the test exec group.

[jerrymarino]

Overall this works quite well for me - thanks for fixing 👍 !

[werkt]

Considering this completed, docs are available at https://bazel.build/extending/exec-groups (not sure if this was ever linked)

[anguslees]

re: no-remote, etc. No, not yet. But that's in the pipeline for the coming quarter or so. The plan is to deprecate that functionality from tags and add it to exec_properties. Doc is in the works.

Did this ever get implemented? I'm trying exec_propertes = {"test.no-remote-exec": "true"} and it seems no-remote-exec is (incorrectly) sent to my remote platform, which then chokes. (Bazel 7.2.0)

[katre]

@anguslees: You should be able to use exec_properties like this: setting test.no-remote-exec should only apply no-remote-exec to the actions in the test exec group (which is the actual test execution).

It's possible that there are bugs here: the overall project to merge tags, execution requirements, and execution properties stalled (mostly because it's complex, no one has time to take it on, and dealing with cleanups in existing codebases is a long slog).

If this is not working properly, please file a new issue with a reproduction and we'll try to get someone to take a look.