Add a mechanism to provide a list of protocol names for constant value extraction. #1170

BalestraPatrick · 2024-02-19T13:56:14Z

This cherry-picks support for producing const values produced via the compiler flags -emit-const-values-path and -const-gather-protocols-file. rules_apple will start using this feature in bazelbuild/rules_apple#2418.

swift/internal/compiling.bzl

keith · 2024-02-28T23:59:26Z

what's the general progress on this?

BalestraPatrick · 2024-02-29T20:51:13Z

I just verified this seems to be working in our project together with bazelbuild/rules_apple#2418

keith

were there any tests for this we could grab?

swift/internal/compiling.bzl

swift/internal/xcode_swift_toolchain.bzl

BalestraPatrick · 2024-03-06T22:24:49Z

@keith There were no tests upstream, but I tried my best at adding some. I also had to disable them on Linux due to the fact that we're using Swift 5.7 there, and from what I can see support for the flags we're using started in Swift 5.8.

BalestraPatrick · 2024-03-07T17:02:48Z

When testing this internally with a remote cache I'm getting Expected output Foo.swiftconstvalues was not created locally., so still need to investigate that.

Edit: removing our usage of --swiftcopt=-disallow-use-new-driver fixed the issue. I believe the old driver might not like this specific flag.

…e extraction. If a module populates `const_gather_protocols` when calling `create_swift_module_context`, any target that directly depends on that module will automatically have that list of protocols passed to the compiler via `-const-gather-protocols-file`. The output group `const_values` will then contain the JSON file with the extracted constant information. This is meant to support Apple's AppIntents framework, which uses the output from `-emit-const-values-path` as an input to the AppIntent bundling tool. PiperOrigin-RevId: 588388131 (cherry picked from commit 18f2f87)

swift/internal/feature_names.bzl

swift/internal/providers.bzl

swift/internal/swift_toolchain.bzl

…e extraction. If a module populates `const_gather_protocols` when calling `create_swift_module_context`, any target that directly depends on that module will automatically have that list of protocols passed to the compiler via `-const-gather-protocols-file`. The output group `const_values` will then contain the JSON file with the extracted constant information. This is meant to support Apple's AppIntents framework, which uses the output from `-emit-const-values-path` as an input to the AppIntent bundling tool. PiperOrigin-RevId: 588388131

Depends on bazelbuild/rules_swift#1170 --------- Co-authored-by: Nicholas Levin <[email protected]> Co-authored-by: Patrick Balestra <[email protected]> Co-authored-by: Luis Padron <[email protected]>

It cherry-picks the rule change of bazelbuild#1170 on top of rules_swift v1.6.0. It's quired to fix app intent on Xcode 15.3+. See bazelbuild/rules_apple#2418.

Since we extracting `swiftconstvalues` from #1170 But we forgot to update the path for swiftconstvalues in the incremental output file map. It leads to the incremental feature not working. For example, demo: https://github.com/vikage/DemoBazelSwiftIncremental When I change file B.swift (No other file uses struct B) but log shows build A.swift and C.swift as well I used `-driver-show-job-lifecycle` and `-driver-show-incremental` flags to show debug log ``` WARNING: Build option --swiftcopt has changed, discarding analysis cache (this can be expensive, see https://bazel.build/advanced/performance/iteration-speed). INFO: Analyzed target //:Demo (99 packages loaded, 1088 targets configured). INFO: From Compiling Swift module //:Demo: remark: Incremental compilation: Read dependency graph '/private/var/tmp/_bazel_elliotvu/e512a8189164dd2ae5386a6bdb0cc34e/execroot/_main/bazel-out/darwin_arm64-fastbuild/bin/_swift_incremental/Demo.output_file_map.priors' remark: Incremental compilation: Enabling incremental cross-module building remark: Incremental compilation: May skip current input: {compile: A.swift.o <= A.swift} remark: Incremental compilation: May skip current input: {compile: B.swift.o <= B.swift} remark: Incremental compilation: May skip current input: {compile: C.swift.o <= C.swift} remark: Incremental compilation: Missing an output; will queue {compile: A.swift.o <= A.swift} remark: Incremental compilation: Missing an output; will queue {compile: B.swift.o <= B.swift} remark: Incremental compilation: Missing an output; will queue {compile: C.swift.o <= C.swift} remark: Incremental compilation: Queuing (initial): {compile: A.swift.o <= A.swift} remark: Incremental compilation: Queuing (initial): {compile: B.swift.o <= B.swift} remark: Incremental compilation: Queuing (initial): {compile: C.swift.o <= C.swift} remark: Found 3 batchable jobs remark: Forming into 1 batch remark: Adding {compile: A.swift} to batch 0 remark: Adding {compile: B.swift} to batch 0 remark: Adding {compile: C.swift} to batch 0 remark: Forming batch job from 3 constituents: A.swift, B.swift, C.swift remark: Starting Emitting module for Demo remark: Finished Emitting module for Demo remark: Starting Compiling A.swift, B.swift, C.swift remark: Finished Compiling A.swift, B.swift, C.swift remark: Incremental compilation: Reading dependencies from A.swift remark: Incremental compilation: Reading dependencies from B.swift remark: Incremental compilation: Reading dependencies from C.swift remark: Incremental compilation: Scheduling all post-compile jobs because something was compiled ``` These are logs after fix. As you can see, only B.swift will be compile ``` INFO: Analyzed target //:Demo (0 packages loaded, 0 targets configured). INFO: From Compiling Swift module //:Demo: remark: Incremental compilation: Read dependency graph '/private/var/tmp/_bazel_elliotvu/e512a8189164dd2ae5386a6bdb0cc34e/execroot/_main/bazel-out/darwin_arm64-fastbuild/bin/_swift_incremental/Demo.output_file_map.priors' remark: Incremental compilation: Enabling incremental cross-module building remark: Incremental compilation: May skip current input: {compile: A.swift.o <= A.swift} remark: Incremental compilation: Scheduling changed input {compile: B.swift.o <= B.swift} remark: Incremental compilation: May skip current input: {compile: C.swift.o <= C.swift} remark: Incremental compilation: Queuing (initial): {compile: B.swift.o <= B.swift} remark: Incremental compilation: not scheduling dependents of B.swift; unknown changes remark: Incremental compilation: Skipping input: {compile: A.swift.o <= A.swift} remark: Incremental compilation: Skipping input: {compile: C.swift.o <= C.swift} remark: Found 1 batchable job remark: Forming into 1 batch remark: Adding {compile: B.swift} to batch 0 remark: Forming batch job from 1 constituents: B.swift remark: Starting Emitting module for Demo remark: Finished Emitting module for Demo remark: Starting Compiling B.swift remark: Finished Compiling B.swift remark: Incremental compilation: Reading dependencies from B.swift remark: Incremental compilation: Fingerprint changed for existing interface of top-level name 'B' in B.swift remark: Incremental compilation: Fingerprint changed for existing implementation of top-level name 'B' in B.swift remark: Incremental compilation: Fingerprint changed for existing interface of type '4Demo1BV' in B.swift remark: Incremental compilation: Fingerprint changed for existing implementation of type '4Demo1BV' in B.swift remark: Incremental compilation: Fingerprint changed for existing interface of potential members of '4Demo1BV' in B.swift remark: Incremental compilation: Fingerprint changed for existing implementation of potential members of '4Demo1BV' in B.swift remark: Incremental compilation: Scheduling all post-compile jobs because something was compiled remark: Skipped Compiling A.swift remark: Skipped Compiling C.swift ``` The solution here should be to remap swiftconstvalues path to `_swift_incremental` directory Related issue: #1291

Since we extracting `swiftconstvalues` from bazelbuild#1170 But we forgot to update the path for swiftconstvalues in the incremental output file map. It leads to the incremental feature not working. For example, demo: https://github.com/vikage/DemoBazelSwiftIncremental When I change file B.swift (No other file uses struct B) but log shows build A.swift and C.swift as well I used `-driver-show-job-lifecycle` and `-driver-show-incremental` flags to show debug log ``` WARNING: Build option --swiftcopt has changed, discarding analysis cache (this can be expensive, see https://bazel.build/advanced/performance/iteration-speed). INFO: Analyzed target //:Demo (99 packages loaded, 1088 targets configured). INFO: From Compiling Swift module //:Demo: remark: Incremental compilation: Read dependency graph '/private/var/tmp/_bazel_elliotvu/e512a8189164dd2ae5386a6bdb0cc34e/execroot/_main/bazel-out/darwin_arm64-fastbuild/bin/_swift_incremental/Demo.output_file_map.priors' remark: Incremental compilation: Enabling incremental cross-module building remark: Incremental compilation: May skip current input: {compile: A.swift.o <= A.swift} remark: Incremental compilation: May skip current input: {compile: B.swift.o <= B.swift} remark: Incremental compilation: May skip current input: {compile: C.swift.o <= C.swift} remark: Incremental compilation: Missing an output; will queue {compile: A.swift.o <= A.swift} remark: Incremental compilation: Missing an output; will queue {compile: B.swift.o <= B.swift} remark: Incremental compilation: Missing an output; will queue {compile: C.swift.o <= C.swift} remark: Incremental compilation: Queuing (initial): {compile: A.swift.o <= A.swift} remark: Incremental compilation: Queuing (initial): {compile: B.swift.o <= B.swift} remark: Incremental compilation: Queuing (initial): {compile: C.swift.o <= C.swift} remark: Found 3 batchable jobs remark: Forming into 1 batch remark: Adding {compile: A.swift} to batch 0 remark: Adding {compile: B.swift} to batch 0 remark: Adding {compile: C.swift} to batch 0 remark: Forming batch job from 3 constituents: A.swift, B.swift, C.swift remark: Starting Emitting module for Demo remark: Finished Emitting module for Demo remark: Starting Compiling A.swift, B.swift, C.swift remark: Finished Compiling A.swift, B.swift, C.swift remark: Incremental compilation: Reading dependencies from A.swift remark: Incremental compilation: Reading dependencies from B.swift remark: Incremental compilation: Reading dependencies from C.swift remark: Incremental compilation: Scheduling all post-compile jobs because something was compiled ``` These are logs after fix. As you can see, only B.swift will be compile ``` INFO: Analyzed target //:Demo (0 packages loaded, 0 targets configured). INFO: From Compiling Swift module //:Demo: remark: Incremental compilation: Read dependency graph '/private/var/tmp/_bazel_elliotvu/e512a8189164dd2ae5386a6bdb0cc34e/execroot/_main/bazel-out/darwin_arm64-fastbuild/bin/_swift_incremental/Demo.output_file_map.priors' remark: Incremental compilation: Enabling incremental cross-module building remark: Incremental compilation: May skip current input: {compile: A.swift.o <= A.swift} remark: Incremental compilation: Scheduling changed input {compile: B.swift.o <= B.swift} remark: Incremental compilation: May skip current input: {compile: C.swift.o <= C.swift} remark: Incremental compilation: Queuing (initial): {compile: B.swift.o <= B.swift} remark: Incremental compilation: not scheduling dependents of B.swift; unknown changes remark: Incremental compilation: Skipping input: {compile: A.swift.o <= A.swift} remark: Incremental compilation: Skipping input: {compile: C.swift.o <= C.swift} remark: Found 1 batchable job remark: Forming into 1 batch remark: Adding {compile: B.swift} to batch 0 remark: Forming batch job from 1 constituents: B.swift remark: Starting Emitting module for Demo remark: Finished Emitting module for Demo remark: Starting Compiling B.swift remark: Finished Compiling B.swift remark: Incremental compilation: Reading dependencies from B.swift remark: Incremental compilation: Fingerprint changed for existing interface of top-level name 'B' in B.swift remark: Incremental compilation: Fingerprint changed for existing implementation of top-level name 'B' in B.swift remark: Incremental compilation: Fingerprint changed for existing interface of type '4Demo1BV' in B.swift remark: Incremental compilation: Fingerprint changed for existing implementation of type '4Demo1BV' in B.swift remark: Incremental compilation: Fingerprint changed for existing interface of potential members of '4Demo1BV' in B.swift remark: Incremental compilation: Fingerprint changed for existing implementation of potential members of '4Demo1BV' in B.swift remark: Incremental compilation: Scheduling all post-compile jobs because something was compiled remark: Skipped Compiling A.swift remark: Skipped Compiling C.swift ``` The solution here should be to remap swiftconstvalues path to `_swift_incremental` directory Related issue: bazelbuild#1291

Since we extracting `swiftconstvalues` from #1170 But we forgot to update the path for swiftconstvalues in the incremental output file map. It leads to the incremental feature not working. For example, demo: https://github.com/vikage/DemoBazelSwiftIncremental When I change file B.swift (No other file uses struct B) but log shows build A.swift and C.swift as well I used `-driver-show-job-lifecycle` and `-driver-show-incremental` flags to show debug log ``` WARNING: Build option --swiftcopt has changed, discarding analysis cache (this can be expensive, see https://bazel.build/advanced/performance/iteration-speed). INFO: Analyzed target //:Demo (99 packages loaded, 1088 targets configured). INFO: From Compiling Swift module //:Demo: remark: Incremental compilation: Read dependency graph '/private/var/tmp/_bazel_elliotvu/e512a8189164dd2ae5386a6bdb0cc34e/execroot/_main/bazel-out/darwin_arm64-fastbuild/bin/_swift_incremental/Demo.output_file_map.priors' remark: Incremental compilation: Enabling incremental cross-module building remark: Incremental compilation: May skip current input: {compile: A.swift.o <= A.swift} remark: Incremental compilation: May skip current input: {compile: B.swift.o <= B.swift} remark: Incremental compilation: May skip current input: {compile: C.swift.o <= C.swift} remark: Incremental compilation: Missing an output; will queue {compile: A.swift.o <= A.swift} remark: Incremental compilation: Missing an output; will queue {compile: B.swift.o <= B.swift} remark: Incremental compilation: Missing an output; will queue {compile: C.swift.o <= C.swift} remark: Incremental compilation: Queuing (initial): {compile: A.swift.o <= A.swift} remark: Incremental compilation: Queuing (initial): {compile: B.swift.o <= B.swift} remark: Incremental compilation: Queuing (initial): {compile: C.swift.o <= C.swift} remark: Found 3 batchable jobs remark: Forming into 1 batch remark: Adding {compile: A.swift} to batch 0 remark: Adding {compile: B.swift} to batch 0 remark: Adding {compile: C.swift} to batch 0 remark: Forming batch job from 3 constituents: A.swift, B.swift, C.swift remark: Starting Emitting module for Demo remark: Finished Emitting module for Demo remark: Starting Compiling A.swift, B.swift, C.swift remark: Finished Compiling A.swift, B.swift, C.swift remark: Incremental compilation: Reading dependencies from A.swift remark: Incremental compilation: Reading dependencies from B.swift remark: Incremental compilation: Reading dependencies from C.swift remark: Incremental compilation: Scheduling all post-compile jobs because something was compiled ``` These are logs after fix. As you can see, only B.swift will be compile ``` INFO: Analyzed target //:Demo (0 packages loaded, 0 targets configured). INFO: From Compiling Swift module //:Demo: remark: Incremental compilation: Read dependency graph '/private/var/tmp/_bazel_elliotvu/e512a8189164dd2ae5386a6bdb0cc34e/execroot/_main/bazel-out/darwin_arm64-fastbuild/bin/_swift_incremental/Demo.output_file_map.priors' remark: Incremental compilation: Enabling incremental cross-module building remark: Incremental compilation: May skip current input: {compile: A.swift.o <= A.swift} remark: Incremental compilation: Scheduling changed input {compile: B.swift.o <= B.swift} remark: Incremental compilation: May skip current input: {compile: C.swift.o <= C.swift} remark: Incremental compilation: Queuing (initial): {compile: B.swift.o <= B.swift} remark: Incremental compilation: not scheduling dependents of B.swift; unknown changes remark: Incremental compilation: Skipping input: {compile: A.swift.o <= A.swift} remark: Incremental compilation: Skipping input: {compile: C.swift.o <= C.swift} remark: Found 1 batchable job remark: Forming into 1 batch remark: Adding {compile: B.swift} to batch 0 remark: Forming batch job from 1 constituents: B.swift remark: Starting Emitting module for Demo remark: Finished Emitting module for Demo remark: Starting Compiling B.swift remark: Finished Compiling B.swift remark: Incremental compilation: Reading dependencies from B.swift remark: Incremental compilation: Fingerprint changed for existing interface of top-level name 'B' in B.swift remark: Incremental compilation: Fingerprint changed for existing implementation of top-level name 'B' in B.swift remark: Incremental compilation: Fingerprint changed for existing interface of type '4Demo1BV' in B.swift remark: Incremental compilation: Fingerprint changed for existing implementation of type '4Demo1BV' in B.swift remark: Incremental compilation: Fingerprint changed for existing interface of potential members of '4Demo1BV' in B.swift remark: Incremental compilation: Fingerprint changed for existing implementation of potential members of '4Demo1BV' in B.swift remark: Incremental compilation: Scheduling all post-compile jobs because something was compiled remark: Skipped Compiling A.swift remark: Skipped Compiling C.swift ``` The solution here should be to remap swiftconstvalues path to `_swift_incremental` directory Related issue: #1291 Co-authored-by: Thanh Vu <[email protected]>

BalestraPatrick force-pushed the constant-value-extraction branch 3 times, most recently from cbf792b to a3529d0 Compare February 24, 2024 22:27

BalestraPatrick commented Feb 27, 2024

View reviewed changes

swift/internal/compiling.bzl Outdated Show resolved Hide resolved

keith mentioned this pull request Feb 29, 2024

Fix app intents with Xcode 15.3+ bazelbuild/rules_apple#2418

Merged

BalestraPatrick marked this pull request as ready for review February 29, 2024 20:51

BalestraPatrick requested review from brentleyjones, luispadron, mattrobmattrob and thii as code owners February 29, 2024 20:51

keith reviewed Mar 1, 2024

View reviewed changes

swift/internal/compiling.bzl Outdated Show resolved Hide resolved

swift/internal/compiling.bzl Outdated Show resolved Hide resolved

swift/internal/compiling.bzl Outdated Show resolved Hide resolved

swift/internal/xcode_swift_toolchain.bzl Outdated Show resolved Hide resolved

BalestraPatrick force-pushed the constant-value-extraction branch 6 times, most recently from b179ff3 to cc6abe8 Compare March 6, 2024 22:23

allevato and others added 9 commits May 16, 2024 10:20

Fixes

a191c7d

Update docs

c10cc40

Run buildifier

1facdf8

Add private feature

6273abd

Fix buildifier

4ef9b96

Update docs

18d3264

Small fixes

c6b9a11

Change minimum Xcode version for const value extraction to 15.0

2aeb7c6

BalestraPatrick added 5 commits May 16, 2024 10:23

Refactor file creation logic to derived_files.bzl

7f092c6

Move const_extract_protocol.json to static file in toolchain

9034980

Add const_vaues tests

a806419

Disable some tests on Linux given the older Swift version

7c2e1ce

Fix conflict resolution

be307b2

BalestraPatrick force-pushed the constant-value-extraction branch from cc6abe8 to be307b2 Compare May 16, 2024 09:14

Update docs and fix buildifier failure

aad53d8

BalestraPatrick force-pushed the constant-value-extraction branch from 2bd5d82 to aad53d8 Compare May 16, 2024 09:35

luispadron approved these changes May 17, 2024

View reviewed changes

swift/internal/feature_names.bzl Outdated Show resolved Hide resolved

swift/internal/providers.bzl Show resolved Hide resolved

swift/internal/swift_toolchain.bzl Outdated Show resolved Hide resolved

keith approved these changes May 17, 2024

View reviewed changes

Fix review comments

c2fde06

BalestraPatrick merged commit 4cc4c95 into bazelbuild:master May 17, 2024
15 checks passed

BalestraPatrick deleted the constant-value-extraction branch May 17, 2024 18:40

vikage mentioned this pull request Dec 31, 2024

Map const-values in output file map for incremental #1471

Merged

adincebic mentioned this pull request Jan 17, 2025

Cherry-pick: fix for incremental compilation #1476

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add a mechanism to provide a list of protocol names for constant value extraction. #1170

Add a mechanism to provide a list of protocol names for constant value extraction. #1170

BalestraPatrick commented Feb 19, 2024 •

edited

Loading

keith commented Feb 28, 2024

BalestraPatrick commented Feb 29, 2024

keith left a comment

BalestraPatrick commented Mar 6, 2024

BalestraPatrick commented Mar 7, 2024 •

edited

Loading

Add a mechanism to provide a list of protocol names for constant value extraction. #1170

Add a mechanism to provide a list of protocol names for constant value extraction. #1170

Conversation

BalestraPatrick commented Feb 19, 2024 • edited Loading

keith commented Feb 28, 2024

BalestraPatrick commented Feb 29, 2024

keith left a comment

Choose a reason for hiding this comment

BalestraPatrick commented Mar 6, 2024

BalestraPatrick commented Mar 7, 2024 • edited Loading

BalestraPatrick commented Feb 19, 2024 •

edited

Loading

BalestraPatrick commented Mar 7, 2024 •

edited

Loading