Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[internal] python: use immutable_input_digests for protobuf codegen #13885

Merged
merged 1 commit into from
Jan 8, 2022
Merged

[internal] python: use immutable_input_digests for protobuf codegen #13885

merged 1 commit into from
Jan 8, 2022

Conversation

tdyas
Copy link
Contributor

@tdyas tdyas commented Dec 14, 2021

For Python protobuf codegen, use the immutable_input_digests feature of process execution for the protoc binary.

[ci skip-rust]

stuhood
stuhood approved these changes Dec 14, 2021
View reviewed changes
src/python/pants/backend/codegen/protobuf/python/rules.py
Comment on lines +181 to +183
immutable_input_digests={
protoc_relpath: downloaded_protoc_binary.digest,
},
Copy link
Sponsor Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Consider exposing immutable_inputs_digests as a property of DownloadedExternalTool, similar to

pants/src/python/pants/jvm/resolve/coursier_setup.py

Lines 132 to 134 in 6f85d25

@property
def immutable_input_digests(self) -> dict[str, Digest]:
return {self.bin_dir: self._digest}
? Could add immutable_inputs_exe too.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I love this idea!

Eric-Arellano
Eric-Arellano approved these changes Dec 14, 2021
View reviewed changes
Copy link
Contributor

@Eric-Arellano Eric-Arellano left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Awesome!! Not blocking, but I'm really curious to see a benchmark.

src/python/pants/backend/codegen/protobuf/python/rules.py
Comment on lines +181 to +183
immutable_input_digests={
protoc_relpath: downloaded_protoc_binary.digest,
},
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I love this idea!

@Eric-Arellano
Copy link
Contributor

You can remove the [internal] from here if you'd like so that we can put it in Performance section of changelog :) it's nice to show users we're trying to improve performance.

@tdyas
Copy link
Contributor Author

tdyas commented Dec 15, 2021

Awesome!! Not blocking, but I'm really curious to see a benchmark.

Is there a particular benchmark you had in mind? Just curious if there is a test project and preferred Pants goal to use.

@Eric-Arellano
Copy link
Contributor

Just curious if there is a test project

Oh yeah I guess we don't have one anymore, now that example-python deleted Protobuf. Could maybe run one of the unit tests n times, but might have too much noise.

Otherwise if you'd be willing to set up a simple temporary testproject, the least noise would be using --no-local-cache --no-pantsd export-codegen testproject::

Again, not blocking. Only curious

python: use immutable_input_digests for protobuf codegen
1ae1c26 
[ci skip-rust]

[ci skip-build-wheels]
@tdyas tdyas force-pushed the python_protobuf_use_immutable_inputs branch from 7ecae2c to 1ae1c26 Compare January 8, 2022 18:11
@tdyas tdyas merged commit 0e0ee2a into pantsbuild:main Jan 8, 2022
@tdyas tdyas deleted the python_protobuf_use_immutable_inputs branch January 8, 2022 21:37
kaos pushed a commit to kaos/pants that referenced this pull request Jan 9, 2022
[internal] python: use immutable_input_digests for protobuf codegen (p…
8815b65 
…antsbuild#13885)

For Python protobuf codegen, use the `immutable_input_digests` feature of process execution for the `protoc` binary.

[ci skip-rust]
asherf pushed a commit to asherf/pants that referenced this pull request Jan 14, 2022
@asherf
[internal] python: use immutable_input_digests for protobuf codegen (p…
1126ce7 
…antsbuild#13885)

For Python protobuf codegen, use the `immutable_input_digests` feature of process execution for the `protoc` binary.

[ci skip-rust]
illicitonion added a commit to illicitonion/pants that referenced this pull request Jan 22, 2022
@illicitonion
Prep for 2.10.0.dev2
bcb22d6 
Internal changes:

* upgrade to Rust v1.58.0 ([pantsbuild#14174](pantsbuild#14174))

* [internal] fix typos in codegen registration ([pantsbuild#14172](pantsbuild#14172))

* Remove per-language indirection for formatter plugins. ([pantsbuild#14166](pantsbuild#14166))

* Pulls `Coordinate` and `ArtifactRequirements` out into a separate file to avoid a circuilar import later ([pantsbuild#14164](pantsbuild#14164))

* Factors lockfile metadata code into class that supports multiple languages as well as versions ([pantsbuild#14116](pantsbuild#14116))

* Adds `.env` file to make vscode auto-imports work correctly out of the box ([pantsbuild#14130](pantsbuild#14130))

* [internal] Rename classes for `generate_lockfiles.py` for clarity ([pantsbuild#14218](pantsbuild#14218))

* [internal] Fix bad Find+Replace ([pantsbuild#14213](pantsbuild#14213))

* [internal] Bump libc from 0.2.106 to 0.2.112 in /src/rust/engine ([pantsbuild#14206](pantsbuild#14206))

* [internal] Bump tempfile from 3.2.0 to 3.3.0 in /src/rust/engine ([pantsbuild#14202](pantsbuild#14202))

* [internal] Bump criterion from 0.3.3 to 0.3.5 in /src/rust/engine ([pantsbuild#14205](pantsbuild#14205))

* [internal] Bump walkdir from 2.3.1 to 2.3.2 in /src/rust/engine ([pantsbuild#14204](pantsbuild#14204))

* [internal] Bump generic-array from 0.14.4 to 0.14.5 in /src/rust/engine ([pantsbuild#14203](pantsbuild#14203))

* [internal] Remove the minimum bucket size of batching to improve stability. ([pantsbuild#14210](pantsbuild#14210))

* [internal] Make `JvmLockfileRequest` generic ([pantsbuild#14201](pantsbuild#14201))

* [internal] add comment re clippy issue ([pantsbuild#14188](pantsbuild#14188))

* [internal] rename some codegen scopes to put language first ([pantsbuild#14187](pantsbuild#14187))

* [internal] Check for ambiguity when running `generate-lockfiles` ([pantsbuild#14178](pantsbuild#14178))

* [internal] Change JVM lockfile format ([pantsbuild#14175](pantsbuild#14175))

* [internal] go: rewrite third party package analysis to not use `go list` ([pantsbuild#14157](pantsbuild#14157))

* [internal] Hotfix `fmt` crashing on non-formattable targets ([pantsbuild#14168](pantsbuild#14168))

* [internal] Simplify `core/goals/package.py` filesystem API usage ([pantsbuild#14162](pantsbuild#14162))

* [internal] java/thrift: register union that was not registered ([pantsbuild#14159](pantsbuild#14159))

* [internal] Bump arc-swap from 1.3.0 to 1.5.0 in /src/rust/engine ([pantsbuild#14148](pantsbuild#14148))

* [internals] Bump tokio-rustls from 0.22.0 to 0.23.2 in /src/rust/engine ([pantsbuild#14149](pantsbuild#14149))

* [internal] Bump num_cpus from 1.13.0 to 1.13.1 in /src/rust/engine ([pantsbuild#14150](pantsbuild#14150))

* [internal] Bump tokio-util from 0.6.7 to 0.6.9 in /src/rust/engine ([pantsbuild#14151](pantsbuild#14151))

* [internal] Bump os_pipe from 0.9.2 to 1.0.0 in /src/rust/engine ([pantsbuild#14152](pantsbuild#14152))

* [internal] Introduce new `BuiltinGoal` subsystem type. ([pantsbuild#13991](pantsbuild#13991))

* [internal] Upgrade python type stubs packages ([pantsbuild#14143](pantsbuild#14143))

* [internal] Bump strum_macros from 0.20.1 to 0.23.1 in /src/rust/engine ([pantsbuild#14141](pantsbuild#14141))

* [internal] Make `generate-lockfiles` goal generic ([pantsbuild#14122](pantsbuild#14122))

* [internal] Bump errno from 0.2.7 to 0.2.8 in /src/rust/engine ([pantsbuild#14137](pantsbuild#14137))

* [internal] Bump colored from 1.9.3 to 2.0.0 in /src/rust/engine ([pantsbuild#14138](pantsbuild#14138))

* [internal] Bump crossbeam-channel from 0.4.4 to 0.5.0 in /src/rust/engine ([pantsbuild#14139](pantsbuild#14139))

* [internal] Bump reqwest from 0.11.4 to 0.11.9 in /src/rust/engine ([pantsbuild#14135](pantsbuild#14135))

* [internal] Further tweak dependabot config ([pantsbuild#14132](pantsbuild#14132))

* [internal] python: use immutable_input_digests for protobuf codegen ([pantsbuild#13885](pantsbuild#13885))

* [internal] python: use immutable_input_digests for protobuf codegen ([pantsbuild#13885](pantsbuild#13885))

* [internal] No need for two f-strings/two strings. ([pantsbuild#14119](pantsbuild#14119))
@illicitonion illicitonion mentioned this pull request Jan 22, 2022
Merged
illicitonion added a commit to illicitonion/pants that referenced this pull request Jan 22, 2022
@illicitonion
Prep for 2.10.0.dev2
6071ee3 
Internal changes:

* upgrade to Rust v1.58.0 ([pantsbuild#14174](pantsbuild#14174))

* [internal] fix typos in codegen registration ([pantsbuild#14172](pantsbuild#14172))

* Remove per-language indirection for formatter plugins. ([pantsbuild#14166](pantsbuild#14166))

* Pulls `Coordinate` and `ArtifactRequirements` out into a separate file to avoid a circuilar import later ([pantsbuild#14164](pantsbuild#14164))

* Factors lockfile metadata code into class that supports multiple languages as well as versions ([pantsbuild#14116](pantsbuild#14116))

* Adds `.env` file to make vscode auto-imports work correctly out of the box ([pantsbuild#14130](pantsbuild#14130))

* [internal] Rename classes for `generate_lockfiles.py` for clarity ([pantsbuild#14218](pantsbuild#14218))

* [internal] Fix bad Find+Replace ([pantsbuild#14213](pantsbuild#14213))

* [internal] Bump libc from 0.2.106 to 0.2.112 in /src/rust/engine ([pantsbuild#14206](pantsbuild#14206))

* [internal] Bump tempfile from 3.2.0 to 3.3.0 in /src/rust/engine ([pantsbuild#14202](pantsbuild#14202))

* [internal] Bump criterion from 0.3.3 to 0.3.5 in /src/rust/engine ([pantsbuild#14205](pantsbuild#14205))

* [internal] Bump walkdir from 2.3.1 to 2.3.2 in /src/rust/engine ([pantsbuild#14204](pantsbuild#14204))

* [internal] Bump generic-array from 0.14.4 to 0.14.5 in /src/rust/engine ([pantsbuild#14203](pantsbuild#14203))

* [internal] Remove the minimum bucket size of batching to improve stability. ([pantsbuild#14210](pantsbuild#14210))

* [internal] Make `JvmLockfileRequest` generic ([pantsbuild#14201](pantsbuild#14201))

* [internal] add comment re clippy issue ([pantsbuild#14188](pantsbuild#14188))

* [internal] rename some codegen scopes to put language first ([pantsbuild#14187](pantsbuild#14187))

* [internal] Check for ambiguity when running `generate-lockfiles` ([pantsbuild#14178](pantsbuild#14178))

* [internal] Change JVM lockfile format ([pantsbuild#14175](pantsbuild#14175))

* [internal] go: rewrite third party package analysis to not use `go list` ([pantsbuild#14157](pantsbuild#14157))

* [internal] Hotfix `fmt` crashing on non-formattable targets ([pantsbuild#14168](pantsbuild#14168))

* [internal] Simplify `core/goals/package.py` filesystem API usage ([pantsbuild#14162](pantsbuild#14162))

* [internal] java/thrift: register union that was not registered ([pantsbuild#14159](pantsbuild#14159))

* [internal] Bump arc-swap from 1.3.0 to 1.5.0 in /src/rust/engine ([pantsbuild#14148](pantsbuild#14148))

* [internals] Bump tokio-rustls from 0.22.0 to 0.23.2 in /src/rust/engine ([pantsbuild#14149](pantsbuild#14149))

* [internal] Bump num_cpus from 1.13.0 to 1.13.1 in /src/rust/engine ([pantsbuild#14150](pantsbuild#14150))

* [internal] Bump tokio-util from 0.6.7 to 0.6.9 in /src/rust/engine ([pantsbuild#14151](pantsbuild#14151))

* [internal] Bump os_pipe from 0.9.2 to 1.0.0 in /src/rust/engine ([pantsbuild#14152](pantsbuild#14152))

* [internal] Introduce new `BuiltinGoal` subsystem type. ([pantsbuild#13991](pantsbuild#13991))

* [internal] Upgrade python type stubs packages ([pantsbuild#14143](pantsbuild#14143))

* [internal] Bump strum_macros from 0.20.1 to 0.23.1 in /src/rust/engine ([pantsbuild#14141](pantsbuild#14141))

* [internal] Make `generate-lockfiles` goal generic ([pantsbuild#14122](pantsbuild#14122))

* [internal] Bump errno from 0.2.7 to 0.2.8 in /src/rust/engine ([pantsbuild#14137](pantsbuild#14137))

* [internal] Bump colored from 1.9.3 to 2.0.0 in /src/rust/engine ([pantsbuild#14138](pantsbuild#14138))

* [internal] Bump crossbeam-channel from 0.4.4 to 0.5.0 in /src/rust/engine ([pantsbuild#14139](pantsbuild#14139))

* [internal] Bump reqwest from 0.11.4 to 0.11.9 in /src/rust/engine ([pantsbuild#14135](pantsbuild#14135))

* [internal] Further tweak dependabot config ([pantsbuild#14132](pantsbuild#14132))

* [internal] python: use immutable_input_digests for protobuf codegen ([pantsbuild#13885](pantsbuild#13885))

* [internal] python: use immutable_input_digests for protobuf codegen ([pantsbuild#13885](pantsbuild#13885))

* [internal] No need for two f-strings/two strings. ([pantsbuild#14119](pantsbuild#14119))
illicitonion added a commit to illicitonion/pants that referenced this pull request Jan 22, 2022
@illicitonion
Prep for 2.10.0.dev2
237e3e1 
Internal changes:

* upgrade to Rust v1.58.0 ([pantsbuild#14174](pantsbuild#14174))

* [internal] fix typos in codegen registration ([pantsbuild#14172](pantsbuild#14172))

* Pulls `Coordinate` and `ArtifactRequirements` out into a separate file to avoid a circuilar import later ([pantsbuild#14164](pantsbuild#14164))

* Factors lockfile metadata code into class that supports multiple languages as well as versions ([pantsbuild#14116](pantsbuild#14116))

* Adds `.env` file to make vscode auto-imports work correctly out of the box ([pantsbuild#14130](pantsbuild#14130))

* [internal] Rename classes for `generate_lockfiles.py` for clarity ([pantsbuild#14218](pantsbuild#14218))

* [internal] Fix bad Find+Replace ([pantsbuild#14213](pantsbuild#14213))

* [internal] Bump libc from 0.2.106 to 0.2.112 in /src/rust/engine ([pantsbuild#14206](pantsbuild#14206))

* [internal] Bump tempfile from 3.2.0 to 3.3.0 in /src/rust/engine ([pantsbuild#14202](pantsbuild#14202))

* [internal] Bump criterion from 0.3.3 to 0.3.5 in /src/rust/engine ([pantsbuild#14205](pantsbuild#14205))

* [internal] Bump walkdir from 2.3.1 to 2.3.2 in /src/rust/engine ([pantsbuild#14204](pantsbuild#14204))

* [internal] Bump generic-array from 0.14.4 to 0.14.5 in /src/rust/engine ([pantsbuild#14203](pantsbuild#14203))

* [internal] Remove the minimum bucket size of batching to improve stability. ([pantsbuild#14210](pantsbuild#14210))

* [internal] Make `JvmLockfileRequest` generic ([pantsbuild#14201](pantsbuild#14201))

* [internal] add comment re clippy issue ([pantsbuild#14188](pantsbuild#14188))

* [internal] rename some codegen scopes to put language first ([pantsbuild#14187](pantsbuild#14187))

* [internal] Check for ambiguity when running `generate-lockfiles` ([pantsbuild#14178](pantsbuild#14178))

* [internal] Change JVM lockfile format ([pantsbuild#14175](pantsbuild#14175))

* [internal] go: rewrite third party package analysis to not use `go list` ([pantsbuild#14157](pantsbuild#14157))

* [internal] Hotfix `fmt` crashing on non-formattable targets ([pantsbuild#14168](pantsbuild#14168))

* [internal] Simplify `core/goals/package.py` filesystem API usage ([pantsbuild#14162](pantsbuild#14162))

* [internal] java/thrift: register union that was not registered ([pantsbuild#14159](pantsbuild#14159))

* [internal] Bump arc-swap from 1.3.0 to 1.5.0 in /src/rust/engine ([pantsbuild#14148](pantsbuild#14148))

* [internals] Bump tokio-rustls from 0.22.0 to 0.23.2 in /src/rust/engine ([pantsbuild#14149](pantsbuild#14149))

* [internal] Bump num_cpus from 1.13.0 to 1.13.1 in /src/rust/engine ([pantsbuild#14150](pantsbuild#14150))

* [internal] Bump tokio-util from 0.6.7 to 0.6.9 in /src/rust/engine ([pantsbuild#14151](pantsbuild#14151))

* [internal] Bump os_pipe from 0.9.2 to 1.0.0 in /src/rust/engine ([pantsbuild#14152](pantsbuild#14152))

* [internal] Introduce new `BuiltinGoal` subsystem type. ([pantsbuild#13991](pantsbuild#13991))

* [internal] Upgrade python type stubs packages ([pantsbuild#14143](pantsbuild#14143))

* [internal] Bump strum_macros from 0.20.1 to 0.23.1 in /src/rust/engine ([pantsbuild#14141](pantsbuild#14141))

* [internal] Make `generate-lockfiles` goal generic ([pantsbuild#14122](pantsbuild#14122))

* [internal] Bump errno from 0.2.7 to 0.2.8 in /src/rust/engine ([pantsbuild#14137](pantsbuild#14137))

* [internal] Bump colored from 1.9.3 to 2.0.0 in /src/rust/engine ([pantsbuild#14138](pantsbuild#14138))

* [internal] Bump crossbeam-channel from 0.4.4 to 0.5.0 in /src/rust/engine ([pantsbuild#14139](pantsbuild#14139))

* [internal] Bump reqwest from 0.11.4 to 0.11.9 in /src/rust/engine ([pantsbuild#14135](pantsbuild#14135))

* [internal] Further tweak dependabot config ([pantsbuild#14132](pantsbuild#14132))

* [internal] python: use immutable_input_digests for protobuf codegen ([pantsbuild#13885](pantsbuild#13885))

* [internal] python: use immutable_input_digests for protobuf codegen ([pantsbuild#13885](pantsbuild#13885))

* [internal] No need for two f-strings/two strings. ([pantsbuild#14119](pantsbuild#14119))

* `LockfileMetadata` - Replace flaky `_header_dict()` mechanism with more robust `header_attrs` mechanism ([pantsbuild#14229](pantsbuild#14229))
illicitonion added a commit that referenced this pull request Jan 22, 2022
@illicitonion
Prep for 2.10.0.dev2 (#14236)
4113608 
Internal changes:

* upgrade to Rust v1.58.0 ([#14174](#14174))

* [internal] fix typos in codegen registration ([#14172](#14172))

* Pulls `Coordinate` and `ArtifactRequirements` out into a separate file to avoid a circuilar import later ([#14164](#14164))

* Factors lockfile metadata code into class that supports multiple languages as well as versions ([#14116](#14116))

* Adds `.env` file to make vscode auto-imports work correctly out of the box ([#14130](#14130))

* [internal] Rename classes for `generate_lockfiles.py` for clarity ([#14218](#14218))

* [internal] Fix bad Find+Replace ([#14213](#14213))

* [internal] Bump libc from 0.2.106 to 0.2.112 in /src/rust/engine ([#14206](#14206))

* [internal] Bump tempfile from 3.2.0 to 3.3.0 in /src/rust/engine ([#14202](#14202))

* [internal] Bump criterion from 0.3.3 to 0.3.5 in /src/rust/engine ([#14205](#14205))

* [internal] Bump walkdir from 2.3.1 to 2.3.2 in /src/rust/engine ([#14204](#14204))

* [internal] Bump generic-array from 0.14.4 to 0.14.5 in /src/rust/engine ([#14203](#14203))

* [internal] Remove the minimum bucket size of batching to improve stability. ([#14210](#14210))

* [internal] Make `JvmLockfileRequest` generic ([#14201](#14201))

* [internal] add comment re clippy issue ([#14188](#14188))

* [internal] rename some codegen scopes to put language first ([#14187](#14187))

* [internal] Check for ambiguity when running `generate-lockfiles` ([#14178](#14178))

* [internal] Change JVM lockfile format ([#14175](#14175))

* [internal] go: rewrite third party package analysis to not use `go list` ([#14157](#14157))

* [internal] Hotfix `fmt` crashing on non-formattable targets ([#14168](#14168))

* [internal] Simplify `core/goals/package.py` filesystem API usage ([#14162](#14162))

* [internal] java/thrift: register union that was not registered ([#14159](#14159))

* [internal] Bump arc-swap from 1.3.0 to 1.5.0 in /src/rust/engine ([#14148](#14148))

* [internals] Bump tokio-rustls from 0.22.0 to 0.23.2 in /src/rust/engine ([#14149](#14149))

* [internal] Bump num_cpus from 1.13.0 to 1.13.1 in /src/rust/engine ([#14150](#14150))

* [internal] Bump tokio-util from 0.6.7 to 0.6.9 in /src/rust/engine ([#14151](#14151))

* [internal] Bump os_pipe from 0.9.2 to 1.0.0 in /src/rust/engine ([#14152](#14152))

* [internal] Introduce new `BuiltinGoal` subsystem type. ([#13991](#13991))

* [internal] Upgrade python type stubs packages ([#14143](#14143))

* [internal] Bump strum_macros from 0.20.1 to 0.23.1 in /src/rust/engine ([#14141](#14141))

* [internal] Make `generate-lockfiles` goal generic ([#14122](#14122))

* [internal] Bump errno from 0.2.7 to 0.2.8 in /src/rust/engine ([#14137](#14137))

* [internal] Bump colored from 1.9.3 to 2.0.0 in /src/rust/engine ([#14138](#14138))

* [internal] Bump crossbeam-channel from 0.4.4 to 0.5.0 in /src/rust/engine ([#14139](#14139))

* [internal] Bump reqwest from 0.11.4 to 0.11.9 in /src/rust/engine ([#14135](#14135))

* [internal] Further tweak dependabot config ([#14132](#14132))

* [internal] python: use immutable_input_digests for protobuf codegen ([#13885](#13885))

* [internal] python: use immutable_input_digests for protobuf codegen ([#13885](#13885))

* [internal] No need for two f-strings/two strings. ([#14119](#14119))

* `LockfileMetadata` - Replace flaky `_header_dict()` mechanism with more robust `header_attrs` mechanism ([#14229](#14229))
@tdyas tdyas restored the python_protobuf_use_immutable_inputs branch July 21, 2022 23:13
@tdyas tdyas deleted the python_protobuf_use_immutable_inputs branch February 20, 2023 20:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@stuhood stuhood stuhood approved these changes

@Eric-Arellano Eric-Arellano Eric-Arellano approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@tdyas @Eric-Arellano @stuhood