Refactor runtimeType usage in codegen #1933

Velfi · 2022-11-01T21:15:54Z

Description

This primarily does three things:

(huge change) unify the instantiation of runtime types during code generation
(small change) update AWS codegen to support customizing specific sections. This works just like the existing customization hook for operations and configs.
(small change) fix some gradle stuff that had to do with trying to use delegation from within a plugin block + issues with the cargo commands

Info for smithy contribs:

awsRuntimeDependency has been renamed to awsRuntimeCrate in order to match smithyRuntimeCrate. Additionally, declaration of the runtime crates has been centralized in software.amazon.smithy.rustsdk.AwsRuntimeDependency for AWS types and software.amazon.smithy.rust.codegen.core.rustlang.CargoDependency for smithy types.
The RuntimeConfig runtime crate functions now return RuntimeTypes instead of CargoDependencies. This better matches their usage in codegen. If you want to access the cargo dependency, use the awsRuntimeCrate/smithyRuntimeCrate functions directly.

before: rust("#T", CargoDependency.SmithyClient(runtimeConfig).asType())
after: rust("#T", runtimeConfig.smithyClient())
Service-specific decorators in SDK codegen have been moved into software.amazon.smithy.rustsdk.servicedecorators. All other decorators have been moved to software.amazon.smithy.rustsdk.decorators.
-RustCodegenDecorator's name field has been removed. If you need to log the name of a decorator, use Kotlin's builtin class field:
```
val someDecorator = SomeDecorator()
print(someDecorator::class)
// prints "class SomeDecorator"
```

The class AwsCodegenDecorator.kt has been renamed to AwsCombinedCodegenDecorator.kt. A new class, now named AwsCodegenDecorator is an AWS-specific subclass of RustCodegenDecorator. AwsCodegenDecorator supports the customization of the FromSdkConfigForBuilder section.

For example, adding this customization to codegen as part of a decorator:

class SleepImplFromSdkConfig : AwsCustomization() {
    override fun section(section: AwsSection): Writable = writable {
        when (section) {
            is AwsSection.FromSdkConfigForBuilder -> rust("builder.set_sleep_impl(input.sleep_impl());")
        }
    }
}

would result in this Rust code being generated:

// In aws-sdk-<service>/src/config.rs
impl From<&SdkConfig> for Builder {
    fn from(input: &#{SdkConfig}) -> Self {
        let mut builder = Builder::default();
        builder.set_sleep_impl(input.sleep_impl());
        builder
    }
}

CombinedCodegenDecorator's orderedDecorators field is now protected instead of private so that the new AwsCodegenDecorator subclass can use it.
runtimeCrate has been renamed to smithyRuntimeCrate and now requires users to pass the "smithy-" prefix as part of the crate name:

before: val smithyHttp = runtimeCrate("http").asType()
after: val smithyHttp = smithyRuntimeCrate("smithy-http").asType()

Testing

ran existing tests

Checklist

I have updated CHANGELOG.next.toml if I made changes to the smithy-rs codegen or runtime crates
I have updated CHANGELOG.next.toml if I made changes to the AWS SDK, generated SDK code, or SDK runtime crates

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

add: AwsCustomization NamedSectionGenerator add: AwsSection for `From<SdkConfig> for service::config::Builder` codegen rename: AwsCodegenDecorator to AwsCombinedCodegenDecorator add: new interface AwsCodegenDecorator that respects AwsCustomizations

…mizations folder

fix: tests broken by moves

format: unify casing of runtime type function names format: unify casing of CargoDependency function names rename: args of FluentClientGenerics.sendBounds method to be more explicit update: make orderedDecorators field of CombinedCodegenDecorator accessible to inheritors fix: spelling issues highlighted by IntelliJ update: make protected fields private where possible

add: missing imports to test

Velfi · 2022-11-01T21:58:01Z

~~Keeping this in draft until I can get around to writing the changelog~~

add: another missing import

github-actions · 2022-11-02T15:51:30Z

A new generated diff is ready to view.

AWS SDK (ignoring whitespace)
No codegen difference in the Server Test
No codegen difference in the Server Test Python

A new doc preview is ready to view.

add: missing copyright headers fix: endpoints test by differentiating between usages of aws-endpoints

github-actions · 2022-11-03T17:50:50Z

A new generated diff is ready to view.

AWS SDK (ignoring whitespace)
No codegen difference in the Server Test
No codegen difference in the Server Test Python

A new doc preview is ready to view.

update: rename servicecustomizations to servicedecorators remove: decorator name field add: CHANGELOG.next.toml entries

github-actions · 2022-11-04T16:34:57Z

A new generated diff is ready to view.

AWS SDK (ignoring whitespace)
No codegen difference in the Server Test
No codegen difference in the Server Test Python

A new doc preview is ready to view.

Info for smithy contribs: - `awsRuntimeDependency` has been renamed to `awsRuntimeCrate` in order to match `smithyRuntimeCrate`. Additionally, declaration of the runtime crates has been centralized in `software.amazon.smithy.rustsdk.AwsRuntimeDependency` for AWS types and `software.amazon.smithy.rust.codegen.core.rustlang.CargoDependency` for smithy types. - The `RuntimeConfig` runtime crate functions now return `RuntimeTypes` instead of `CargoDependencies`. This better matches their usage in codegen. If you want to access the cargo dependency, use the `awsRuntimeCrate`/`smithyRuntimeCrate` functions directly. before: `rust("#T", CargoDependency.SmithyClient(runtimeConfig).asType())` after: `rust("#T", runtimeConfig.smithyClient())` - Service-specific decorators in SDK codegen have been moved into `software.amazon.smithy.rustsdk.servicedecorators`. All other decorators have been moved to `software.amazon.smithy.rustsdk.decorators`. -`RustCodegenDecorator`'s name field has been removed. If you need to log the name of a decorator, use Kotlin's builtin `class` field: ```kotlin val someDecorator = SomeDecorator() print(someDecorator::class) // prints "class SomeDecorator" ``` - The class `AwsCodegenDecorator.kt` has been renamed to `AwsCombinedCodegenDecorator.kt`. A new class, now named `AwsCodegenDecorator` is an AWS-specific subclass of `RustCodegenDecorator`. `AwsCodegenDecorator` supports the customization of the `FromSdkConfigForBuilder` section. For example, adding this customization to codegen as part of a decorator: ```kotlin class SleepImplFromSdkConfig : AwsCustomization() { override fun section(section: AwsSection): Writable = writable { when (section) { is AwsSection.FromSdkConfigForBuilder -> rust("builder.set_sleep_impl(input.sleep_impl());") } } } ``` would result in this Rust code being generated: ```rust // In aws-sdk-<service>/src/config.rs impl From<&SdkConfig> for Builder { fn from(input: &#{SdkConfig}) -> Self { let mut builder = Builder::default(); builder.set_sleep_impl(input.sleep_impl()); builder } } ``` - `CombinedCodegenDecorator`'s `orderedDecorators` field is now protected instead of private so that the new `AwsCodegenDecorator` subclass can use it. - `runtimeCrate` has been renamed to `smithyRuntimeCrate` and now requires users to pass the "smithy-" prefix as part of the crate name: before: `val smithyHttp = runtimeCrate("http").asType()` after: `val smithyHttp = smithyRuntimeCrate("smithy-http").asType()`

Velfi · 2022-11-04T18:55:30Z

[this comment was moved into the PR description]

github-actions · 2022-11-04T19:42:26Z

A new generated diff is ready to view.

AWS SDK (ignoring whitespace)
No codegen difference in the Server Test
No codegen difference in the Server Test Python

A new doc preview is ready to view.

jdisanti

This is awesome! Lots of nice changes in here, and I love the push for more consistency on the runtime types.

jdisanti · 2022-11-04T22:33:46Z

...gen/src/main/kotlin/software/amazon/smithy/rustsdk/decorators/AwsCombinedCodegenDecorator.kt

+    }
+}
+
+fun generateImplFromRefSdkConfigForConfigBuilder(


Shouldn't this be in SdkConfigDecorator?

I think no because I feel that all the section writing code should be defined in one place. The combined decorator is also the place where we have access to the full list of decorators that modify an AwsSection.

Organizationally, this isn't going to work out well since people will think to look in SdkConfigDecorator for this code rather than in AwsCombinedCodegenDecorator. The full list of decorators can be passed into the customizations if it's required (and I think we do that in codegen-core/codegen-client).

Russell is adding the ability to do this as part of #1953, I can take the change from that or wait for the merge.

...gen/src/main/kotlin/software/amazon/smithy/rustsdk/decorators/AwsCombinedCodegenDecorator.kt

...en/src/main/kotlin/software/amazon/smithy/rustsdk/decorators/HttpRequestChecksumDecorator.kt

...odegen/src/main/kotlin/software/amazon/smithy/rustsdk/decorators/IntegrationTestDecorator.kt

aws/sdk-codegen/src/main/kotlin/software/amazon/smithy/rustsdk/decorators/UserAgentDecorator.kt

...n/kotlin/software/amazon/smithy/rust/codegen/client/smithy/customize/RustCodegenDecorator.kt

...en-core/src/main/kotlin/software/amazon/smithy/rust/codegen/core/rustlang/CargoDependency.kt

settings.gradle.kts

tools/crate-hasher/src/file_list.rs

remove: need to specify `CargoDependency` refactor: get runtime types from `RuntimeType` instead of `RuntimeConfig`

github-actions · 2022-11-09T17:33:52Z

A new generated diff is ready to view.

A new doc preview is ready to view.

update: RuntimeType usage

format: run `pre-commit run --all-files`

github-actions · 2022-11-09T23:41:19Z

A new generated diff is ready to view.

A new doc preview is ready to view.

…n with other conversion methods

github-actions · 2022-11-10T16:20:39Z

A new generated diff is ready to view.

A new doc preview is ready to view.

rcoh · 2022-11-10T18:56:14Z

aws/sdk-codegen/src/main/kotlin/software/amazon/smithy/rustsdk/traits/PresignableTrait.kt

@@ -11,8 +11,8 @@ import software.amazon.smithy.model.traits.AnnotationTrait

 /** Synthetic trait that indicates an operation is presignable. */
 // TODO(https://github.com/awslabs/smithy/pull/897) this can be replaced when the trait is added to Smithy.
-class PresignableTrait(val syntheticOperationId: ShapeId) : AnnotationTrait(ID, Node.objectNode()) {


I guess this was never used?

rcoh · 2022-11-10T19:35:30Z

...en-core/src/main/kotlin/software/amazon/smithy/rust/codegen/core/smithy/protocols/AwsJson.kt

-        "HeaderMap" to RuntimeType.http.member("HeaderMap"),
-        "JsonError" to CargoDependency.smithyJson(runtimeConfig).asType().member("deserialize::Error"),
-        "Response" to RuntimeType.http.member("Response"),
+        "Bytes" to Bytes.toType().resolve("Bytes"),


personally, I do actually appreciate the prefix so I know this is a rust library but not a super strong preference

rcoh · 2022-11-10T19:36:37Z

...n-core/src/main/kotlin/software/amazon/smithy/rust/codegen/core/smithy/protocols/Ec2Query.kt

-        "HeaderMap" to RuntimeType.http.member("HeaderMap"),
-        "Response" to RuntimeType.http.member("Response"),
-        "XmlError" to CargoDependency.smithyXml(runtimeConfig).asType().member("decode::XmlError"),
+        "Bytes" to Bytes.toType().resolve("Bytes"),


not sure if replacing the single-source-of-truth Bytes with a resolved package everywhere is useful? Maybe a CommonTypes module?

rcoh · 2022-11-10T19:48:39Z

can you move the "changelog" comment into the PR description so it's easier to find?

github-actions · 2022-11-10T23:20:45Z

A new generated diff is ready to view.

A new doc preview is ready to view.

Velfi · 2022-11-16T18:30:02Z

I'm going to close this and re-open separate PRs, each with a subset of the changes.

Velfi added 4 commits November 1, 2022 16:01

move: all AWS-service-specific customizations into a new servicecusto…

8b668ad

…mizations folder

fix: various small lints and compile warnings

813e388

fix: tests broken by moves

Velfi requested review from a team as code owners November 1, 2022 21:15

Velfi added 2 commits November 1, 2022 16:52

fix: build.gradle.kts improper delegation issue

f652e30

add: missing imports to test

Merge branch 'main' into refactor/unify-runtime-crate-usage-in-codegen

a47ab41

Velfi marked this pull request as draft November 1, 2022 21:57

Velfi added 4 commits November 1, 2022 17:10

fix: set default RuntimeConfig cratePrefix to "aws"

9979d76

add: another missing import

add: another missing import

9a50129

update: sha256 dep of crate-hasher tool

b8678ff

update: Cargo.lock of crate-hasher tool

263d9c6

Velfi mentioned this pull request Nov 2, 2022

Feature: configurable connectors for AWS clients #1918

Merged

1 task

update: make doc-only usage of awsConfig import it as a dev dependency

52a55fd

add: missing copyright headers fix: endpoints test by differentiating between usages of aws-endpoints

Velfi added 2 commits November 4, 2022 10:47

update: move SleepImplFromSdkConfig into ResiliencyDecorator

ced89c5

update: rename servicecustomizations to servicedecorators remove: decorator name field add: CHANGELOG.next.toml entries

Merge branch 'main' into refactor/unify-runtime-crate-usage-in-codegen

40759ad

Velfi marked this pull request as ready for review November 4, 2022 15:50

jdisanti reviewed Nov 4, 2022

View reviewed changes

Velfi added 3 commits November 8, 2022 15:05

undo: decorator ordering change

84acfd2

remove: need to specify `CargoDependency` refactor: get runtime types from `RuntimeType` instead of `RuntimeConfig`

remove: code that didn't do anything

840d13a

Merge branch 'main' into refactor/unify-runtime-crate-usage-in-codegen

3e8a0a4

Velfi added the needs-sdk-review label Nov 8, 2022

Velfi added the needs-server-review label Nov 8, 2022

fix: server module missing import

8d48f5c

Velfi added 4 commits November 9, 2022 14:09

refactor: simplify RuntimeType

f651183

update: RuntimeType usage

Merge branch 'main' into refactor/unify-runtime-crate-usage-in-codegen

6daebef

Merge branch 'main' into refactor/unify-runtime-crate-usage-in-codegen

ab09d99

fix: outdated RuntimeType usage

1c95fb7

format: run `pre-commit run --all-files`

rename: RuntimeType.asType() to RuntimeType.toType() in conventio…

7aae03d

…n with other conversion methods

update: derive attribute type sorting

2c0ed8b

rcoh approved these changes Nov 10, 2022

View reviewed changes

Merge branch 'main' into refactor/unify-runtime-crate-usage-in-codegen

558ddc8

Velfi removed the needs-sdk-review label Nov 11, 2022

Velfi closed this Nov 16, 2022

Velfi mentioned this pull request Nov 16, 2022

update: CargoDependency companion fn names for smithy runtime crates #1996

Merged

jjant deleted the refactor/unify-runtime-crate-usage-in-codegen branch November 18, 2022 09:51

Velfi removed the needs-server-review label Dec 2, 2022

Velfi mentioned this pull request Dec 6, 2022

refactor: RuntimeType creation and member resolution #2066

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Refactor runtimeType usage in codegen #1933

Refactor runtimeType usage in codegen #1933

Velfi commented Nov 1, 2022 •

edited

Loading

Velfi commented Nov 1, 2022 •

edited

Loading

github-actions bot commented Nov 2, 2022

github-actions bot commented Nov 3, 2022

github-actions bot commented Nov 4, 2022

Velfi commented Nov 4, 2022 •

edited

Loading

github-actions bot commented Nov 4, 2022

jdisanti left a comment

jdisanti Nov 4, 2022

Velfi Nov 8, 2022

jdisanti Nov 9, 2022 •

edited

Loading

Velfi Nov 10, 2022

github-actions bot commented Nov 9, 2022

github-actions bot commented Nov 9, 2022

github-actions bot commented Nov 10, 2022

rcoh Nov 10, 2022

rcoh Nov 10, 2022

rcoh Nov 10, 2022

rcoh commented Nov 10, 2022

github-actions bot commented Nov 10, 2022

Velfi commented Nov 16, 2022

Refactor runtimeType usage in codegen #1933

Refactor runtimeType usage in codegen #1933

Conversation

Velfi commented Nov 1, 2022 • edited Loading

Description

Testing

Checklist

Velfi commented Nov 1, 2022 • edited Loading

github-actions bot commented Nov 2, 2022

github-actions bot commented Nov 3, 2022

github-actions bot commented Nov 4, 2022

Velfi commented Nov 4, 2022 • edited Loading

github-actions bot commented Nov 4, 2022

jdisanti left a comment

Choose a reason for hiding this comment

jdisanti Nov 4, 2022

Choose a reason for hiding this comment

Velfi Nov 8, 2022

Choose a reason for hiding this comment

jdisanti Nov 9, 2022 • edited Loading

Choose a reason for hiding this comment

Velfi Nov 10, 2022

Choose a reason for hiding this comment

github-actions bot commented Nov 9, 2022

github-actions bot commented Nov 9, 2022

github-actions bot commented Nov 10, 2022

rcoh Nov 10, 2022

Choose a reason for hiding this comment

rcoh Nov 10, 2022

Choose a reason for hiding this comment

rcoh Nov 10, 2022

Choose a reason for hiding this comment

rcoh commented Nov 10, 2022

github-actions bot commented Nov 10, 2022

Velfi commented Nov 16, 2022

Velfi commented Nov 1, 2022 •

edited

Loading

Velfi commented Nov 1, 2022 •

edited

Loading

Velfi commented Nov 4, 2022 •

edited

Loading

jdisanti Nov 9, 2022 •

edited

Loading