Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

(aws-s3): server access logging support automatic date-based partitioning #28141

Closed
2 tasks
mrgrain opened this issue Nov 27, 2023 · 4 comments · Fixed by #28790
Closed
2 tasks

(aws-s3): server access logging support automatic date-based partitioning #28141

mrgrain opened this issue Nov 27, 2023 · 4 comments · Fixed by #28790
Labels
@aws-cdk/aws-s3 Related to Amazon S3 effort/medium Medium work item – several days of effort feature-request A feature should be added or improved. p1

Comments

@mrgrain
Copy link
Contributor

mrgrain commented Nov 27, 2023

Describe the feature

S3 Bucket Logging Configuration should support automatic date-based partitioning for delivered logs.

Use Case

See: https://aws.amazon.com/about-aws/whats-new/2023/11/amazon-s3-server-access-logging-date-partitioning/

Proposed Solution

Probably need to introduce a new property and/or enum class to allow setting the TargetObjectKeyFormat

Other Information

https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-s3-bucket-partitionedprefix.html
https://docs.aws.amazon.com/AmazonS3/latest/userguide/ServerLogs.html

Acknowledgements

  • I may be able to implement this feature request
  • This feature might incur a breaking change

CDK version used

any

Environment details (OS name and version, etc.)

any
@mrgrain mrgrain added feature-request A feature should be added or improved. needs-triage This issue or PR still needs to be triaged. labels Nov 27, 2023
@github-actions github-actions bot added the @aws-cdk/aws-s3 Related to Amazon S3 label Nov 27, 2023
@msambol
Copy link
Contributor

msambol commented Nov 27, 2023

I'll take this.

@msambol
Copy link
Contributor

msambol commented Nov 27, 2023

@mrgrain This depends on an update to s3.generated.ts. I think I should be using spec2cdk to update that? It won't pick up the new property.

❯ ./bin/spec2cdk ../../../packages/aws-cdk-lib/aws-s3/lib/ -s AWS::S3
Generating 1 modules...
Summary:
  Service files:  1
  Resources:      7
🐢  Completed in: 54.442ms

@mrgrain
Copy link
Contributor Author

mrgrain commented Nov 27, 2023 •
edited
Loading

I'm guessing we'll need to wait for the next AWS Service Spec Update PR to come through. That happens every Monday.

Alternatively you can update @aws-cdk/aws-service-spec locally to start development.

If that doesn't work either, than the CFN schema might not be released yet.

@pahud pahud added p1 effort/medium Medium work item – several days of effort and removed needs-triage This issue or PR still needs to be triaged. labels Nov 27, 2023
@github-actions github-actions bot mentioned this issue Dec 1, 2023
Closed
@sakurai-ryo sakurai-ryo mentioned this issue Jan 20, 2024
Merged
@mergify mergify bot closed this as completed in #28790 Jan 25, 2024
mergify bot pushed a commit that referenced this issue Jan 25, 2024
@sakurai-ryo
feat(s3): date-based partitioning for log objects (#28790)
2952408 
S3 server access logging newly supports date-based partitioning.
This feature allows key formats of log objects to be partitioned by date, which is useful when querying from Athena.

In CloudFormation, `TargetObjectKeyFormat` was added.
This object has the `PartitionedPrefix` and the `SimplePrefix` property, only one of which is allowed.
The `SimplePrefix` is the key format that originally existed.
The `PartitionedPrefix` is the new key format and allows the user to select either the event time or delivery time as the date source.

```yaml
Resources:
  S3BucketServerAccessLogEnabledTest:
    Type: AWS::S3::Bucket
    Properties:
      BucketName: access-log-test
      LoggingConfiguration:
        DestinationBucketName: myDestinationBucket
        LogFilePrefix: 'hello'
        TargetObjectKeyFormat:
          # You can deploy in either of the following formats
          SimplePrefix: {} # 1
          PartitionedPrefix: {} # 2
          PartitionedPrefix: # 3
            PartitionDateSource: EventTime # | DeliveryTime
```

whats-new
https://aws.amazon.com/about-aws/whats-new/2023/11/amazon-s3-server-access-logging-date-partitioning

CloudFormation User Guides
https://docs.aws.amazon.com/ja_jp/AWSCloudFormation/latest/UserGuide/aws-properties-s3-bucket-loggingconfiguration.html#cfn-s3-bucket-loggingconfiguration-targetobjectkeyformat

S3 User Guides
https://docs.aws.amazon.com/AmazonS3/latest/userguide/ServerLogs.html#server-access-logging-overview

Closes #28141

----

*By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
@github-actions GitHub Actions
Copy link

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.

Vandita2020 pushed a commit to Vandita2020/aws-cdk that referenced this issue Jan 30, 2024
@sakurai-ryo @Vandita2020
feat(s3): date-based partitioning for log objects (aws#28790)
f101d12 
S3 server access logging newly supports date-based partitioning.
This feature allows key formats of log objects to be partitioned by date, which is useful when querying from Athena.

In CloudFormation, `TargetObjectKeyFormat` was added.
This object has the `PartitionedPrefix` and the `SimplePrefix` property, only one of which is allowed.
The `SimplePrefix` is the key format that originally existed.
The `PartitionedPrefix` is the new key format and allows the user to select either the event time or delivery time as the date source.

```yaml
Resources:
  S3BucketServerAccessLogEnabledTest:
    Type: AWS::S3::Bucket
    Properties:
      BucketName: access-log-test
      LoggingConfiguration:
        DestinationBucketName: myDestinationBucket
        LogFilePrefix: 'hello'
        TargetObjectKeyFormat:
          # You can deploy in either of the following formats
          SimplePrefix: {} # 1
          PartitionedPrefix: {} # 2
          PartitionedPrefix: # 3
            PartitionDateSource: EventTime # | DeliveryTime
```

whats-new
https://aws.amazon.com/about-aws/whats-new/2023/11/amazon-s3-server-access-logging-date-partitioning

CloudFormation User Guides
https://docs.aws.amazon.com/ja_jp/AWSCloudFormation/latest/UserGuide/aws-properties-s3-bucket-loggingconfiguration.html#cfn-s3-bucket-loggingconfiguration-targetobjectkeyformat

S3 User Guides
https://docs.aws.amazon.com/AmazonS3/latest/userguide/ServerLogs.html#server-access-logging-overview

Closes aws#28141

----

*By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
SankyRed pushed a commit that referenced this issue Feb 8, 2024
@sakurai-ryo @SankyRed
feat(s3): date-based partitioning for log objects (#28790)
cc09ccf 
S3 server access logging newly supports date-based partitioning.
This feature allows key formats of log objects to be partitioned by date, which is useful when querying from Athena.

In CloudFormation, `TargetObjectKeyFormat` was added.
This object has the `PartitionedPrefix` and the `SimplePrefix` property, only one of which is allowed.
The `SimplePrefix` is the key format that originally existed.
The `PartitionedPrefix` is the new key format and allows the user to select either the event time or delivery time as the date source.

```yaml
Resources:
  S3BucketServerAccessLogEnabledTest:
    Type: AWS::S3::Bucket
    Properties:
      BucketName: access-log-test
      LoggingConfiguration:
        DestinationBucketName: myDestinationBucket
        LogFilePrefix: 'hello'
        TargetObjectKeyFormat:
          # You can deploy in either of the following formats
          SimplePrefix: {} # 1
          PartitionedPrefix: {} # 2
          PartitionedPrefix: # 3
            PartitionDateSource: EventTime # | DeliveryTime
```

whats-new
https://aws.amazon.com/about-aws/whats-new/2023/11/amazon-s3-server-access-logging-date-partitioning

CloudFormation User Guides
https://docs.aws.amazon.com/ja_jp/AWSCloudFormation/latest/UserGuide/aws-properties-s3-bucket-loggingconfiguration.html#cfn-s3-bucket-loggingconfiguration-targetobjectkeyformat

S3 User Guides
https://docs.aws.amazon.com/AmazonS3/latest/userguide/ServerLogs.html#server-access-logging-overview

Closes #28141

----

*By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
@aws-cdk/aws-s3 Related to Amazon S3 effort/medium Medium work item – several days of effort feature-request A feature should be added or improved. p1
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

feat(s3): date-based partitioning for log objects sakurai-ryo/aws-cdk
3 participants
@pahud @mrgrain @msambol