Skip to content

add event exporter helper#46651

Merged
fspmarshall merged 1 commit intomasterfrom
fspmarshall/bulk-event-export-helper
Sep 25, 2024
Merged

add event exporter helper#46651
fspmarshall merged 1 commit intomasterfrom
fspmarshall/bulk-event-export-helper

Conversation

@fspmarshall
Copy link
Copy Markdown
Contributor

Implements an exporter helper type intended to make it easier to work with the new bulk event export API introduced in #46399. Fundamentally, this type is basically just doing a fancy version of:

for chunk := range getChunks(date) {
    for event := range getEvents(chunk) {
        export(event)
    }
}

Special care has been taken to ensure that per-event processing and cursor state tracking is simple and performant. Logging has also had custom rate limits applied to help mitigate issues we've seen in testing where event export logs can easily become overwhelming due to the high throughput in large clusters.

This is part of an ongoing body of work related to #46193

@fspmarshall fspmarshall added backport/branch/v14 no-changelog Indicates that a PR does not require a changelog entry labels Sep 16, 2024
@github-actions github-actions Bot requested review from eriktate and jakule September 16, 2024 22:13
@github-actions github-actions Bot added audit-log Issues related to Teleports Audit Log size/lg labels Sep 16, 2024
@fspmarshall fspmarshall force-pushed the fspmarshall/bulk-event-export-api branch from 9d34d52 to 330fdb5 Compare September 17, 2024 15:25
@fspmarshall fspmarshall force-pushed the fspmarshall/bulk-event-export-helper branch from 2a860ae to b5d684c Compare September 17, 2024 16:10
@fspmarshall fspmarshall force-pushed the fspmarshall/bulk-event-export-api branch 3 times, most recently from 6449ac8 to ada64d4 Compare September 19, 2024 19:15
@fspmarshall fspmarshall force-pushed the fspmarshall/bulk-event-export-helper branch from b5d684c to a84ad6c Compare September 19, 2024 19:16
Base automatically changed from fspmarshall/bulk-event-export-api to master September 19, 2024 19:49
@fspmarshall fspmarshall force-pushed the fspmarshall/bulk-event-export-helper branch from a84ad6c to 8c9924c Compare September 19, 2024 20:16
Comment thread lib/events/export/export.go Outdated
Comment thread lib/events/export/export_test.go Outdated
Comment thread lib/events/export/export.go
Comment thread lib/events/export/export.go
Comment thread lib/events/export/export.go Outdated
Comment thread lib/events/export/export.go Outdated
Comment thread lib/events/export/export_test.go
@fspmarshall fspmarshall force-pushed the fspmarshall/bulk-event-export-helper branch from 8c9924c to c1e76e2 Compare September 19, 2024 21:00
Comment thread lib/events/export/date_exporter.go
@fspmarshall fspmarshall force-pushed the fspmarshall/bulk-event-export-helper branch from c1e76e2 to b15d673 Compare September 19, 2024 21:14
Comment thread lib/events/export/date_exporter.go Outdated
@fspmarshall fspmarshall force-pushed the fspmarshall/bulk-event-export-helper branch from b15d673 to b12ede4 Compare September 25, 2024 20:10
@fspmarshall fspmarshall added this pull request to the merge queue Sep 25, 2024
Merged via the queue into master with commit 00c154b Sep 25, 2024
@fspmarshall fspmarshall deleted the fspmarshall/bulk-event-export-helper branch September 25, 2024 21:15
@public-teleport-github-review-bot
Copy link
Copy Markdown

@fspmarshall See the table below for backport results.

Branch Result
branch/v14 Failed
branch/v15 Failed
branch/v16 Failed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

audit-log Issues related to Teleports Audit Log no-changelog Indicates that a PR does not require a changelog entry size/lg

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants