proto/accessgraph: Add RPC for sending k8s audit logs to Access Graph#59566
Merged
proto/accessgraph: Add RPC for sending k8s audit logs to Access Graph#59566
Conversation
camscale
commented
Sep 25, 2025
tigrato
reviewed
Sep 30, 2025
| // effective configuration. | ||
| // 3. Server sends second response providing the starting `resume_state` (possilby empty). | ||
| // 4. Client sends subsequent `KubeAuditLogStreamRequest` messages. | ||
| rpc KubeAuditLogStream(stream KubeAuditLogStreamRequest) returns (stream KubeAuditLogStreamResponse); |
Contributor
There was a problem hiding this comment.
A client that wants to push logs for multi clusters should create one or the same stream can be used for multiple kubernetes logs?
Contributor
Author
There was a problem hiding this comment.
One stream for multiple clusters is the intended use, but I haven't really considered scalability. The server side loops over the stream in a single goroutine currently which might be a bottleneck if many clusters are providing their log over the same stream. However that's a detail not necessarily relevant to this grpc spec.
I'll add another point to the doc comment:
// - Logs for multiple clusters can be streamed across the one stream, but
// they must be all for the same KubeAuditLogSource. Multiple streams may
// be used for the same log source if needed for scalability.
Contributor
Author
There was a problem hiding this comment.
I have revised this further after adding dynamic behaviour to the fetcher (detecting clusters coming and going and handling discovery_config resources) - you can still stream multiple log streams over the one grpc stream, but each stream is explicitly started so that the resumption cursor is sent when the log stream starts and not when the grpc connection is created.
camscale
force-pushed
the
camh/tag/kube-audit-log-grpc
branch
from
98ad7de to
1891ce0
Compare
Contributor
Author
|
reviewers - this is now ready for review, with a revision after adding dynamic behaviour to the code. |
camscale
added
the
no-changelog
juliaogris
approved these changes
Oct 15, 2025
tigrato
approved these changes
Oct 15, 2025
public-teleport-github-review-bot
bot
removed request for
r0mant and
smallinsky
Add a `KubeAuditLogsStream()` rpc to the `AccessGraphService` for streaming Kubernetes apiserver audit logs from the Teleport discovery service to access graph. This is intended for EKS audit logs which are made available via CloudWatch, but can accommodate other k8s services. The audit log messages are represented as a `google.protobuf.Struct` so as to not depend on the k8s.io .proto files, but also as k8s typically uses protos internally only - the expectation is that we'll receive the apiserver audit logs as json-encoded strings. This encode easily as a `google.protobuf.Struct`.
Generate proto and grpc code for changes to accessgraph/v1alpha1:
make grpc/host
These changes add the `KubeAuditLogsStream()` rpc and associated types.
camscale
force-pushed
the
camh/tag/kube-audit-log-grpc
branch
from
1891ce0 to
5a16dbd
Compare
github-merge-queue
bot
removed this pull request from the merge queue due to failed status checks
mmcallister
pushed a commit
that referenced
this pull request
Nov 6, 2025
…#59566) * proto/accessgraph: Add RPC for sending k8s audit logs to Access Graph Add a `KubeAuditLogsStream()` rpc to the `AccessGraphService` for streaming Kubernetes apiserver audit logs from the Teleport discovery service to access graph. This is intended for EKS audit logs which are made available via CloudWatch, but can accommodate other k8s services. The audit log messages are represented as a `google.protobuf.Struct` so as to not depend on the k8s.io .proto files, but also as k8s typically uses protos internally only - the expectation is that we'll receive the apiserver audit logs as json-encoded strings. This encode easily as a `google.protobuf.Struct`. * proto: Generate protos for accessgraph Generate proto and grpc code for changes to accessgraph/v1alpha1: make grpc/host These changes add the `KubeAuditLogsStream()` rpc and associated types.
mmcallister
pushed a commit
that referenced
this pull request
Nov 19, 2025
…#59566) * proto/accessgraph: Add RPC for sending k8s audit logs to Access Graph Add a `KubeAuditLogsStream()` rpc to the `AccessGraphService` for streaming Kubernetes apiserver audit logs from the Teleport discovery service to access graph. This is intended for EKS audit logs which are made available via CloudWatch, but can accommodate other k8s services. The audit log messages are represented as a `google.protobuf.Struct` so as to not depend on the k8s.io .proto files, but also as k8s typically uses protos internally only - the expectation is that we'll receive the apiserver audit logs as json-encoded strings. This encode easily as a `google.protobuf.Struct`. * proto: Generate protos for accessgraph Generate proto and grpc code for changes to accessgraph/v1alpha1: make grpc/host These changes add the `KubeAuditLogsStream()` rpc and associated types.
mmcallister
pushed a commit
that referenced
this pull request
Nov 20, 2025
…#59566) * proto/accessgraph: Add RPC for sending k8s audit logs to Access Graph Add a `KubeAuditLogsStream()` rpc to the `AccessGraphService` for streaming Kubernetes apiserver audit logs from the Teleport discovery service to access graph. This is intended for EKS audit logs which are made available via CloudWatch, but can accommodate other k8s services. The audit log messages are represented as a `google.protobuf.Struct` so as to not depend on the k8s.io .proto files, but also as k8s typically uses protos internally only - the expectation is that we'll receive the apiserver audit logs as json-encoded strings. This encode easily as a `google.protobuf.Struct`. * proto: Generate protos for accessgraph Generate proto and grpc code for changes to accessgraph/v1alpha1: make grpc/host These changes add the `KubeAuditLogsStream()` rpc and associated types.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add a
KubeAuditLogsStream()rpc to theAccessGraphServiceforstreaming Kubernetes apiserver audit logs from the Teleport discovery
service to access graph. This is intended for EKS audit logs which are
made available via CloudWatch, but can accommodate other k8s services.
The audit log messages are represented as a
google.protobuf.Structsoas to not depend on the k8s.io .proto files, but also as k8s typically
uses protos internally only - the expectation is that we'll receive the
apiserver audit logs as json-encoded strings. This encodes easily as a
google.protobuf.Struct.Link: https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/apiserver/pkg/apis/audit/v1/generated.proto
Issue: https://github.com/gravitational/access-graph/issues/1589