Require all audit events to be able to be trimmed#45931
Require all audit events to be able to be trimmed#45931capnspacehook merged 3 commits intomasterfrom
Conversation
|
@capnspacehook - this PR will require admin approval to merge due to its size. Consider breaking it up into a series smaller changes. |
8ba4aac to
a6a2172
Compare
a6a2172 to
c6dc83b
Compare
|
This implementation does fix #36442 which will prevent the event handler from crashing. But I don't think this fully addresses https://github.com/gravitational/teleport-private/issues/1299. Big events are now partially logged but the trimmed parts are lost. FYI, database events are duplicated to session recordings so it's kinda ok that db events are partially logged. But other types will likely lose the trimmed parts forever. |
|
@greedy52 trimming isn't ideal, but it should be a relatively rare occurrence. Since only certain fields are trimmed, most fields will still have the original data fully intact. Currently larger events are dropped, so trimming certain fields is better than dropping the whole thing. When I was discussing this with @r0mant he did mention that it would probably be possible to spit large events into several smaller events, but for now we are just going to trim events because events that are too large are pretty rare and trimming is much simpler to implement. |
Not opposing to this solution. Just saying this does not fully fix https://github.com/gravitational/teleport-private/issues/1299. We should either leave the issue open, or create a separate issue to track that trimmed parts are missing from audit logs. |
| event = v.TrimToMaxSize(MaxProtoMessageSizeBytes) | ||
| default: | ||
| event = event.TrimToMaxSize(MaxProtoMessageSizeBytes) | ||
| if event.Size() > MaxProtoMessageSizeBytes { |
There was a problem hiding this comment.
Will this still be possible after trimming, or will it be rare? Any way to prevent the event handler from crashing when running into this case?
There was a problem hiding this comment.
This will still be possible but should be very rare, as some metadata is not trimmed but is unlikely to be very large. I don't think returning an error here will cause the event handler to crash, an error could be returned here before and I don't believe this caused the event handler to crash.
c6dc83b to
27f6641
Compare
….*' events can be converted from EventFields
27f6641 to
18bf453
Compare
|
@capnspacehook See the table below for backport results.
|
We have encountered several audit events that sometimes are too large for a specific backend to process. Historically we've just addressed the offending events on a per-event basis. This PR requires all audit events to implement
TrimToMaxSizeto trim themselves to prevent this from happening again, and/or making it much easier to deal with large events in the future.The
TrimToMaxSizemethod has been implemented for all events, however for many events you will notice that the event is returned unchanged. When implementing these methods I only trimmed fields that are generally controlled by user input, or that can easily grow quite large (error messages for example). Many events have no such fields and as such don't attempt to trim themselves. Trimming general event metadata (UserMetadata,ConnectionMetadata, etc) was not in scope but this can be easily added in the future.The existing code that calls
TrimToMaxSizemethods have been modified to take into account the fact that all audit events implementTrimToMaxSizeand that trimming to a specific size may not be possible.I recommend you review commit by commit. Most of the additions are
TrimToMaxSizemethods which are very repetitive and verbose.Fixes https://github.com/gravitational/teleport-private/issues/1299.
Fixes #36442.
changelog: allow all audit events to be trimmed if necessary