Emit new event for DynamoDB requests via app access by GavinFrazar · Pull Request #17595 · gravitational/teleport

GavinFrazar · 2022-10-19T19:39:06Z

Relevant issue: #15572

I will open a PR to the webapps repo to add support for this new event in the Web UI audit log.

What

This PR adds auditing for the new AppSessionDynamoDBRequest event.

When an HTTP request for dynamodb is sent to the app service, an audit event is emitted to both the audit log and a session stream.
Users will be able to see the request event in the audit log, as well as by playing back the associated app session chunk with tsh play --format=json <chunk id>

Why

We want to add database-access for DynamoDB, and for that we plan to emit dynamo HTTP request events containing the request JSON body directly to the audit log.
The existing generic AppSessionRequest was not suitable, because we intentionally filter those from being emitted to the audit log - they are only sent to a session stream and uploaded later (typically after 5 minutes).
We will continue to support dynamodb via app-access; to be consistent with the planned database-access DynamoDB audit logging, we want to emit app access dynamo request events to the audit log and session stream.

How

I refactored the audit logging in the app service. A new file lib/srv/app/common/audit.go contains the definition of an interface for emitting audit events on session start/end/chunk/request.

This was to consolidate the audit logging in one consistent place.

I added an IsDynamoDB method to the types.Application interface, which can be used to determine if an app is for DynamoDB

I added a new event/code for dynamodb events.

I parse dynamo http request bodies into our wrapper for protobuf/types.Struct - this is because the body is already JSON and we need a type that is basically equivalent to map[string]interface{} so that the protobuf can be marshalled to JSON properly.
Example event body in WebUI using this data type for the body:

...
  "body": {
    "TableName": "test"
  },

If I had used bytes for the body type instead, the result is audit logs with base64 encoded json - which is unreadable. Example event body viewed in Web UI, if I use bytes as the body type:

...
"body": "eyJUYWJsZU5hbWUiOiAibmljLXRlc3QifQ==",

If I had used string for the body type and just copied the request body into the string, then when the protobuf is marshaled to JSON for display to the user, the string (which is already JSON) would be escaped to make valid JSON. Example event body viewed in Web UI if I use string as the body type:

...
  "body": "{\"TableName\": \"test\"}"

The branch for this PR is branched off of the protos change PR branch. Github will automatically retarget this PR base to master when the protos changes merge.

We will include a similar event for dynamodb via database-access. We split the events so that app and database access events are not coupled. This way we do not have to include optional database/app metadata in one event too.

tobiaszheller

lgtm but would appreciate someone more experienced to take closer look

GavinFrazar · 2022-10-26T18:36:20Z

@smallinsky addressed your concerns

…proto

* Use generic console app uri to test that we differentiate request by endpoint instead of app uri * Use a dynamodb request which has a body to test that we include the body in the audit event * Test for expected body JSON

GavinFrazar · 2022-11-15T02:11:43Z

@gabrielcorado @mdwn could one of you take a look? I need a group 2 approver apparently

edit: disregard I guess? It failed the reviewer check before but now it seems to be happy. idk why

github-actions · 2022-11-15T22:59:10Z

@GavinFrazar See the table below for backport results.

Branch	Result
branch/v11	Failed

GavinFrazar · 2022-11-16T01:05:57Z

Backport to v11 will be pending an answer from product team @smallinsky

* Add a new event for app access requests sent to AWS DynamoDB

* Emit new event for DynamoDB requests via app access (#17595) * Add a new event for app access requests sent to AWS DynamoDB * Refactor app access (#19387) * Move logic out of RoundTrip and into ServeHTTP as a middleware before handing off to oxy forwarder * Move AWS signing service code into lib/utils/aws/signing.go * use app server close context for audit event emitting * add go doc comments. * refactor request rewriting to make the copy in a more robust way. * pass status code as uint32 rather than casting in audit emitter * clone request instead of making a new request, and rewrite url to force https * update header handling * Set oxy forwarder to PassHostHeader=false to ensure the host header is the URL being sought. * Remove code that deleted forwarding headers previously, we should keep those (X-Forwarded-*). * Audit log the AWS Host sought rather than the incoming request Host header (prior behavior maintained, we just rewrite the request differently using Clone). * Remove obsolete header copying helper func * Remove api error header from backport

GavinFrazar added 6 commits October 16, 2022 22:30

protobuf update

51cb57a

Update protos

0d4572d

Update oneof

f5e4071

Move AppMetaData up with the other metadata and add a 'target' field

0d054f9

Remove operation plane

7e770c3

GavinFrazar added the backport/branch/v11 label Oct 19, 2022

GavinFrazar marked this pull request as ready for review October 19, 2022 19:39

github-actions Bot requested review from espadolini and tobiaszheller October 19, 2022 19:39

github-actions Bot added application-access audit-log Issues related to Teleports Audit Log labels Oct 19, 2022

GavinFrazar commented Oct 19, 2022

View reviewed changes

Comment thread lib/service/service.go

GavinFrazar commented Oct 19, 2022

View reviewed changes

Comment thread lib/utils/aws/aws.go Outdated

zmb3 reviewed Oct 19, 2022

View reviewed changes

Comment thread lib/srv/app/aws/handler.go Outdated

Comment thread lib/srv/app/aws/handler.go

Comment thread lib/srv/app/aws/handler_test.go Outdated

Comment thread lib/utils/aws/aws.go Outdated

GavinFrazar mentioned this pull request Oct 19, 2022

Add new audit event for app-access DynamoDB requests gravitational/webapps#1279

Merged

GavinFrazar force-pushed the gavinfrazar/improve_dynamodb_audit_events branch from c961714 to 295a111 Compare October 19, 2022 23:32

tobiaszheller approved these changes Oct 21, 2022

View reviewed changes

Comment thread lib/service/service.go Outdated

Comment thread lib/service/service.go Outdated

Comment thread lib/srv/app/common/audit.go Outdated

tobiaszheller approved these changes Oct 24, 2022

View reviewed changes

GavinFrazar requested a review from smallinsky October 25, 2022 02:20

smallinsky reviewed Oct 25, 2022

View reviewed changes

Comment thread lib/srv/app/aws/handler.go Outdated

Comment thread api/types/app.go Outdated

Comment thread lib/srv/app/common/audit.go

GavinFrazar added 8 commits October 26, 2022 11:36

Merge branch 'master' into gavinfrazar/improve_dynamodb_audit_events_…

fd9e2a5

…proto

Merge branch 'master' into gavinfrazar/improve_dynamodb_audit_events_…

5bee7e8

…proto

Merge branch 'master' into gavinfrazar/improve_dynamodb_audit_events_…

4ac6c44

…proto

Merge branch 'master' into gavinfrazar/improve_dynamodb_audit_events_…

a521794

…proto

Fix typo

4dc3ed2

Configure signing service with transport instead of http client

8624a7f

Protect from resource exhaustion attacks

df9ef4d

Add IsDynamoDB to types.Application

54ccadf

GavinFrazar added 7 commits November 1, 2022 18:02

Rename ok->shouldSkipCleanup to make the purpose of it more clear

1ef86b7

Refactor request body decoding into aws utils

e1487e2

Use request instead of signed request for audit event

70cf583

Determine if req is for a dynamo endpoint instead of checking app uri

cb93e23

Remove obsolete app func IsDynamoDB

d146200

Update handler test

165c294

* Use generic console app uri to test that we differentiate request by endpoint instead of app uri * Use a dynamodb request which has a body to test that we include the body in the audit event * Test for expected body JSON

fix lint

c7a8c65

GavinFrazar force-pushed the gavinfrazar/improve_dynamodb_audit_events branch from f68a446 to c7a8c65 Compare November 2, 2022 01:05

Base automatically changed from gavinfrazar/improve_dynamodb_audit_events_proto to master November 2, 2022 01:23

Merge branch 'master' into gavinfrazar/improve_dynamodb_audit_events

6da639b

espadolini removed their request for review November 4, 2022 09:57

smallinsky approved these changes Nov 14, 2022

View reviewed changes

Merge branch 'master' into gavinfrazar/improve_dynamodb_audit_events

63e97db

GavinFrazar requested review from gabrielcorado and mdwn November 15, 2022 02:11

github-actions Bot removed request for gabrielcorado and mdwn November 15, 2022 06:57

GavinFrazar added 3 commits November 14, 2022 23:19

Fixup merge

48d0301

Merge branch 'master' into gavinfrazar/improve_dynamodb_audit_events

b7a5dbf

Merge branch 'master' into gavinfrazar/improve_dynamodb_audit_events

0649531

GavinFrazar enabled auto-merge (squash) November 15, 2022 22:39

GavinFrazar merged commit d9b80fb into master Nov 15, 2022

GavinFrazar deleted the gavinfrazar/improve_dynamodb_audit_events branch November 16, 2022 01:05

GavinFrazar added a commit that referenced this pull request Dec 24, 2022

Emit new event for DynamoDB requests via app access (#17595)

86dc591

* Add a new event for app access requests sent to AWS DynamoDB

GavinFrazar mentioned this pull request Dec 24, 2022

[v11] add app access dynamodb event #19667

Merged

GavinFrazar added a commit that referenced this pull request Dec 29, 2022

Emit new event for DynamoDB requests via app access (#17595) (#19667)

0fba3cd

* Add a new event for app access requests sent to AWS DynamoDB

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Emit new event for DynamoDB requests via app access#17595

Emit new event for DynamoDB requests via app access#17595
GavinFrazar merged 43 commits intomasterfrom
gavinfrazar/improve_dynamodb_audit_events

GavinFrazar commented Oct 19, 2022 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

tobiaszheller left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

GavinFrazar commented Oct 26, 2022

Uh oh!

GavinFrazar commented Nov 15, 2022 •

edited

Loading

Uh oh!

github-actions Bot commented Nov 15, 2022

Uh oh!

GavinFrazar commented Nov 16, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

GavinFrazar commented Oct 19, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What

Why

How

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

tobiaszheller left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

GavinFrazar commented Oct 26, 2022

Uh oh!

GavinFrazar commented Nov 15, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Nov 15, 2022

Uh oh!

GavinFrazar commented Nov 16, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

GavinFrazar commented Oct 19, 2022 •

edited

Loading

GavinFrazar commented Nov 15, 2022 •

edited

Loading