Skip to content

add dynamodb database access#18843

Merged
GavinFrazar merged 52 commits intomasterfrom
gavinfrazar/add-dynamodb-database-access
Dec 30, 2022
Merged

add dynamodb database access#18843
GavinFrazar merged 52 commits intomasterfrom
gavinfrazar/add-dynamodb-database-access

Conversation

@GavinFrazar
Copy link
Copy Markdown
Contributor

@GavinFrazar GavinFrazar commented Nov 29, 2022

This PR adds DynamoDB support to database access #17842

DAX, dynamodb, and dynamodb streams are all supported.

Why

We want to bring DynamoDB into database access for consistency and to enable some things not possible via app access.
This change will enable the use of NoSQLWorkbench GUI client for DynamoDB.

How

Configure one Teleport database for each AWS region you want in your cluster.

  • This is what enables NoSQL Workbench via "local" connections, because otherwise it sends requests to the non-existent "local" AWS region, and I cannot find a way to get it to respect HTTPS_PROXY env variable.

Example Configuration

  databases:
    - name: "home-dynamodb"
      protocol: "dynamodb"
# optional uri, if uri is set then AWS region can be extracted from that
# or if AWS region is already set then the regions must match.
      # uri: "dynamodb.us-west-2.amazonaws.com:443"
      static_labels:
        env: "dev"
      aws:
        region: "us-west-2"
        account_id: "278576220453"

RBAC

  • --db-name is not supported.
  • --db-user should be an AWS role, either "role/SomeRole" or "SomeRole" is fine. The full ARN is constructed using configured AWS account ID and the AWS partition (determined by region).

tsh

  • DynamoDB has no "shell-session" type of cli that I can find, which is unique among the db protocols we support. Therefore tsh db connect prints a user message telling them to use tsh proxy db --tunnel instead.
  • "The connect command" in tsh db ls -v is instead tsh proxy db --tunnel as well.
  • For now we require the --tunnel flag. I may add non-tunnel later, but it will only work with the aws cli, not NoSQLWorkbench, since NoSQLWorkbench does not have a way to configure trusted CA bundle.

Example usage:

  1. Setup local proxy tunnel: $ tsh proxy db --tunnel --port 8000 --db-user=GavinDynamoDBRole home-dynamodb
  2. (optionally) use aws cli: $ aws dynamodb list-tables --endpoint-url=http://localhost:8000
  3. (optionally) use NoSQLWorkbench:
  • click "Launch" Amazon DynamoDB > "Operation builder" > "Add connection" > "DynamoDB local" tab.
  • Name the connection as you want and match the local proxy port setting, then click "connect".
  • All operations thereon will be in the configured AWS region.
    image
  1. (optionally) use AWS sdk for your apps, for example python w/ boto3:
$ python3
Python 3.10.4 (main, Mar 31 2022, 03:37:37) [Clang 12.0.0 ] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import boto3
>>> clt = boto3.client('dynamodb', endpoint_url='http://localhost:8000')
>>> res = clt.list_tables()
>>> print(res)
{'TableNames': *snip output*}
>>> 

This PR is stacked against the app access PR #19387 for merge.

TODO: update webapps repo to recognize the DynamoDB db access request event codes.

Base automatically changed from gavinfrazar/add-dynamodb-database-access-protos to master December 1, 2022 05:13
@GavinFrazar GavinFrazar force-pushed the gavinfrazar/add-dynamodb-database-access branch from 021f223 to 36f9e07 Compare December 16, 2022 00:46
@GavinFrazar GavinFrazar changed the base branch from master to gavinfrazar/refactor-app-access-middleware December 16, 2022 00:49
@GavinFrazar
Copy link
Copy Markdown
Contributor Author

@smallinsky this is ready I just need to finish the writeup before I mark it ready for review by others, but since you are already familiar with what this PR accomplishes you can take a look if you want

@GavinFrazar GavinFrazar force-pushed the gavinfrazar/add-dynamodb-database-access branch from 72c80f4 to 9539b09 Compare December 16, 2022 08:12
@GavinFrazar GavinFrazar marked this pull request as ready for review December 16, 2022 08:12
@github-actions github-actions Bot added database-access Database access related issues and PRs size/lg tsh tsh - Teleport's command line tool for logging into nodes running Teleport. labels Dec 16, 2022
Comment thread api/go.mod Outdated
Comment thread api/types/database.go Outdated
Comment thread lib/srv/db/dynamodb/engine.go Outdated
Comment thread lib/srv/db/dynamodb/engine.go Outdated
Comment thread lib/srv/db/dynamodb/engine.go Outdated
Comment thread lib/srv/db/dynamodb/engine.go Outdated
Comment thread lib/srv/db/dynamodb/test.go Outdated
Comment thread lib/srv/db/dynamodb/test.go Outdated
Comment thread lib/srv/db/dynamodb/test.go Outdated
@github-actions
Copy link
Copy Markdown
Contributor

@GavinFrazar - this PR will require admin approval to merge due to its size. Consider breaking it up into a series smaller changes.

Comment thread lib/srv/db/dynamodb/engine.go Outdated
Comment thread lib/srv/db/dynamodb/engine.go Outdated
Comment thread lib/srv/db/dynamodb/engine.go Outdated
Comment thread lib/srv/db/dynamodb/engine.go Outdated
Comment thread api/types/database.go Outdated
Comment thread lib/srv/db/dynamodb/engine.go Outdated
Copy link
Copy Markdown
Contributor

@greedy52 greedy52 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

First pass!

Comment thread api/types/database.go Outdated
Comment thread api/utils/aws/endpoint.go Outdated
Comment thread api/utils/aws/endpoint.go Outdated
Comment thread lib/srv/db/dynamodb/engine.go Outdated
Comment thread lib/srv/db/dynamodb/engine.go Outdated
Comment thread lib/srv/db/dynamodb/engine.go Outdated
Comment thread lib/srv/db/dynamodb/engine.go Outdated
Comment thread lib/srv/db/dynamodb/engine.go Outdated
Comment thread tool/tsh/db.go Outdated
Comment thread tool/tsh/proxy.go Outdated
@GavinFrazar GavinFrazar force-pushed the gavinfrazar/add-dynamodb-database-access branch from f4aeffb to 0dc5ffe Compare December 21, 2022 03:02
Copy link
Copy Markdown
Contributor

@greedy52 greedy52 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me. Also tested it out today and worked great!

One note I believe dax support we have here today is only for for dax APIs.

I doubt if there are any customer needs dax endpoints access from Teleport (as we can access dynamodb directly). If so, we may consider doing dax endpoints support separately where use AWS API for discovery and dax client for engine. I can create a ticket for this.

Comment thread lib/events/codes.go Outdated
Comment thread tool/tsh/db.go Outdated
Base automatically changed from gavinfrazar/refactor-app-access-middleware to master December 23, 2022 18:05
GavinFrazar and others added 8 commits December 23, 2022 16:34
* add more comments
* revert changes that were not necessary
* copy http requests like we do elsewhere
* delete unused file
* fix typos
* misc code cleanup
Co-authored-by: rosstimothy <39066650+rosstimothy@users.noreply.github.com>
Co-authored-by: rosstimothy <39066650+rosstimothy@users.noreply.github.com>
@GavinFrazar GavinFrazar enabled auto-merge (squash) December 28, 2022 18:01
@GavinFrazar GavinFrazar disabled auto-merge December 28, 2022 18:01
@GavinFrazar GavinFrazar enabled auto-merge (squash) December 28, 2022 18:03
Comment thread lib/srv/db/dynamodb/engine.go Outdated
Comment thread lib/srv/db/dynamodb/engine.go Outdated
@GavinFrazar GavinFrazar force-pushed the gavinfrazar/add-dynamodb-database-access branch from cf5a353 to 3b51733 Compare December 29, 2022 18:47
@GavinFrazar
Copy link
Copy Markdown
Contributor Author

Due to PR size this requires admin approval to merge - who can do that? @r0mant @smallinsky

@smallinsky
Copy link
Copy Markdown
Contributor

smallinsky commented Dec 30, 2022

Due to PR size this requires admin approval to merge - who can do that?

I think that we need ask @r0mant or @zmb3 for approval.

Copy link
Copy Markdown
Collaborator

@r0mant r0mant left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bot.

@github-actions github-actions Bot removed the request for review from avatus December 30, 2022 16:37
@GavinFrazar GavinFrazar merged commit 5bb0ef1 into master Dec 30, 2022
@GavinFrazar GavinFrazar deleted the gavinfrazar/add-dynamodb-database-access branch December 30, 2022 20:43
GavinFrazar added a commit that referenced this pull request Jan 31, 2023
GavinFrazar added a commit that referenced this pull request Jan 31, 2023
GavinFrazar added a commit that referenced this pull request Feb 3, 2023
GavinFrazar added a commit that referenced this pull request Feb 3, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

database-access Database access related issues and PRs size/lg tsh tsh - Teleport's command line tool for logging into nodes running Teleport.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants