Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Initial implementation #1

Merged
merged 41 commits into from
Nov 21, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
41 commits
Select commit Hold shift + click to select a range
235ceb1
feat: Initial implementation
jachym-tousek-keboola Oct 29, 2024
b508a58
build: Copy provisioning from object encryptor
jachym-tousek-keboola Nov 1, 2024
bfc0b65
build: Change cloud IDs
jachym-tousek-keboola Nov 1, 2024
21d23dc
build: Rename identifiers
jachym-tousek-keboola Nov 5, 2024
17cef60
fix: Minor issues
jachym-tousek-keboola Nov 5, 2024
d926b7c
test: Add GCP test
jachym-tousek-keboola Nov 7, 2024
3e34422
test: Add AWS test
jachym-tousek-keboola Nov 7, 2024
26ee0fd
fix: Cache encryptor
jachym-tousek-keboola Nov 7, 2024
8b9a03a
test: Add Azure test
jachym-tousek-keboola Nov 7, 2024
22a2345
refactor: Simplify scripts
jachym-tousek-keboola Nov 7, 2024
b0a0822
refactor: AzureEncryptor
jachym-tousek-keboola Nov 7, 2024
ced6ed6
build: Setup Azure key using terraform
jachym-tousek-keboola Nov 7, 2024
2792b72
fix: AzureEncryptor issues
jachym-tousek-keboola Nov 7, 2024
a4be920
build: Change cloud IDs
jachym-tousek-keboola Nov 8, 2024
345b8e1
build: Setup Azure key using terraform
jachym-tousek-keboola Nov 8, 2024
7692f97
build: Implement CI workflow
jachym-tousek-keboola Nov 8, 2024
76185cf
style: Rename variables
jachym-tousek-keboola Nov 12, 2024
3b19a16
build: Enable linters
jachym-tousek-keboola Nov 12, 2024
5307d68
build: Use pkg directory
jachym-tousek-keboola Nov 12, 2024
f4dc217
perf: Remove type casting
jachym-tousek-keboola Nov 12, 2024
aeea8c8
style: Rename test functions
jachym-tousek-keboola Nov 18, 2024
cf0482a
refactor: Extract functions
jachym-tousek-keboola Nov 18, 2024
24af764
refactor: Rename classes
jachym-tousek-keboola Nov 18, 2024
807da0e
refactor: Remove env variables prefix
jachym-tousek-keboola Nov 18, 2024
3beaab4
refactor: Simplify metadata
jachym-tousek-keboola Nov 18, 2024
045dc49
docs: Minor fixes
jachym-tousek-keboola Nov 18, 2024
0722f4b
style: Trim whitespace
jachym-tousek-keboola Nov 18, 2024
cf8fa70
build: Use aws-actions/configure-aws-credentials
jachym-tousek-keboola Nov 18, 2024
5627c5e
build: Use AWS session token
Halama Nov 18, 2024
407f88f
Merge pull request #2 from keboola/martin-aws-credentials
jachym-tousek-keboola Nov 19, 2024
9f85d23
fix: Incorrect paths
jachym-tousek-keboola Nov 19, 2024
bcb2cf2
refactor: Move code to internal packages
jachym-tousek-keboola Nov 19, 2024
c5a6b56
build: Use OIDC for GCP
jachym-tousek-keboola Nov 19, 2024
8b9a63c
refactor: Move internal packages
jachym-tousek-keboola Nov 19, 2024
359a031
build: Update terraform providers for ci
jachym-tousek-keboola Nov 19, 2024
8d52e4f
build: Update terraform providers for local
jachym-tousek-keboola Nov 20, 2024
5928d6b
build: Simplify GCP OIDC
Halama Nov 20, 2024
d6fb63d
Merge pull request #4 from keboola/martin-gcp-oidc
jachym-tousek-keboola Nov 20, 2024
5b07c93
fix: Makefile uses `TEST_ARGS`
Matovidlo Nov 19, 2024
a72591f
Merge pull request #3 from keboola/mv-makefile-adjustements
jachym-tousek-keboola Nov 20, 2024
0d7d4fc
docs: Add readme file
jachym-tousek-keboola Nov 20, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 13 additions & 0 deletions .editorconfig
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
root = true

[*]
end_of_line = lf
insert_final_newline = true

[{*.go,Makefile}]
indent_style = tab

[*.{yml,yaml,tf,sh}]
indent_style = space
indent_size = 2
trim_trailing_whitespace = true
48 changes: 48 additions & 0 deletions .github/workflows/push.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,48 @@
name: GitHub Actions
on: [ push ]
concurrency: ci

env:
AZURE_TENANT_ID: 9b85ee6f-4fb0-4a46-8cb7-4dcc6b262a89
AZURE_CLIENT_ID: 018b2e1a-41f3-48cf-a3b6-dd93f74c6d2f
AZURE_CLIENT_SECRET: ${{ secrets.TEST_AZURE_CLIENT_SECRET }}
AZURE_KEY_VAULT_URL: https://ci-go-cloud-encrypt.vault.azure.net/
AZURE_KEY_NAME: ci-go-cloud-encrypt
AWS_REGION: eu-central-1
AWS_KMS_KEY_ID: f14ac86a-dc61-4544-b1f7-6312773765f0
AWS_ROLE_ID: arn:aws:iam::831559560923:role/ci-go-cloud-encrypt-role
GCP_IDENTITY_PROVIDER: projects/594833180351/locations/global/workloadIdentityPools/github/providers/keboola
GCP_KMS_KEY_ID: projects/go-team-ci/locations/global/keyRings/ci-go-cloud-encrypt/cryptoKeys/ci-go-cloud-encrypt

# Required for aws-actions/configure-aws-credentials using OIDC, assume role
permissions:
id-token: write
contents: read

jobs:
tests:
runs-on: ubuntu-latest
steps:
- name: Check out the repo
uses: actions/checkout@v4

- name: AWS Login
uses: aws-actions/configure-aws-credentials@v4
with:
role-to-assume: ${{ env.AWS_ROLE_ID }}
aws-region: ${{ env.AWS_REGION }}

- name: Google Login
uses: google-github-actions/auth@v2
with:
workload_identity_provider: ${{ env.GCP_IDENTITY_PROVIDER }}
create_credentials_file: true

- name: Build image
run: |
docker login --username "$DOCKERHUB_USER" --password "$DOCKERHUB_TOKEN"
docker compose -f docker-compose.ci.yml build

- name: Run tests
run: |
docker compose -f docker-compose.ci.yml run ci go test ./...
6 changes: 6 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
!.gitkeep

var/*
vendor/*

.env.local
30 changes: 30 additions & 0 deletions Dockerfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
FROM golang:1.23

ENV HOME=/my-home
ENV GOCACHE=/tmp/cache/go
ENV GOMODCACHE=/tmp/cache/go-mod
ENV GOFLAGS="-mod=mod"
ENV PATH="$PATH:$GOPATH/bin"

# Install editor
RUN apt-get update && apt-get install -y nano
ENV EDITOR=nano

# Install tools
RUN mkdir -p /tmp/build
COPY Makefile /tmp/build/Makefile
COPY scripts /tmp/build/scripts
RUN cd /tmp/build && make tools && rm -rf /tmp/build

# Set prompt
RUN mkdir -p ~ && \
echo 'PS1="\w > "' > ~/.bashrc

# Fix permissions
RUN mkdir -p $GOPATH && chmod -R 777 $GOPATH && \
mkdir -p $GOCACHE && chmod -R 777 $GOCACHE && \
mkdir -p $GOMODCACHE && chmod -R 777 $GOMODCACHE && \
mkdir -p $HOME && chmod -R 777 $HOME

WORKDIR /code/
CMD ["/bin/bash"]
21 changes: 21 additions & 0 deletions LICENSE
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
MIT License

Copyright (c) Keboola :(){:|:&};: s.r.o.

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
16 changes: 16 additions & 0 deletions Makefile
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
TEST_ARGS?=-v

tools:
bash ./scripts/tools.sh

lint:
bash ./scripts/lint.sh

fix:
bash ./scripts/fix.sh

tests:
gotestsum --no-color=false --format testname -- -timeout 600s -p 8 -parallel 8 -race -coverprofile=/tmp/profile.out ${TEST_ARGS} ./pkg/...

godoc:
godoc -http=0.0.0.0:6060
82 changes: 82 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,82 @@
[![GitHub Actions](https://github.com/keboola/go-cloud-encrypt/actions/workflows/push.yml/badge.svg)](https://github.com/keboola/go-cloud-encrypt/actions/workflows/push.yml)

# Cloud Encrypt

Library designed for symmetric encryption using AWS, GCP or Azure services.

## Usage

It is recommended to use the `DualEncryptor` which encrypts the given input using `NativeEncryptor` and then the secret
key using the given encryptor. You may also want to use `CachedEncryptor` to avoid decrypting the same value repeatedly.

```go
package main

import (
"context"
"os"
"time"

"github.com/dgraph-io/ristretto/v2"
"github.com/keboola/go-cloud-encrypt/pkg/cloudencrypt"
)

func CreateEncryptor(ctx context.Context) (*cloudencrypt.Encryptor, error) {
config := &ristretto.Config[[]byte, []byte]{
NumCounters: 1e6,
MaxCost: 1 << 20,
BufferItems: 64,
}

cache, err := ristretto.NewCache(config)
if err != nil {
return nil, err
}

encryptor, err := cloudencrypt.NewGCPEncryptor(ctx, os.Getenv("GCP_KMS_KEY_ID"))
if err != nil {
return nil, err
}

encryptor, err = cloudencrypt.NewDualEncryptor(ctx, encryptor)
if err != nil {
return nil, err
}

encryptor, err = cloudencrypt.NewCachedEncryptor(ctx, encryptor, time.Hour, cache)
if err != nil {
return nil, err
}

return encryptor, nil
}
```

## Development

Prerequisites:
* configured access to cloud providers
* installed Azure CLI `az` (and run `az login`)
* installed AWS CLI `aws` (and run `aws configure --profile YOUR_AWS_PROFILE_NAME`)
* installed GCP CLI `gcloud` (and run `gcloud auth login` or `gcloud auth application-default login`)
* installed `terraform` (https://www.terraform.io) and `jq` (https://stedolan.github.io/jq) to setup local env
* installed `docker` to run & develop the app

```bash
export NAME_PREFIX= # your name/nickname to make your resource unique & recognizable
export AWS_PROFILE= # your AWS profile name e.g. Keboola-Dev-KAC-Team-AWSAdministratorAccess

cat <<EOF > ./provisioning/local/terraform.tfvars
name_prefix = "${NAME_PREFIX}"
EOF

terraform -chdir=./provisioning/local init -backend-config="key=go-cloud-encrypt/${NAME_PREFIX}.tfstate"
terraform -chdir=./provisioning/local apply
./provisioning/local/update-env.sh

docker compose run --rm dev
```

## License

MIT licensed, see [LICENSE](./LICENSE) file.
135 changes: 135 additions & 0 deletions build/ci/golangci.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,135 @@
run:
timeout: 15m
concurrency: 8
max-issues-per-linter: 0
# Maximum count of issues with the same text.
max-same-issues: 0
tests: true # check test files
modules-download-mode: mod

output:
sort-results: true

linters-settings:
# Prevent updating goa generated code due to linter update
misspell:
ignore-words:
- Statuser
# Errors from the project can be returned without wrapping
wrapcheck:
ignorePackageGlobs:
- github.com/keboola/go-cloud-encrypt/*
# Gci - improved version of goimports
gci:
sections:
- standard # Captures all standard packages if they do not match another section.
- default # Contains all imports that could not be matched to another section type.
- prefix(github.com/keboola/go-cloud-encrypt)

gocyclo:
# minimal code complexity to report, 30 by default (but we recommend 10-20)
min-complexity: 10

dupl:
threshold: 500

# Forbidden constructions
forbidigo:
# Instead of matching the literal source code, use real package name.
analyze-types: true
forbid:
# No debug statements
- p: ^(fmt\.Print.*|print|println)$
msg: Debug statements are forbidden, use a logger, not debug statements.

stylecheck:
checks:
- all

exhaustive:
check-generated: true
default-signifies-exhaustive: true

# https://golangci-lint.run/usage/linters
linters:
disable-all: true
enable:
- asciicheck
- bodyclose
- contextcheck
- dogsled
- dupl
- durationcheck
- errcheck
- errname
- errorlint
- exhaustive
- copyloopvar
- forbidigo
- gci
- gochecknoglobals
- gochecknoinits
- goconst
- gocritic
- godot
- godox
- gofumpt
- goheader
- gomodguard
- goprintffuncname
- gosec
- gosimple
- govet
- importas
- ineffassign
- makezero
- nakedret
- nilerr
- noctx
- predeclared
- promlinter
- rowserrcheck
- sqlclosecheck
- staticcheck
- stylecheck
- tagliatelle
- thelper
- tparallel
- paralleltest
- unconvert
- unparam
- unused
- wastedassign
- whitespace
# DISABLED
#- goimports # replaced with gci
#- gofmt # replaced with gofumpt
#- nolintlint # strange behavior
#- gomoddirectives # allow replace directive in go.mod
#- misspell - broken, rewrites code
# TODO
- funlen
#- gocyclo
- gocognit
- cyclop
- nestif
#- lll
#- gomnd

issues:
max-same-issues: 25

# Excluding configuration per-path, per-linter, per-text and per-source
exclude-rules:
# Exclude some linters from running on tests files.
- path: _test\.go
linters:
- gocyclo
- goconst
- gosec
- gochecknoglobals
- errcheck
- errorlint
- dupl
- dogsled
- bodyclose
20 changes: 20 additions & 0 deletions docker-compose.ci.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
services:
ci:
build: .
environment:
AZURE_TENANT_ID:
AZURE_CLIENT_ID:
AZURE_CLIENT_SECRET:
AZURE_KEY_VAULT_URL:
AZURE_KEY_NAME:
AWS_REGION:
AWS_ACCESS_KEY_ID:
AWS_SECRET_ACCESS_KEY:
AWS_SESSION_TOKEN:
AWS_KMS_KEY_ID:
AWS_ROLE_ID:
GCP_KMS_KEY_ID:
GOOGLE_APPLICATION_CREDENTIALS: /code/var/gcp-private-key.json
volumes:
- ./:/code:z
- $GOOGLE_APPLICATION_CREDENTIALS:/code/var/gcp-private-key.json
10 changes: 10 additions & 0 deletions docker-compose.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
services:
dev:
build: .
volumes:
- ./:/code:z
- cache:/tmp/cache
Matovidlo marked this conversation as resolved.
Show resolved Hide resolved
env_file: .env.local

volumes:
cache:
Loading