Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
108 commits
Select commit Hold shift + click to select a range
3c309f6
Begin rewrite in pure Python
b-long Jun 23, 2025
bddba91
Organize: git mv src/otdf_python/test_*.py tests/
b-long Jul 30, 2025
fcf723e
Format according to 'ruff'
b-long Jul 30, 2025
b4df15d
Fix static analysis
b-long Jul 30, 2025
4dddd0e
Cleanup and organize tests/test_validate_otdf_python.py
b-long Jul 30, 2025
098ba22
Remove 'TDFConfig' type from 'otdf_python.tdf'
b-long Jul 30, 2025
f789166
Fix description & formatting
b-long Jul 30, 2025
4d586a3
Add 'pydantic-settings' to dev & update dependencies
b-long Jul 30, 2025
bc705ec
Correct version number
b-long Jul 30, 2025
9269a9e
Cleanup and fix OIDC tests
b-long Jul 30, 2025
2a3841c
Comment old style integration test
b-long Jul 30, 2025
e38526c
Execute majority of tests
b-long Jul 30, 2025
762a939
Allow import from 'tests'
b-long Jul 31, 2025
bb9d16b
Fix string encryption test
b-long Aug 1, 2025
8df1854
Remove dead code
b-long Aug 1, 2025
d6f19f0
Adjust integration test
b-long Aug 1, 2025
5f63636
Remove old build scripts
b-long Aug 1, 2025
0384137
Update README
b-long Aug 1, 2025
01b3b9d
Update GHA triggers
b-long Aug 1, 2025
912ff4a
Fix endpoint URL and TLS verification
b-long Aug 4, 2025
8b4a535
✅ Significant update 143 out of 150 tests passing
b-long Aug 4, 2025
0669650
Run all tests, except integration
b-long Aug 8, 2025
e6a83d3
Update GHA configuration
b-long Aug 8, 2025
c792967
Mark integration tests
b-long Aug 8, 2025
fa796e5
Fix mocked tests/test_kas_client.py
b-long Aug 8, 2025
a822bc6
Mark integration tests
b-long Aug 8, 2025
e381632
Only build for 3.13 (temporary)
b-long Aug 8, 2025
78d760d
Merge branch 'main' into chore/rewrite
b-long Aug 8, 2025
eb14223
Update license
b-long Aug 8, 2025
adfdbdd
Enable and fix integration tests in CI
b-long Aug 8, 2025
30a6375
Improve support for plaintext
b-long Aug 8, 2025
8d06656
Make log collection optional
b-long Aug 8, 2025
336d169
Fix tests for plaintext
b-long Aug 8, 2025
9ecee9e
Fix docstrings
b-long Aug 8, 2025
6976cbb
Fix docstrings
b-long Aug 9, 2025
a96a138
Extract Connect RPC class
b-long Aug 9, 2025
2d48aea
Fix additional roundtrip testing
b-long Aug 9, 2025
6361b22
Fix tests after kas_client updates
b-long Aug 9, 2025
3cc1ebc
Expand KAS client integration tests
b-long Aug 9, 2025
cc47a52
Fix mimeType
b-long Aug 9, 2025
66e04e1
Expand testing, fix compression bug
b-long Aug 9, 2025
e820de7
Auto-use check_for_otdfctl fixture
b-long Aug 10, 2025
781c8c0
Expand static analysis, fix FURB188
b-long Aug 10, 2025
01b8169
Use 'NULL_POLICY_UUID' for now
b-long Aug 10, 2025
1dc83a8
Update kas_client.py & tdf.py, expand tests
b-long Aug 10, 2025
f2e28ba
Expand & organize integration tests
b-long Aug 10, 2025
fbc5529
Expand static analysis, fix PT018
b-long Aug 10, 2025
791b6c9
Use configurable attrs in testing
b-long Aug 10, 2025
5d132c1
Use configurable attrs in testing
b-long Aug 10, 2025
b66294e
Examine entitlements in CI
b-long Aug 10, 2025
ed87839
Extract 'temp_credentials_file' fixture
b-long Aug 10, 2025
155121f
Rename file
b-long Aug 10, 2025
468db62
Modernize release workflows
b-long Aug 11, 2025
dc56ef0
Modernize release workflows
b-long Aug 11, 2025
58f5558
Update release workflow
b-long Aug 11, 2025
02f51cd
Merge pull request #61 from b-long/chore/rewrite-in-python
b-long Aug 11, 2025
d2a0347
Manage 'otdf-python-proto' as a sub-package
b-long Aug 12, 2025
3448670
Update README
b-long Aug 12, 2025
8cdc401
Merge pull request #63 from b-long/chore/packaging-updates
b-long Aug 12, 2025
04a650c
Manage 'otdf-python-proto' as a sub-package
b-long Aug 12, 2025
4c4119e
Merge pull request #64 from b-long/chore/packaging-updates-again
b-long Aug 12, 2025
20e986c
Support Python 3.10+
b-long Aug 12, 2025
ca7903d
Merge pull request #65 from b-long/chore/packaging-updates-additional…
b-long Aug 12, 2025
8f642ae
Fix version number
b-long Aug 12, 2025
78d583f
Merge pull request #66 from b-long/chore/packaging-fix-version-number
b-long Aug 12, 2025
0736b3e
Fix Python version requirement
b-long Aug 13, 2025
cf31aed
Bump version 0.3.0a4 -> 0.3.0a5
b-long Aug 13, 2025
be91b69
Merge pull request #67 from b-long/chore/fix-python-version-requirement
b-long Aug 13, 2025
6f01c1e
Fix version extract command
b-long Aug 13, 2025
30b032b
Undo file name change
b-long Aug 13, 2025
84cf5c1
Merge branch 'chore/fix-python-version-redux' into develop
b-long Aug 13, 2025
ace14e7
More support for PE flows, cleanup & improved typing (#70)
b-long Aug 23, 2025
9891ea8
Chore/update docs and release process (#72)
b-long Sep 4, 2025
5ed358b
Merge branch 'main' into develop
b-long Sep 4, 2025
9cd6d44
chore: fix release-please config
b-long Sep 5, 2025
a4c6cb0
chore: fix version number
b-long Sep 5, 2025
652e742
chore: use standard 'workflow_call'
b-long Sep 5, 2025
0cc0959
chore: clean up publishing
b-long Sep 5, 2025
dbc9f80
fix: fix publishing
b-long Sep 5, 2025
3bb4283
chore: release 0.3.0a10
b-long Sep 5, 2025
de00583
fix: fix publishing
b-long Sep 5, 2025
ce8a520
chore: release 0.3.0a11
b-long Sep 5, 2025
5f3b907
chore: release develop (#81)
github-actions[bot] Sep 5, 2025
8e5b86f
chore: align version numbers
b-long Sep 5, 2025
ee6452e
chore: add 'otdf-python-proto/uv.lock' file
b-long Sep 5, 2025
af69a88
chore: add 'otdf-python-proto/uv.lock' file
b-long Sep 5, 2025
cca9e3e
fix: omit README from Github releases
b-long Sep 5, 2025
35d06e3
chore: document legacy version
b-long Sep 5, 2025
f61b020
fix: address pre-commit (lint) issues
b-long Sep 5, 2025
c699d02
chore: verbose output for pypi uploads
b-long Sep 5, 2025
0e09171
fix: use correct 'extra-files' for uv.lock
b-long Sep 5, 2025
c1395b1
chore: release 0.3.1
b-long Sep 5, 2025
e24a665
chore: release develop (#82)
github-actions[bot] Sep 5, 2025
56117b4
chore: organize docs
b-long Sep 5, 2025
6a1d57b
fix: remove unnecessary 'ncipollo/release-action'
b-long Sep 5, 2025
0cc49f9
chore: add developer doc
b-long Sep 5, 2025
e5b4efd
chore: CI improvements (#88)
b-long Sep 10, 2025
238715f
fix: guarantee target-version decrypt support (#84)
b-long Sep 10, 2025
13be279
chore: cleanup and release (#93)
b-long Sep 10, 2025
57d723d
chore: fix release-please
b-long Sep 10, 2025
fe1ee2d
fix: release-please configuration (#95)
b-long Sep 11, 2025
6cfaee6
fix: update prerelease config for develop branch
b-long Sep 11, 2025
9f2d2ed
chore(develop): release otdf-python 0.3.1 (#96)
github-actions[bot] Sep 11, 2025
085f056
fix: fix .release-please-config.json file (#97)
b-long Sep 11, 2025
646345c
chore(develop): release otdf-python 0.3.2 (#98)
github-actions[bot] Sep 11, 2025
11497fb
fix: release configuration (#99)
b-long Sep 11, 2025
63ef99a
fix: add develop-specific release-please files and update workflow
b-long Sep 11, 2025
6cb25d8
chore(develop): release otdf-python 0.3.1 (#100)
github-actions[bot] Sep 11, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions .env-docker
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
OPENTDF_PLATFORM_HOST="localhost"
OPENTDF_PLATFORM_PORT=8080
OPENTDF_PLATFORM_URL="http://localhost:8080"

KEYCLOAK_URL="http://localhost:8888/auth"
OIDC_OP_TOKEN_ENDPOINT="http://localhost:8888/auth/realms/opentdf/protocol/openid-connect/token"
56 changes: 56 additions & 0 deletions .github/check_entitlements.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,56 @@
#!/bin/bash
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

For improved robustness, it's a good practice to add set -euo pipefail at the beginning of your shell scripts. This will cause the script to exit immediately if a command fails, preventing unexpected behavior.

Suggested change
#!/bin/bash
#!/bin/bash
set -euo pipefail


# Derive additional environment variables
TOKEN_URL="${OIDC_OP_TOKEN_ENDPOINT}"
OTDF_HOST_AND_PORT="${OPENTDF_PLATFORM_HOST}"
OTDF_CLIENT="${OPENTDF_CLIENT_ID}"
OTDF_CLIENT_SECRET="${OPENTDF_CLIENT_SECRET}"

echo "🔧 Environment Configuration:"
echo " TOKEN_URL: ${TOKEN_URL}"
echo " OTDF_HOST_AND_PORT: ${OTDF_HOST_AND_PORT}"
echo " OTDF_CLIENT: ${OTDF_CLIENT}"
echo " OTDF_CLIENT_SECRET: ${OTDF_CLIENT_SECRET}"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

Printing secrets like OTDF_CLIENT_SECRET and the bearer token (line 31) to the logs is a significant security risk. Even in debug or CI environments, logs can be exposed. It's best practice to redact secrets from all log output.

Suggested change
echo " OTDF_CLIENT_SECRET: ${OTDF_CLIENT_SECRET}"
echo " OTDF_CLIENT_SECRET: [REDACTED]"

echo ""

get_token() {
curl -k --location "$TOKEN_URL" \
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

Using curl -k disables SSL certificate validation, which is a security risk. While this might be for local development against a service with a self-signed certificate, it's a dangerous practice that could be accidentally used against other environments. It would be safer to either use a proper certificate or, if this is strictly for a local, trusted environment, add a prominent comment explaining why this is necessary and that it should not be used elsewhere.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The use of curl -k (and grpcurl -plaintext on line 39) disables certificate validation, which is a security risk. While this may be acceptable for a local test environment, it would be beneficial to add a comment explaining why this is necessary and to warn against using this script in production-like environments.

--header "X-VirtruPubKey;" \
--header "Content-Type: application/x-www-form-urlencoded" \
--data-urlencode "grant_type=client_credentials" \
--data-urlencode "client_id=$OTDF_CLIENT" \
--data-urlencode "client_secret=$OTDF_CLIENT_SECRET"
}

echo "🔐 Getting access token..."
BEARER=$( get_token | jq -r '.access_token' )
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

It's good practice to check if the BEARER token was successfully retrieved. If the curl or jq command fails, the script will continue with an empty token, which could lead to confusing errors later.

Suggested change
BEARER=$( get_token | jq -r '.access_token' )
BEARER=$( get_token | jq -r '.access_token' )
if [ -z "$BEARER" ]; then
echo "❌ Failed to get access token."
exit 1
fi

# NOTE: It's always okay to print this token, because it will
# only be valid / available in dummy / dev scenarios
[[ "${DEBUG:-}" == "1" ]] && echo "Got Access Token: ${BEARER}"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

Printing the bearer token to logs is a security risk. Please redact it to avoid exposing credentials.

Suggested change
[[ "${DEBUG:-}" == "1" ]] && echo "Got Access Token: ${BEARER}"
[[ "${DEBUG:-}" == "1" ]] && echo "Got Access Token: [REDACTED]"

echo ""

# Array of usernames to check
USERNAMES=("opentdf" "sample-user" "sample-user-1" "cli-client" "opentdf-sdk")

for USERNAME in "${USERNAMES[@]}"; do
echo "👤 Fetching entitlements for username: ${USERNAME}"
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"

grpcurl -plaintext \
-H "authorization: Bearer $BEARER" \
-d "{
\"entities\": [
{
\"userName\": \"$USERNAME\"
}
]
}" \
"$OTDF_HOST_AND_PORT" \
authorization.AuthorizationService/GetEntitlements

echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
echo "✅ Entitlements retrieval complete for ${USERNAME}!"
echo ""
done

echo "🎉 All entitlement checks completed!"
61 changes: 61 additions & 0 deletions .github/start_opentdf_docker.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,61 @@
#!/usr/bin/env bash

set -euo pipefail

if ! [ -d platform ]; then
git clone https://github.com/opentdf/platform.git
fi
cd platform
git checkout 3360befcb3e6e9791d7bfd2e89128aee0e7d2818 # Branch 'DSPX-1539-keytoolnomore'
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Pinning to a specific commit hash is great for reproducibility. However, it can become stale. It would be helpful to add a comment explaining how and when to update this hash, or consider using a Git submodule to manage this dependency more formally.


yq -i '.realms[0].clients[0].client.directAccessGrantsEnabled = true | .realms[0].clients[0].client.serviceAccountsEnabled = true' service/cmd/keycloak_data.yaml

yq -i '.realms[0].clients[1].client.directAccessGrantsEnabled = true | .realms[0].clients[1].client.serviceAccountsEnabled = true' service/cmd/keycloak_data.yaml

yq -i '.realms[0].clients[4].client.directAccessGrantsEnabled = true | .realms[0].clients[4].client.serviceAccountsEnabled = true' service/cmd/keycloak_data.yaml
Comment on lines +11 to +15
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

These yq commands are repetitive. You can use a loop to make the script more concise and easier to maintain if more clients need to be updated in the future.

Suggested change
yq -i '.realms[0].clients[0].client.directAccessGrantsEnabled = true | .realms[0].clients[0].client.serviceAccountsEnabled = true' service/cmd/keycloak_data.yaml
yq -i '.realms[0].clients[1].client.directAccessGrantsEnabled = true | .realms[0].clients[1].client.serviceAccountsEnabled = true' service/cmd/keycloak_data.yaml
yq -i '.realms[0].clients[4].client.directAccessGrantsEnabled = true | .realms[0].clients[4].client.serviceAccountsEnabled = true' service/cmd/keycloak_data.yaml
for i in 0 1 4; do
yq -i ".realms[0].clients[${i}].client.directAccessGrantsEnabled = true | .realms[0].clients[${i}].client.serviceAccountsEnabled = true" service/cmd/keycloak_data.yaml
done



if ! [ -d ./keys ]; then
go mod download

go mod verify

.github/scripts/init-temp-keys.sh
cp opentdf-example.yaml opentdf.yaml

# Edit 'opentdf.yaml' for our use case
yq -i 'del(.db) | .services.entityresolution.url = "http://localhost:8888/auth" | .server.auth.issuer = "http://localhost:8888/auth/realms/opentdf"' opentdf.yaml
# The above expression can also be written as 3 separate commands:
# yq -i 'del(.db)' opentdf.yaml
# yq -i '.services.entityresolution.url = "http://localhost:8888/auth"' opentdf.yaml
# yq -i '.server.auth.issuer = "http://localhost:8888/auth/realms/opentdf"' opentdf.yaml

yq -i '
.server.cryptoProvider = {
"type": "standard",
"standard": {
"keys": [
{
"kid": "r1",
"alg": "rsa:2048",
"private": "kas-private.pem",
"cert": "kas-cert.pem"
},
{
"kid": "e1",
"alg": "ec:secp256r1",
"private": "kas-ec-private.pem",
"cert": "kas-ec-cert.pem"
}
]
}
}
' opentdf.yaml
chmod -R 700 ./keys
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The permission 700 grants execute permissions to files, which is unnecessary for keys and could be a minor security risk. A more secure approach is to set 700 for directories and 600 for files. You can achieve this with:

find ./keys -type d -exec chmod 700 {} +;
find ./keys -type f -exec chmod 600 {} +;

fi

docker compose up -d --wait --wait-timeout 360

go run ./service provision keycloak

go run ./service provision fixtures
50 changes: 0 additions & 50 deletions .github/workflows/build-golang-macos.yaml

This file was deleted.

65 changes: 0 additions & 65 deletions .github/workflows/build-golang-ubuntu.yaml

This file was deleted.

72 changes: 35 additions & 37 deletions .github/workflows/build-python.yaml
Original file line number Diff line number Diff line change
@@ -1,44 +1,42 @@
---
name: Build Python package(s)

# Build otdf-python wheel using uv and output the wheel path for downstream workflows
name: "Build Python Wheel"
on:
push:
branches:
- disabled
push:
branches:
- chore/rewrite
pull_request:

jobs:
build:

runs-on: ubuntu-22.04
strategy:
matrix:
go-version: [1.24.x]
build:
runs-on: ubuntu-22.04
outputs:
wheel: ${{ steps.find_wheel.outputs.wheel_path }}
steps:
- name: Checkout this repo
uses: actions/checkout@v4

steps:
- uses: actions/checkout@v4
# - name: Setup Go
# uses: actions/setup-go@v4
# with:
# go-version: ${{ matrix.go-version }}
# cache-dependency-path: go.sum
# - name: Install dependencies
# run: go get .
# - name: Test with Go
# run: go test -timeout 40s -run ^TestHello$ gotdf_python -count=1 # go test
- name: Set up uv
uses: astral-sh/setup-uv@v6
with:
enable-cache: true
cache-dependency-glob: "uv.lock"

- name: Set up Python
uses: actions/setup-python@v4
with:
python-version: '3.12'
- name: Install dependencies
run: |
pip install poetry
- name: Invoke pylint with all dependencies
run: |
# Since we don't have our wheel build / install configured yet we use '--no-root'
poetry install --no-root
- name: Build otdf-python wheel using uv
run: |
uv sync --frozen
uv build
shell: bash

# poetry install
- name: Find built wheel
id: find_wheel
run: |
wheel_path=$(ls dist/*.whl | head -n1)
echo "wheel_path=$wheel_path" >> $GITHUB_OUTPUT
shell: bash

# Bring this back later
# poetry run pytest tests/
# - name: Upload wheel as artifact
# uses: actions/upload-artifact@v4
# with:
# name: python-wheel
# path: dist/*.whl
# overwrite: true
34 changes: 0 additions & 34 deletions .github/workflows/lint-on-macos.yaml

This file was deleted.

34 changes: 0 additions & 34 deletions .github/workflows/lint-on-ubuntu.yaml

This file was deleted.

Loading
Loading