Skip to content

Conversation

@pflynn-virtru
Copy link
Member

@pflynn-virtru pflynn-virtru commented Sep 4, 2025

Proposed Changes

This pull request introduces comprehensive documentation and architectural decisions to support custom assertion providers in the OpenTDF SDK. The changes focus on enabling integration with hardware-backed and cloud-based key management systems, improving troubleshooting for assertion-related errors, and ensuring extensibility and security across SDKs.

Custom Assertion Provider Architecture:

  • Added a new architectural decision record (adr/decisions/2025-10-16-custom-assertion-providers.md) detailing the adoption of the Binder/Validator pattern for assertion signing and validation, supporting hardware security modules, smart cards, and cloud KMS integration.
  • Defined AssertionBinder and AssertionValidator interfaces, with schema-based dispatch and registration methods for extensible provider integration.
  • Outlined verification modes (PermissiveMode, FailFast, StrictMode) and DEK fallback logic for robust assertion validation and forward compatibility.
  • Documented security considerations, including mandatory cryptographic bindings and secure key management practices.

Troubleshooting and Developer Guidance:

  • Added a new troubleshooting guide (docs/Assertions-Troubleshooting.md) covering common errors with key loading, assertion validation, TDF creation, and decryption, along with solutions and debugging tips.
  • Included practical examples, debugging techniques, and references to example code and specifications to help developers implement and test custom assertion providers.

These changes ensure the SDK remains secure, extensible, and developer-friendly while supporting advanced cryptographic integrations.

Checklist

  • I have added or updated unit tests
  • I have added or updated integration tests (if appropriate)
  • I have added or updated documentation

Testing Instructions

Implement pluggable assertion signing and validation providers to enable
integration with external signing mechanisms like HSMs, smart cards (CAC/PIV),
and cloud KMS services.

## What's New

### Core Interfaces
- Add `AssertionSigningProvider` interface for custom signing implementations
- Add `AssertionValidationProvider` interface for custom validation logic
- Maintain full backward compatibility with existing DEK-based assertions

### Built-in Providers
- `DefaultSigningProvider/DefaultValidationProvider`: Existing DEK-based behavior
- `X509SigningProvider/X509ValidationProvider`: X.509 certificate support with x5c headers
- `PKCS11Provider`: Template for hardware token integration

### SDK Integration
- Add `WithAssertionSigningProvider()` option for TDF creation
- Add `WithReaderAssertionValidationProvider()` option for TDF reading
- Automatically fall back to default providers when none specified

### Examples
- Add comprehensive assertion CLI commands (sign, verify, list)
- Support reading assertions from TDF files
- Add `--x509-verify` flag to decrypt command for X.509 validation

## Technical Details

The implementation follows a provider pattern that allows developers to:
1. Supply custom signing logic while maintaining SDK compatibility
2. Integrate with hardware security modules and smart cards
3. Use X.509 certificates for identity-based assertions
4. Maintain complete backward compatibility with existing code

All providers use the standard SDK assertion binding (`assertionHash` and
`assertionSig` claims) ensuring full interoperability between tools.

## Testing

- Added provider interface tests with mock implementations
- Added X.509 provider tests with self-signed certificates
- Verified interoperability with otdfctl-created TDFs
- Tested backward compatibility with existing assertions
@github-actions github-actions bot added comp:sdk A software development kit, including library, for client applications and inter-service communicati comp:examples size/xl labels Sep 4, 2025
@github-actions
Copy link
Contributor

github-actions bot commented Sep 4, 2025

Benchmark results, click to expand

Benchmark authorization.GetDecisions Results:

Metric Value
Approved Decision Requests 1000
Denied Decision Requests 0
Total Time 169.826476ms

Benchmark authorization.v2.GetMultiResourceDecision Results:

Metric Value
Approved Decision Requests 1000
Denied Decision Requests 0
Total Time 96.310574ms

Standard Benchmark Metrics Skipped or Failed

Bulk Benchmark Results

Metric Value
Total Decrypts 100
Successful Decrypts 100
Failed Decrypts 0
Total Time 345.257234ms
Throughput 289.64 requests/second

TDF3 Benchmark Results:

Metric Value
Total Requests 5000
Successful Requests 5000
Failed Requests 0
Concurrent Requests 50
Total Time 36.942498007s
Average Latency 367.265958ms
Throughput 135.35 requests/second

NANOTDF Benchmark Results:

Metric Value
Total Requests 5000
Successful Requests 5000
Failed Requests 0
Concurrent Requests 50
Total Time 25.777301652s
Average Latency 257.070745ms
Throughput 193.97 requests/second

@github-actions
Copy link
Contributor

github-actions bot commented Sep 5, 2025

Benchmark results, click to expand

Benchmark authorization.GetDecisions Results:

Metric Value
Approved Decision Requests 1000
Denied Decision Requests 0
Total Time 184.171883ms

Benchmark authorization.v2.GetMultiResourceDecision Results:

Metric Value
Approved Decision Requests 1000
Denied Decision Requests 0
Total Time 96.676921ms

Standard Benchmark Metrics Skipped or Failed

Bulk Benchmark Results

Metric Value
Total Decrypts 100
Successful Decrypts 100
Failed Decrypts 0
Total Time 347.09613ms
Throughput 288.10 requests/second

TDF3 Benchmark Results:

Metric Value
Total Requests 5000
Successful Requests 5000
Failed Requests 0
Concurrent Requests 50
Total Time 36.589224949s
Average Latency 363.7831ms
Throughput 136.65 requests/second

NANOTDF Benchmark Results:

Metric Value
Total Requests 5000
Successful Requests 5000
Failed Requests 0
Concurrent Requests 50
Total Time 25.609274398s
Average Latency 255.299414ms
Throughput 195.24 requests/second

@github-actions
Copy link
Contributor

github-actions bot commented Sep 5, 2025

Benchmark results, click to expand

Benchmark authorization.GetDecisions Results:

Metric Value
Approved Decision Requests 1000
Denied Decision Requests 0
Total Time 181.710987ms

Benchmark authorization.v2.GetMultiResourceDecision Results:

Metric Value
Approved Decision Requests 1000
Denied Decision Requests 0
Total Time 102.207188ms

Standard Benchmark Metrics Skipped or Failed

Bulk Benchmark Results

Metric Value
Total Decrypts 100
Successful Decrypts 100
Failed Decrypts 0
Total Time 355.455849ms
Throughput 281.33 requests/second

TDF3 Benchmark Results:

Metric Value
Total Requests 5000
Successful Requests 5000
Failed Requests 0
Concurrent Requests 50
Total Time 38.242845277s
Average Latency 380.34329ms
Throughput 130.74 requests/second

NANOTDF Benchmark Results:

Metric Value
Total Requests 5000
Successful Requests 5000
Failed Requests 0
Concurrent Requests 50
Total Time 26.248033022s
Average Latency 261.388463ms
Throughput 190.49 requests/second

@pflynn-virtru pflynn-virtru marked this pull request as ready for review September 5, 2025 17:29
@pflynn-virtru pflynn-virtru requested review from a team as code owners September 5, 2025 17:29
@github-actions
Copy link
Contributor

Benchmark results, click to expand

Benchmark authorization.GetDecisions Results:

Metric Value
Approved Decision Requests 1000
Denied Decision Requests 0
Total Time 166.974401ms

Benchmark authorization.v2.GetMultiResourceDecision Results:

Metric Value
Approved Decision Requests 1000
Denied Decision Requests 0
Total Time 86.580955ms

Standard Benchmark Metrics Skipped or Failed

Bulk Benchmark Results

Metric Value
Total Decrypts 100
Successful Decrypts 100
Failed Decrypts 0
Total Time 348.725865ms
Throughput 286.76 requests/second

TDF3 Benchmark Results:

Metric Value
Total Requests 5000
Successful Requests 5000
Failed Requests 0
Concurrent Requests 50
Total Time 36.678728922s
Average Latency 365.05833ms
Throughput 136.32 requests/second

NANOTDF Benchmark Results:

Metric Value
Total Requests 5000
Successful Requests 5000
Failed Requests 0
Concurrent Requests 50
Total Time 25.299112154s
Average Latency 252.187565ms
Throughput 197.64 requests/second

Enhance `--magic-word` handling with a more robust assertion provider setup, including default validation using `NoopAssertionValidationProvider` and state-aware `MagicWordAssertionProvider`.
@github-actions
Copy link
Contributor

Benchmark results, click to expand

Benchmark authorization.GetDecisions Results:

Metric Value
Approved Decision Requests 1000
Denied Decision Requests 0
Total Time 168.187219ms

Benchmark authorization.v2.GetMultiResourceDecision Results:

Metric Value
Approved Decision Requests 1000
Denied Decision Requests 0
Total Time 107.203728ms

Standard Benchmark Metrics Skipped or Failed

Bulk Benchmark Results

Metric Value
Total Decrypts 100
Successful Decrypts 100
Failed Decrypts 0
Total Time 356.702927ms
Throughput 280.35 requests/second

TDF3 Benchmark Results:

Metric Value
Total Requests 5000
Successful Requests 5000
Failed Requests 0
Concurrent Requests 50
Total Time 37.777895113s
Average Latency 375.988804ms
Throughput 132.35 requests/second

NANOTDF Benchmark Results:

Metric Value
Total Requests 5000
Successful Requests 5000
Failed Requests 0
Concurrent Requests 50
Total Time 26.129911958s
Average Latency 260.095998ms
Throughput 191.35 requests/second

… command

Streamline assertion provider integration:
- Remove unnecessary comments and redundant field (`AssertionProvider`) in `MagicWordAssertionProvider`.
- Adjust `WithAssertionProviderFactory` to use a pointer for consistency.
- Update regex in `decrypt.go` for precise assertion matching.
Streamline SDK by removing unused `assertion_binding.go`, including obsolete legacy binding logic and functions for TDF assertion handling.
@github-actions
Copy link
Contributor

Benchmark results, click to expand

Benchmark authorization.GetDecisions Results:

Metric Value
Approved Decision Requests 1000
Denied Decision Requests 0
Total Time 172.742883ms

Benchmark authorization.v2.GetMultiResourceDecision Results:

Metric Value
Approved Decision Requests 1000
Denied Decision Requests 0
Total Time 97.878579ms

Standard Benchmark Metrics Skipped or Failed

Bulk Benchmark Results

Metric Value
Total Decrypts 100
Successful Decrypts 100
Failed Decrypts 0
Total Time 354.388525ms
Throughput 282.18 requests/second

TDF3 Benchmark Results:

Metric Value
Total Requests 5000
Successful Requests 5000
Failed Requests 0
Concurrent Requests 50
Total Time 37.270893901s
Average Latency 370.606763ms
Throughput 134.15 requests/second

NANOTDF Benchmark Results:

Metric Value
Total Requests 5000
Successful Requests 5000
Failed Requests 0
Concurrent Requests 50
Total Time 25.701914861s
Average Latency 255.525215ms
Throughput 194.54 requests/second

@github-actions
Copy link
Contributor

Benchmark results, click to expand

Benchmark authorization.GetDecisions Results:

Metric Value
Approved Decision Requests 1000
Denied Decision Requests 0
Total Time 178.572034ms

Benchmark authorization.v2.GetMultiResourceDecision Results:

Metric Value
Approved Decision Requests 1000
Denied Decision Requests 0
Total Time 97.796969ms

Standard Benchmark Metrics Skipped or Failed

Bulk Benchmark Results

Metric Value
Total Decrypts 100
Successful Decrypts 100
Failed Decrypts 0
Total Time 354.655887ms
Throughput 281.96 requests/second

TDF3 Benchmark Results:

Metric Value
Total Requests 5000
Successful Requests 5000
Failed Requests 0
Concurrent Requests 50
Total Time 36.924851193s
Average Latency 366.87036ms
Throughput 135.41 requests/second

NANOTDF Benchmark Results:

Metric Value
Total Requests 5000
Successful Requests 5000
Failed Requests 0
Concurrent Requests 50
Total Time 25.58324034s
Average Latency 255.073549ms
Throughput 195.44 requests/second

…c key signing providers

Transition to `PublicKeySigningProvider`, replacing `DefaultSigningProvider` for consistent key-based signing. Introduce `SystemMetadataAssertionProvider` for system metadata handling. Refine assertion provider mechanism in SDK and CLI examples.
Introduce `Verify` method in `Assertion` to validate binding signatures. Update `AssertionProvider` and `Reader` interfaces to support enhanced validation mechanisms. Replace obsolete aggregate hash logic with root signature for assertion binding. Simplify and extend implementation for better clarity and modularity.
…y-based assertions

Transition `PayloadKeyProvider` to `PublicKeySigningProvider` for key-based assertions. Introduce `KeyAssertionProvider` for improved assertion setup. Update related interfaces and examples to use RSA keys for signing and validation. Streamline CLI commands for flexible assertion provider setup.
…ertionRegistry`

Introduce `AssertionRegistry` as a replacement for `AssertionProviderFactory`, streamlining assertion management. Update SDK and examples to use assertion builders and validators. Replace provider terminology with builder-focused nomenclature for consistency. Modify TDF assertion setup and CLI examples for improved clarity and usability.
…igning

Transition from `SignWithProvider` to `Sign` for improved assertion signing. Simplify and streamline signature generation using direct key-based mechanisms. Update related assertion handling and verification methods to align with new design.
…lidation

Streamline assertion handling by replacing `KeyAssertionProvider` with `KeyAssertionBuilder` and `KeyAssertionValidator`. Remove obsolete signing and validation providers for clarity. Update CLI and examples to support private key-based validation registry setup. Refine TDF assertion validation flow for improved security and modularity.
…thAssertionRegistryReader`

Update SDK and examples to align with updated nomenclature. Replace references to `WithAssertionProviderFactory` with `WithAssertionRegistryReader` for consistency. Adjust decrypt command and related SDK logic to reflect this change.
@strantalis
Copy link
Member

I think we should remove the slog statements for now and discuss in the near future how we want to introduce logging within the sdk.

@strantalis strantalis closed this Oct 20, 2025
@strantalis strantalis reopened this Oct 20, 2025
@github-actions github-actions bot added the docs Documentation label Oct 20, 2025
@github-actions
Copy link
Contributor

Benchmark results, click to expand

Benchmark authorization.GetDecisions Results:

Metric Value
Approved Decision Requests 1000
Denied Decision Requests 0
Total Time 132.822216ms

Benchmark authorization.v2.GetMultiResourceDecision Results:

Metric Value
Approved Decision Requests 1000
Denied Decision Requests 0
Total Time 81.097206ms

Standard Benchmark Metrics Skipped or Failed

Bulk Benchmark Results

Metric Value
Total Decrypts 100
Successful Decrypts 100
Failed Decrypts 0
Total Time 358.812351ms
Throughput 278.70 requests/second

TDF3 Benchmark Results:

Metric Value
Total Requests 5000
Successful Requests 5000
Failed Requests 0
Concurrent Requests 50
Total Time 37.312183868s
Average Latency 370.889235ms
Throughput 134.00 requests/second

NANOTDF Benchmark Results:

Metric Value
Total Requests 5000
Successful Requests 5000
Failed Requests 0
Concurrent Requests 50
Total Time 26.331874264s
Average Latency 262.365728ms
Throughput 189.88 requests/second

…tion

Include `statementValue` during `KeyAssertionBinder` setup to simplify public key statement handling. Refactor related tests and examples to use the updated constructor, improving clarity and consistency.
Eliminate `slog` logging throughout assertion-related methods and components for a leaner implementation. Simplify legacy v1 format handling by removing unnecessary logging and unused variables. Update context handling for enhanced clarity and maintainability.
…ication

Improve assertion structure by adding HMAC-based cryptographic binding that combines manifest root signature and assertion content. Update verification logic to validate binding signature and assertion hash for enhanced security and consistency. Refactor example for improved clarity and extensibility.
…dation refinements

Update ADR to include schema registration, refinements in binder/validator usage, enhanced fail-secure validation, and interoperability details for assertion bindings. Simplify examples and improve implementation guidance for clarity.
…rtion usage

Simplify description for encryption/decryption examples with enhanced clarity. Refine assertion examples to highlight efficient operations and implementation details. Update security guidance and configuration options for better readability.
@github-actions
Copy link
Contributor

Benchmark results, click to expand

Benchmark authorization.GetDecisions Results:

Metric Value
Approved Decision Requests 1000
Denied Decision Requests 0
Total Time 175.887779ms

Benchmark authorization.v2.GetMultiResourceDecision Results:

Metric Value
Approved Decision Requests 1000
Denied Decision Requests 0
Total Time 106.079195ms

Standard Benchmark Metrics Skipped or Failed

Bulk Benchmark Results

Metric Value
Total Decrypts 100
Successful Decrypts 100
Failed Decrypts 0
Total Time 362.745908ms
Throughput 275.68 requests/second

TDF3 Benchmark Results:

Metric Value
Total Requests 5000
Successful Requests 5000
Failed Requests 0
Concurrent Requests 50
Total Time 39.837936323s
Average Latency 396.603173ms
Throughput 125.51 requests/second

NANOTDF Benchmark Results:

Metric Value
Total Requests 5000
Successful Requests 5000
Failed Requests 0
Concurrent Requests 50
Total Time 27.493296645s
Average Latency 274.143232ms
Throughput 181.86 requests/second

@github-actions
Copy link
Contributor

Refactor `SystemMetadataAssertionProvider` to standardize V1 schema handling for cross-SDK compatibility and remove legacy V2 schema support. Introduce `DEKAssertionValidator` to enable fallback validation for DEK-signed assertions. Update security checks to enforce cryptographic bindings and enhance tampering detection. Adjust tests and documentation to reflect these changes.
@github-actions
Copy link
Contributor

Benchmark results, click to expand

Benchmark authorization.GetDecisions Results:

Metric Value
Approved Decision Requests 1000
Denied Decision Requests 0
Total Time 178.693103ms

Benchmark authorization.v2.GetMultiResourceDecision Results:

Metric Value
Approved Decision Requests 1000
Denied Decision Requests 0
Total Time 101.317002ms

Standard Benchmark Metrics Skipped or Failed

Bulk Benchmark Results

Metric Value
Total Decrypts 100
Successful Decrypts 100
Failed Decrypts 0
Total Time 370.348794ms
Throughput 270.02 requests/second

TDF3 Benchmark Results:

Metric Value
Total Requests 5000
Successful Requests 5000
Failed Requests 0
Concurrent Requests 50
Total Time 38.969360171s
Average Latency 387.901774ms
Throughput 128.31 requests/second

NANOTDF Benchmark Results:

Metric Value
Total Requests 5000
Successful Requests 5000
Failed Requests 0
Concurrent Requests 50
Total Time 27.22017912s
Average Latency 271.45504ms
Throughput 183.69 requests/second

@github-actions
Copy link
Contributor

…ompatibility

Streamline assertion signature computation by introducing reusable methods `ComputeAggregateHash` and `ComputeAssertionSignature`. Refactor `DEKAssertionValidator`, `KeyAssertionBinder`, and `SystemMetadataAssertionProvider` to adopt standardized signature handling, ensuring cross-SDK interoperability and tampering detection. Update tests and examples to reflect new behavior and schema handling refinements.
@github-actions
Copy link
Contributor

Benchmark results, click to expand

Benchmark authorization.GetDecisions Results:

Metric Value
Approved Decision Requests 1000
Denied Decision Requests 0
Total Time 180.18186ms

Benchmark authorization.v2.GetMultiResourceDecision Results:

Metric Value
Approved Decision Requests 1000
Denied Decision Requests 0
Total Time 113.233753ms

Standard Benchmark Metrics Skipped or Failed

Bulk Benchmark Results

Metric Value
Total Decrypts 100
Successful Decrypts 100
Failed Decrypts 0
Total Time 356.872133ms
Throughput 280.21 requests/second

TDF3 Benchmark Results:

Metric Value
Total Requests 5000
Successful Requests 5000
Failed Requests 0
Concurrent Requests 50
Total Time 39.720357931s
Average Latency 394.732979ms
Throughput 125.88 requests/second

NANOTDF Benchmark Results:

Metric Value
Total Requests 5000
Successful Requests 5000
Failed Requests 0
Concurrent Requests 50
Total Time 27.648491975s
Average Latency 275.662757ms
Throughput 180.84 requests/second

@github-actions
Copy link
Contributor

…gnature handling

Refactor assertion components to incorporate `ShouldUseHexEncoding` for encoding format detection directly from the manifest. Simplify constructors for `KeyAssertionBinder`, `DEKAssertionValidator`, and `SystemMetadataAssertionProvider` by removing redundant `useHex` and `aggregateHash` parameters. Update tests, examples, and validation logic for improved clarity, consistency, and tampering protection.
@github-actions
Copy link
Contributor

Benchmark results, click to expand

Benchmark authorization.GetDecisions Results:

Metric Value
Approved Decision Requests 1000
Denied Decision Requests 0
Total Time 160.267611ms

Benchmark authorization.v2.GetMultiResourceDecision Results:

Metric Value
Approved Decision Requests 1000
Denied Decision Requests 0
Total Time 85.960552ms

Standard Benchmark Metrics Skipped or Failed

Bulk Benchmark Results

Metric Value
Total Decrypts 100
Successful Decrypts 100
Failed Decrypts 0
Total Time 368.521133ms
Throughput 271.35 requests/second

TDF3 Benchmark Results:

Metric Value
Total Requests 5000
Successful Requests 5000
Failed Requests 0
Concurrent Requests 50
Total Time 39.695309563s
Average Latency 395.39723ms
Throughput 125.96 requests/second

NANOTDF Benchmark Results:

Metric Value
Total Requests 5000
Successful Requests 5000
Failed Requests 0
Concurrent Requests 50
Total Time 27.47259857s
Average Latency 273.864662ms
Throughput 182.00 requests/second

@github-actions
Copy link
Contributor

…nifest processing

Unified assertion signature computation by introducing `VerifyAssertionSignatureFormat` helper. Refactored `DEKAssertionValidator`, `SystemMetadataAssertionProvider`, and related test helpers to eliminate redundant logic and use standardized manifest processing. Enhanced tampering detection and cross-SDK compatibility with updated encoding format handling.
@github-actions
Copy link
Contributor

Benchmark results, click to expand

Benchmark authorization.GetDecisions Results:

Metric Value
Approved Decision Requests 1000
Denied Decision Requests 0
Total Time 179.894988ms

Benchmark authorization.v2.GetMultiResourceDecision Results:

Metric Value
Approved Decision Requests 1000
Denied Decision Requests 0
Total Time 104.309253ms

Standard Benchmark Metrics Skipped or Failed

Bulk Benchmark Results

Metric Value
Total Decrypts 100
Successful Decrypts 100
Failed Decrypts 0
Total Time 367.337483ms
Throughput 272.23 requests/second

TDF3 Benchmark Results:

Metric Value
Total Requests 5000
Successful Requests 5000
Failed Requests 0
Concurrent Requests 50
Total Time 39.839844766s
Average Latency 396.461592ms
Throughput 125.50 requests/second

NANOTDF Benchmark Results:

Metric Value
Total Requests 5000
Successful Requests 5000
Failed Requests 0
Concurrent Requests 50
Total Time 27.894417596s
Average Latency 277.904232ms
Throughput 179.25 requests/second

@github-actions
Copy link
Contributor

RunE: encrypt,
Args: cobra.MinimumNArgs(1),
}
encryptCmd.Flags().StringSliceVarP(&dataAttributes, "data-attributes", "a", []string{"https://example.com/attr/attr1/value/value1"}, "space separated list of data attributes")
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

revert

Adds diagrams for assertions

---------

Signed-off-by: Scott Hamrick <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp:examples comp:sdk A software development kit, including library, for client applications and inter-service communicati docs Documentation size/xl

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants