adr(docs): Proto Template ADR

[alkalescent]

Proposed Changes

Checklist

• I have added or updated unit tests
• I have added or updated integration tests (if appropriate)
• I have added or updated documentation

Testing Instructions

[gemini-code-assist]

Summary of Changes

Hello @alkalescent, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request introduces an Architectural Decision Record (ADR) that outlines a strategic approach to eliminate value duplication within Protobuf definitions. The document proposes and justifies the adoption of 'Proto Template Generation with Preprocessing' as the preferred method. This solution aims to enhance the maintainability, consistency, and developer experience of Protobuf schemas by centralizing common validation rules and constants, thereby reducing the risk of errors and simplifying future updates.

Highlights

• Problem Statement: The pull request addresses the issue of extensive value duplication (e.g., regex patterns, validation rules) within Protobuf definitions, which leads to maintenance drift, inconsistency, reduced readability, and error-prone updates.
• Proposed Solution: The chosen solution is 'Proto Template Generation with Preprocessing'. This involves using .proto.template files with variable substitution to generate standard .proto files, centralizing common values.
• Decision Drivers: Key drivers for this decision include improving maintainability, ensuring consistency, enhancing developer experience, managing build complexity, maintaining backward compatibility, and integrating with existing toolchains.
• Considered Options: Three options were considered: Intermediate Message Abstraction, Proto Template Generation with Preprocessing, and External Constants via Imports. The template generation approach was selected for its balance of benefits.
• Consequences: The chosen approach offers benefits such as single-source definition of rules, standard protobuf output, and organized constants. However, it introduces a new preprocessing step and requires developers to learn template syntax.
• Implementation Plan: The implementation will proceed in phases, starting with setting up the preprocessing pipeline, creating a constants library, incrementally converting existing proto files, and developing IDE support.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

---

Proto templates rise, Duplication now will cease, Code generation.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request introduces an Architecture Decision Record (ADR) for using a template-based approach to generate Protobuf files, aiming to reduce value duplication. The ADR is well-structured and detailed. My review includes suggestions to correct a Go code example, clarify a regex pattern, improve a Makefile example, and use more precise terminology for a diagram. I've also noted that one of the considered options lacks a detailed pros and cons analysis, which would make the ADR more complete.

[github-actions]

Benchmark results, click to expand

Benchmark authorization.GetDecisions Results:

Metric	Value
Approved Decision Requests	1000
Denied Decision Requests	0
Total Time	172.436759ms

Benchmark authorization.v2.GetMultiResourceDecision Results:

Metric	Value
Approved Decision Requests	1000
Denied Decision Requests	0
Total Time	95.953775ms

Standard Benchmark Metrics Skipped or Failed

Bulk Benchmark Results

Metric	Value
Total Decrypts	100
Successful Decrypts	100
Failed Decrypts	0
Total Time	358.159118ms
Throughput	279.21 requests/second

TDF3 Benchmark Results:

Metric	Value
Total Requests	5000
Successful Requests	5000
Failed Requests	0
Concurrent Requests	50
Total Time	36.7437447s
Average Latency	365.234761ms
Throughput	136.08 requests/second

NANOTDF Benchmark Results:

Metric	Value
Total Requests	5000
Successful Requests	5000
Failed Requests	0
Concurrent Requests	50
Total Time	25.300715621s
Average Latency	252.223462ms
Throughput	197.62 requests/second

[github-actions]

Benchmark results, click to expand

Benchmark authorization.GetDecisions Results:

Metric	Value
Approved Decision Requests	1000
Denied Decision Requests	0
Total Time	172.017143ms

Benchmark authorization.v2.GetMultiResourceDecision Results:

Metric	Value
Approved Decision Requests	1000
Denied Decision Requests	0
Total Time	107.391051ms

Standard Benchmark Metrics Skipped or Failed

Bulk Benchmark Results

Metric	Value
Total Decrypts	100
Successful Decrypts	100
Failed Decrypts	0
Total Time	359.218437ms
Throughput	278.38 requests/second

TDF3 Benchmark Results:

Metric	Value
Total Requests	5000
Successful Requests	5000
Failed Requests	0
Concurrent Requests	50
Total Time	37.557396144s
Average Latency	374.156469ms
Throughput	133.13 requests/second

NANOTDF Benchmark Results:

Metric	Value
Total Requests	5000
Successful Requests	5000
Failed Requests	0
Concurrent Requests	50
Total Time	26.165414196s
Average Latency	260.611166ms
Throughput	191.09 requests/second

[github-actions]

Benchmark results, click to expand

Benchmark authorization.GetDecisions Results:

Metric	Value
Approved Decision Requests	1000
Denied Decision Requests	0
Total Time	181.110774ms

Benchmark authorization.v2.GetMultiResourceDecision Results:

Metric	Value
Approved Decision Requests	1000
Denied Decision Requests	0
Total Time	98.257209ms

Standard Benchmark Metrics Skipped or Failed

Bulk Benchmark Results

Metric	Value
Total Decrypts	100
Successful Decrypts	100
Failed Decrypts	0
Total Time	340.95777ms
Throughput	293.29 requests/second

TDF3 Benchmark Results:

Metric	Value
Total Requests	5000
Successful Requests	5000
Failed Requests	0
Concurrent Requests	50
Total Time	37.873077797s
Average Latency	376.209029ms
Throughput	132.02 requests/second

NANOTDF Benchmark Results:

Metric	Value
Total Requests	5000
Successful Requests	5000
Failed Requests	0
Concurrent Requests	50
Total Time	25.701299652s
Average Latency	256.26425ms
Throughput	194.54 requests/second

[github-actions]

Benchmark results, click to expand

Benchmark authorization.GetDecisions Results:

Metric	Value
Approved Decision Requests	1000
Denied Decision Requests	0
Total Time	148.181585ms

Benchmark authorization.v2.GetMultiResourceDecision Results:

Metric	Value
Approved Decision Requests	1000
Denied Decision Requests	0
Total Time	76.277725ms

Standard Benchmark Metrics Skipped or Failed

Bulk Benchmark Results

Metric	Value
Total Decrypts	100
Successful Decrypts	100
Failed Decrypts	0
Total Time	358.314781ms
Throughput	279.08 requests/second

TDF3 Benchmark Results:

Metric	Value
Total Requests	5000
Successful Requests	5000
Failed Requests	0
Concurrent Requests	50
Total Time	36.933190592s
Average Latency	367.453731ms
Throughput	135.38 requests/second

NANOTDF Benchmark Results:

Metric	Value
Total Requests	5000
Successful Requests	5000
Failed Requests	0
Concurrent Requests	50
Total Time	25.282883926s
Average Latency	251.609205ms
Throughput	197.76 requests/second

[github-actions]

Benchmark results, click to expand

Benchmark authorization.GetDecisions Results:

Metric	Value
Approved Decision Requests	1000
Denied Decision Requests	0
Total Time	175.27563ms

Benchmark authorization.v2.GetMultiResourceDecision Results:

Metric	Value
Approved Decision Requests	1000
Denied Decision Requests	0
Total Time	97.938846ms

Standard Benchmark Metrics Skipped or Failed

Bulk Benchmark Results

Metric	Value
Total Decrypts	100
Successful Decrypts	100
Failed Decrypts	0
Total Time	360.619006ms
Throughput	277.30 requests/second

TDF3 Benchmark Results:

Metric	Value
Total Requests	5000
Successful Requests	5000
Failed Requests	0
Concurrent Requests	50
Total Time	38.230315814s
Average Latency	380.463964ms
Throughput	130.79 requests/second

NANOTDF Benchmark Results:

Metric	Value
Total Requests	5000
Successful Requests	5000
Failed Requests	0
Concurrent Requests	50
Total Time	26.899220908s
Average Latency	268.201484ms
Throughput	185.88 requests/second

[github-actions]

Benchmark results, click to expand

Benchmark authorization.GetDecisions Results:

Metric	Value
Approved Decision Requests	1000
Denied Decision Requests	0
Total Time	175.857238ms

Benchmark authorization.v2.GetMultiResourceDecision Results:

Metric	Value
Approved Decision Requests	1000
Denied Decision Requests	0
Total Time	101.896518ms

Standard Benchmark Metrics Skipped or Failed

Bulk Benchmark Results

Metric	Value
Total Decrypts	100
Successful Decrypts	100
Failed Decrypts	0
Total Time	371.093481ms
Throughput	269.47 requests/second

TDF3 Benchmark Results:

Metric	Value
Total Requests	5000
Successful Requests	5000
Failed Requests	0
Concurrent Requests	50
Total Time	37.257013027s
Average Latency	370.928729ms
Throughput	134.20 requests/second

NANOTDF Benchmark Results:

Metric	Value
Total Requests	5000
Successful Requests	5000
Failed Requests	0
Concurrent Requests	50
Total Time	25.729000701s
Average Latency	256.561464ms
Throughput	194.33 requests/second

[jakedoublev]

Can you please capture an option 3 (common objects like IdFqnIdentifier), which maintains the oneof restriction/validation and is already in use in some places? https://github.com/opentdf/platform/blob/main/service/common/common.proto#L8-L29 This seems like it's probably the cheapest and most maintainable solution?

[jakedoublev]

Can you please mention how an external publish of the protos to a proto registry like Buf would fit into this? I assume after protos are generated in CI?

[jakedoublev]

I think one note here is if you update a proto file and want to consume the gencode locally, you'll also need to have run the preprocessing script. Today the generated Go code is checked in with git, so the validation would need to check that all gencode has no difference between a reprocess from source to gencode and that checked in.

Also, today, other language SDKs or clients can pull *.proto files directly and generate their own connect/gRPC clients, which would go away with this plan if builds/pre-processing are strictly happening in CI.