feat(policy)!: namespace Registered Resources

[alkalescent]

Summary

• Adds namespace support to Registered Resources in service layer, DB queries, and migrations
• Updates identifier library for registered resource value namespace handling
• Adds integration and unit tests for namespace-scoped registered resources

Depends on #3110 (proto changes) being merged first.

This is part 2 of splitting #3106 into two PRs.

Test plan

• Integration tests pass for registered resources with namespace
• Unit tests pass for identifier library changes
• Migration applies cleanly
• Existing registered resource functionality unaffected

[gemini-code-assist]

Summary of Changes

Hello, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request introduces fundamental support for namespacing Registered Resources across the system. The primary goal is to allow for greater organization and isolation of resources by associating them with specific namespaces. This involves significant changes to the data model, identifier handling, and service layer logic to ensure that resources can be uniquely identified and managed within their respective namespaces, while also maintaining compatibility with existing, non-namespaced resources.

Highlights

• Namespace Support for Registered Resources: Introduced a namespace_id column to the registered_resources table, enabling Registered Resources to be scoped within specific namespaces. This change allows for resources with the same name to exist in different namespaces.
• Identifier Library Updates: The lib/identifier/registered_resource_value.go file was updated to support a new Fully Qualified Name (FQN) format for Registered Resource Values, which now includes a namespace. The parsing logic was enhanced to handle both the new namespaced FQN format (https://<namespace>/rr/<name>/value/<value>) and the legacy format (https://reg_res/<name>/value/<value>), ensuring backward compatibility.
• Database Schema and Query Modifications: A new migration adds the namespace_id column and updates uniqueness constraints on registered_resources. The previous global unique constraint on name was replaced with two new unique indexes: one for namespaced resources (unique within a namespace) and another for legacy resources (globally unique where namespace_id is NULL). Database queries for creating, getting, and listing Registered Resources and their values were modified to incorporate namespace filtering and return namespace details.
• Same-Namespace Enforcement: Implemented logic to enforce that action attribute values associated with a Registered Resource Value must belong to the same namespace as the Registered Resource itself, preventing cross-namespace data linkage.
• Comprehensive Testing: Extensive integration and unit tests were added and updated to cover the new namespace-scoped functionality, including creation, retrieval, listing, and validation of Registered Resources and their values, as well as testing the FQN parsing and generation for both new and legacy formats.

🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Changelog

• lib/identifier/registered_resource_value.go
  • Added Namespace field to FullyQualifiedRegisteredResourceValue struct.
  • Updated registeredResourceValueFqnRegex to match the new namespaced FQN format.
  • Introduced legacyRegisteredResourceValueFqnRegex for backward compatibility.
  • Created matchFqnParts helper function to extract and validate FQN components.
  • Modified parseRegisteredResourceValueFqn to attempt parsing with the new format first, then fall back to the legacy format.
  • Updated FQN() method to construct FQNs based on whether a namespace is present, using the new format for namespaced resources and the legacy format otherwise.
• lib/identifier/registered_resource_value_test.go
  • Expanded TestRegisteredResourceValueFQN with new test cases for namespaced FQNs, including various naming conventions and case normalization.
  • Updated TestParseRegisteredResourceValueFqn to include new format tests and legacy format tests, and adjusted invalid format tests to reflect the new FQN structure.
  • Modified TestParseRegisteredResourceValueFqn assertions to check the wantNamespace field.
  • Extended TestRegisteredResourceValueRoundTrip with namespaced and legacy test cases, and added assertion for Namespace field.
• service/integration/registered_resources_test.go
  • Modified CreateRegisteredResource test calls to include NamespaceId for namespaced resource creation.
  • Added assertions for Namespace field in created Registered Resources.
  • Renamed Test_CreateRegisteredResource_WithNonUniqueName_Fails to Test_CreateRegisteredResource_WithNonUniqueName_SameNamespace_Fails and updated its logic to test uniqueness within a namespace.
  • Added new test cases for namespace-scoped Registered Resources, including creation with Namespace FQN, creation of same-named resources in different namespaces, getting resources by name with Namespace FQN, and listing resources filtered by Namespace ID or FQN.
  • Added tests to verify namespace information is correctly returned in Registered Resource and Registered Resource Value responses.
  • Introduced Test_LegacyRegisteredResources_NoNamespace_StillAccessible to confirm backward compatibility.
  • Added Test_SameNamespaceEnforcement_DifferentNamespace_Fails and Test_SameNamespaceEnforcement_SameNamespace_Succeeds to validate cross-namespace attribute value linking restrictions.
  • Added helper functions getNamespaceID and getNamespaceFQN.
• service/policy/db/attribute_values.sql.go
  • Added getAttributeValueNamespaceIDs query and its corresponding Go struct and method to retrieve namespace IDs for a batch of attribute values.
• service/policy/db/migrations/20260302000000_add_namespace_to_registered_resources.sql
  • Added a new migration file.
  • Added a nullable namespace_id column to the registered_resources table with a foreign key constraint to attribute_namespaces.
  • Dropped the existing global unique constraint on registered_resources.name.
  • Created a unique index registered_resources_namespace_name_unique for namespaced resources (unique name within namespace_id where namespace_id is not NULL).
  • Created a unique index registered_resources_name_unique for legacy resources (unique name where namespace_id is NULL).
  • Added an index idx_registered_resources_namespace for namespace-scoped queries.
• service/policy/db/models.go
  • Added NamespaceID field of type pgtype.UUID to the RegisteredResource struct.
• service/policy/db/queries/attribute_values.sql
  • Added getAttributeValueNamespaceIDs SQL query to fetch attribute value IDs and their corresponding namespace IDs.
• service/policy/db/queries/registered_resources.sql
  • Modified createRegisteredResource to accept namespace_id or namespace_fqn and return the created resource along with its namespace details.
  • Updated getRegisteredResource to include namespace information in the returned JSON object and allow filtering by namespace_fqn.
  • Adjusted listRegisteredResources to filter by namespace_id or namespace_fqn and include namespace details in the response.
  • Modified getRegisteredResourceValue and listRegisteredResourceValues to include namespace information and resource name in their output.
• service/policy/db/registered_resources.go
  • Added hydrateNamespaceFromInterface helper function to convert raw SQL namespace data into a policy.Namespace struct.
  • Updated CreateRegisteredResource to handle NamespaceId and NamespaceFqn from the request, and to unmarshal the returned namespace data.
  • Modified GetRegisteredResource to support NamespaceFqn in the request identifier and to hydrate the namespace from the query result.
  • Updated ListRegisteredResources to accept NamespaceId and NamespaceFqn for filtering and to hydrate namespaces for each listed resource.
  • Modified GetRegisteredResourceValue and ListRegisteredResourceValues to hydrate namespace information for the associated Registered Resource.
  • Implemented same-namespace enforcement in createRegisteredResourceActionAttributeValues by fetching the namespace of the Registered Resource Value and validating that all linked attribute values belong to the same namespace using the new getAttributeValueNamespaceIDs query.
• service/policy/registeredresources/registered_resources_test.go
  • Updated TestCreateRegisteredResource_Valid_Succeeds to include test cases for creating resources with NamespaceId and NamespaceFqn.
  • Modified TestCreateRegisteredResource_Invalid_Fails to include validation errors for missing namespace, invalid Namespace ID, and invalid Namespace FQN, and updated existing name validation tests to include a valid NamespaceId.

Activity

• This pull request is part 2 of splitting a larger change (feat(policy)!: namespace Registered Resources #3106) into smaller, more manageable PRs.
• It depends on proto changes from feat(policy)!: add namespace field to RegisteredResource proto #3110 being merged first, indicating a structured approach to feature development.
• The author has outlined a comprehensive test plan covering integration tests for namespaced registered resources, unit tests for identifier library changes, migration application, and ensuring existing functionality remains unaffected.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

---

Namespaces now define, Resources neatly align, No more global strife.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request introduces namespace support for Registered Resources across the service layer, database queries, and migrations. It updates the identifier library to handle new namespaced FQN formats for registered resource values, while maintaining backward compatibility with the legacy format. The changes are well-supported by new integration and unit tests.

My review identifies a critical issue that could lead to a panic when handling legacy resources, and a medium-severity validation gap for namespaces in FQNs. I've provided suggestions to address both. Overall, the changes are well-structured and the test coverage is good.

[gemini-code-assist]

Code Review

This pull request introduces namespace support for Registered Resources, which is a significant and breaking change. The modifications span across the database layer with new migrations and updated queries, the identifier library for FQN parsing, and includes a comprehensive set of new integration and unit tests. Overall, the changes are well-structured. However, I've identified a few issues related to correctness and performance that should be addressed. Specifically, there's a bug in FQN generation that could break round-trip serialization, missing validation for namespace components in FQNs, and an inefficient query being used for namespace enforcement.

_{Note: Security Review did not run due to the size of the PR.}

[github-actions]

X-Test Failure Report

✅ go-v0.9.0
✅ java-v0.9.0
✅ js-v0.9.0
opentdfplatformCLOM7V.dockerbuild

[github-actions]

X-Test Failure Report

✅ go-v0.9.0
✅ java-v0.9.0
✅ js-v0.9.0
opentdfplatform5QJ6BI.dockerbuild

[github-actions]

X-Test Failure Report

✅ go-v0.9.0
✅ java-v0.9.0
✅ js-v0.9.0
opentdfplatformSXTH1G.dockerbuild

[github-actions]

Benchmark results, click to expand

Benchmark authorization.GetDecisions Results:

Metric	Value
Approved Decision Requests	1000
Denied Decision Requests	0
Total Time	212.011467ms

Benchmark authorization.v2.GetMultiResourceDecision Results:

Metric	Value
Approved Decision Requests	1000
Denied Decision Requests	0
Total Time	104.71423ms

Benchmark Statistics

Name	№ Requests	Avg Duration	Min Duration	Max Duration

Bulk Benchmark Results

Metric	Value
Total Decrypts	100
Successful Decrypts	100
Failed Decrypts	0
Total Time	400.338127ms
Throughput	249.79 requests/second

TDF3 Benchmark Results:

Metric	Value
Total Requests	5000
Successful Requests	5000
Failed Requests	0
Concurrent Requests	50
Total Time	44.789471237s
Average Latency	444.892892ms
Throughput	111.63 requests/second

[github-actions]

X-Test Results

✅ go-pull-3111
✅ go-v0.9.0
✅ java-pull-3111
✅ java-v0.9.0
✅ js-pull-3111
✅ js-v0.9.0
cukes-report
test-cases-mapping-report
bats-test-results
opentdfplatformWACPHK.dockerbuild

[github-actions]

X-Test Failure Report

[github-actions]

Benchmark results, click to expand

Benchmark authorization.GetDecisions Results:

Metric	Value
Approved Decision Requests	1000
Denied Decision Requests	0
Total Time	192.027298ms

Benchmark authorization.v2.GetMultiResourceDecision Results:

Metric	Value
Approved Decision Requests	1000
Denied Decision Requests	0
Total Time	95.955256ms

Benchmark Statistics

Name	№ Requests	Avg Duration	Min Duration	Max Duration

Bulk Benchmark Results

Metric	Value
Total Decrypts	100
Successful Decrypts	100
Failed Decrypts	0
Total Time	390.431935ms
Throughput	256.13 requests/second

TDF3 Benchmark Results:

Metric	Value
Total Requests	5000
Successful Requests	5000
Failed Requests	0
Concurrent Requests	50
Total Time	40.003323381s
Average Latency	398.584545ms
Throughput	124.99 requests/second

[github-actions]

Benchmark results, click to expand

Benchmark authorization.GetDecisions Results:

Metric	Value
Approved Decision Requests	1000
Denied Decision Requests	0
Total Time	187.654121ms

Benchmark authorization.v2.GetMultiResourceDecision Results:

Metric	Value
Approved Decision Requests	1000
Denied Decision Requests	0
Total Time	97.895819ms

Benchmark Statistics

Name	№ Requests	Avg Duration	Min Duration	Max Duration

Bulk Benchmark Results

Metric	Value
Total Decrypts	100
Successful Decrypts	100
Failed Decrypts	0
Total Time	360.038595ms
Throughput	277.75 requests/second

TDF3 Benchmark Results:

Metric	Value
Total Requests	5000
Successful Requests	5000
Failed Requests	0
Concurrent Requests	50
Total Time	39.194702883s
Average Latency	389.847466ms
Throughput	127.57 requests/second

[github-actions]

X-Test Results

✅ go-pull-3111
✅ go-v0.9.0
✅ java-pull-3111
✅ java-v0.9.0
✅ js-pull-3111
✅ js-v0.9.0
test-cases-mapping-report
bats-test-results
cukes-report
opentdfplatformKMO32W.dockerbuild

[github-actions]

Benchmark results, click to expand

Benchmark authorization.GetDecisions Results:

Metric	Value
Approved Decision Requests	1000
Denied Decision Requests	0
Total Time	198.052205ms

Benchmark authorization.v2.GetMultiResourceDecision Results:

Metric	Value
Approved Decision Requests	1000
Denied Decision Requests	0
Total Time	97.302659ms

Benchmark Statistics

Name	№ Requests	Avg Duration	Min Duration	Max Duration

Bulk Benchmark Results

Metric	Value
Total Decrypts	100
Successful Decrypts	100
Failed Decrypts	0
Total Time	374.409352ms
Throughput	267.09 requests/second

TDF3 Benchmark Results:

Metric	Value
Total Requests	5000
Successful Requests	5000
Failed Requests	0
Concurrent Requests	50
Total Time	39.592099242s
Average Latency	393.918796ms
Throughput	126.29 requests/second

[github-actions]

X-Test Results

✅ go-pull-3111
✅ go-v0.9.0
✅ java-pull-3111
✅ java-v0.9.0
✅ js-pull-3111
✅ js-v0.9.0
test-cases-mapping-report
bats-test-results
cukes-report
opentdfplatformLMT172.dockerbuild

[alkalescent]

Last question (I think): this migration adds the columns/indices for a namespaces column on the registered resource table. Will migrating existing registered resources be fully a manual script instead of something we do in SQL to try to infer based on assigned action-attribute-values (where attribute values are namespaced) which namespace a RR should belong to?

I'm leaning towards handling existing RRs through a manual script via otdfctl. There's a branch with the script
in progress here: opentdf/otdfctl@DSPX-2496-ns-rr...DSPX-2496-migration-script

The main reason is by convention and best practice we've kept the schema migrations separate from data migrations since data migrations are often more difficult to revert. A script allows users to migrate in a user-friendly way (prompting, tables, etc) on their own timeline (since we did this in a backwards-compatible way).

Inferring namespace from action-attribute-values also has a bit of complexity in cases. For example, we ignore RRs that have not only no AAVs but also AAVs with mixed namepaces?

[jakedoublev]

Last question (I think): this migration adds the columns/indices for a namespaces column on the registered resource table. Will migrating existing registered resources be fully a manual script instead of something we do in SQL to try to infer based on assigned action-attribute-values (where attribute values are namespaced) which namespace a RR should belong to?

I'm leaning towards handling existing RRs through a manual script via otdfctl. There's a branch with the script in progress here: opentdf/otdfctl@DSPX-2496-ns-rr...DSPX-2496-migration-script

The main reason is by convention and best practice we've kept the schema migrations separate from data migrations since data migrations are often more difficult to revert. A script allows users to migrate in a user-friendly way (prompting, tables, etc) on their own timeline (since we did this in a backwards-compatible way).

Inferring namespace from action-attribute-values also has a bit of complexity in cases. For example, we ignore RRs that have not only no AAVs but also AAVs with mixed namepaces?

This makes sense. Thanks for outlining your thoughts.

Invalidated by push of 4f65a8f

[github-actions]

Benchmark results, click to expand

Benchmark authorization.GetDecisions Results:

Metric	Value
Approved Decision Requests	1000
Denied Decision Requests	0
Total Time	204.424711ms

Benchmark authorization.v2.GetMultiResourceDecision Results:

Metric	Value
Approved Decision Requests	1000
Denied Decision Requests	0
Total Time	102.029821ms

Benchmark Statistics

Name	№ Requests	Avg Duration	Min Duration	Max Duration

Bulk Benchmark Results

Metric	Value
Total Decrypts	100
Successful Decrypts	100
Failed Decrypts	0
Total Time	379.4402ms
Throughput	263.55 requests/second

TDF3 Benchmark Results:

Metric	Value
Total Requests	5000
Successful Requests	5000
Failed Requests	0
Concurrent Requests	50
Total Time	40.832519639s
Average Latency	406.646218ms
Throughput	122.45 requests/second

[github-actions]

X-Test Results

✅ go-pull-3111
✅ go-v0.9.0
✅ java-pull-3111
✅ java-v0.9.0
✅ js-v0.9.0
✅ js-pull-3111
test-cases-mapping-report
bats-test-results
cukes-report
opentdfplatformDLQ6K7.dockerbuild

[github-actions]

Benchmark results, click to expand

Benchmark authorization.GetDecisions Results:

Metric	Value
Approved Decision Requests	1000
Denied Decision Requests	0
Total Time	190.584603ms

Benchmark authorization.v2.GetMultiResourceDecision Results:

Metric	Value
Approved Decision Requests	1000
Denied Decision Requests	0
Total Time	104.796455ms

Benchmark Statistics

Name	№ Requests	Avg Duration	Min Duration	Max Duration

Bulk Benchmark Results

Metric	Value
Total Decrypts	100
Successful Decrypts	100
Failed Decrypts	0
Total Time	380.027757ms
Throughput	263.14 requests/second

TDF3 Benchmark Results:

Metric	Value
Total Requests	5000
Successful Requests	5000
Failed Requests	0
Concurrent Requests	50
Total Time	40.016046916s
Average Latency	397.86008ms
Throughput	124.95 requests/second

[github-actions]

X-Test Failure Report

✅ go-v0.9.0
✅ java-pull-3111
✅ java-v0.9.0
✅ js-pull-3111
✅ js-v0.9.0
test-cases-mapping-report
bats-test-results
cukes-report
opentdfplatformNPRZWD.dockerbuild

[github-actions]

X-Test Results

✅ go-pull-3111
✅ go-v0.9.0
✅ java-pull-3111
✅ java-v0.9.0
✅ js-pull-3111
✅ js-v0.9.0
test-cases-mapping-report
bats-test-results
cukes-report
opentdfplatformNPRZWD.dockerbuild

[elizabethhealy]

addition of namespaceid lgtm!