feat(policy): add soft-delete/deactivation to namespaces, attribute definitions, attribute values #96 #108 #191

jakedoublev · 2024-02-13T15:28:06Z

This work encompasses the following (including multiple breaking changes):

Tables namespaces, attribute_definitions, attribute_values updated via migration to add active boolean state
cascading deactivation from namespace -> attr -> values (in DB implementation via SQL trigger function provided in migration up/down)
integration tests:
- cascading behavior ns -> attr -> val
- integration tests proving no deactivation bubbling up behavior val -> attr -> ns
protos:
- updated to provide a state enum back on all 3 resources (with helpful comments about defaults)
- new example grpcurl requests/responses with these updates
- all three LIST rpc's filterable by state as a common Message type (including an ANY enum option that returns both active TRUE and FALSE rows) and defaulting to ACTIVE if not specified
preservation of DB delete functionality/tests which will be exposed in newly separate rpc's

This PR does not include:

Unsafe RPCs for actual deletion: Policy API: Enable admins to perform unsafe mutations on data in their platform #115
Unsafe RPCs for dangerous mutations (same issue)
Prevention of update mutation of INACTIVE namespaces/attributes/attributeValues
Prevention of creation of new attributes/values on a prior created then deactivated namespace
Provision of parent or child state in GET responses beyond that of the resource requested (i.e. namespace & value state are not given in a GET for an attribute, even though the attribute's state is given)

…t new state column value

…tate in sql queries

… rename delete to deactivate which is its current purpose

…e sure to log db-interpreted state and not the proto enum value which can be different than the db (i.e. 'any' which queries both 'active' and 'inactive')

dmihalcik-virtru

Some minor suggestions and comments you can take or leave

proto/common/common.proto

proto/namespaces/namespaces.proto

internal/db/attributes.go

migrations/20240212000000_add_active_state_enums.sql

internal/db/attributes.go

…clarity

… proto messages but keep database column for active state a boolean

…o fixtures

proto/attributes/attributes.proto

…of status enum

… as zero value and real value of boolean fields in some policy messages (see issue: golang/protobuf#438)

…bool variable and not the wrapper type

jakedoublev · 2024-02-16T22:03:36Z

Introduced a Google-maintained wrapper for boolean proto fields due to conflicts with unset Go zero values for booleans (false), the protos generating with the JSON omitempty flag, and truly false field values. See the issue and recommended approach in the golang/protobuf repo.

🤖 I have created a release *beep* *boop* --- ## [0.1.0](sdk-v0.1.0...sdk/v0.1.0) (2024-04-22) ### Features * add structured schema policy config ([#51](#51)) ([8a6b876](8a6b876)) * **auth:** add authorization via casbin ([#417](#417)) ([292f2bd](292f2bd)) * in-process service to service communication ([#311](#311)) ([ec5eb76](ec5eb76)) * **kas:** support HSM and standard crypto ([#497](#497)) ([f0cbe03](f0cbe03)) * key access server assignments ([#111](#111)) ([a48d686](a48d686)), closes [#117](#117) * key access server registry impl ([#66](#66)) ([cf6b3c6](cf6b3c6)) * **namespaces CRUD:** protos, generated SDK, db interactivity for namespaces table ([#54](#54)) ([b3f32b1](b3f32b1)) * **PLAT-3112:** Initial consumption of ec_key_pair functions by nanotdf ([#586](#586)) ([5e2cba0](5e2cba0)) * **policy:** add FQN pivot table ([#208](#208)) ([abb734c](abb734c)) * **policy:** add soft-delete/deactivation to namespaces, attribute definitions, attribute values [#96](#96) [#108](#108) ([#191](#191)) ([02e92a6](02e92a6)) * **resourcemapping:** resource mapping implementation ([#83](#83)) ([c144db1](c144db1)) * **sdk:** BACK-1966 get auth wired up to SDK using `Options` ([#271](#271)) ([f1bacab](f1bacab)) * **sdk:** BACK-1966 implement fetching a DPoP token ([#45](#45)) ([dbd3cf9](dbd3cf9)) * **sdk:** BACK-1966 make the unwrapper retrieve public keys as well ([#260](#260)) ([7d051a1](7d051a1)) * **sdk:** BACK-1966 pull rewrap into auth config ([#252](#252)) ([84017aa](84017aa)) * **sdk:** Include auth token in grpc ([#367](#367)) ([75cb5cd](75cb5cd)) * **sdk:** normalize token exchange ([#546](#546)) ([9059dff](9059dff)) * **sdk:** Pass dpop key through to `rewrap` ([#435](#435)) ([2d283de](2d283de)) * **sdk:** read `expires_in` from token response and use it to refresh access tokens ([#445](#445)) ([8ecbe79](8ecbe79)) * **sdk:** sdk stub ([#10](#10)) ([8dfca6a](8dfca6a)) * **sdk:** take a function so that callers can use this the way that they want ([#340](#340)) ([72059cb](72059cb)) * **subject-mappings:** refactor to meet db schema ([#59](#59)) ([59a073b](59a073b)) * **tdf:** implement tdf3 encrypt and decrypt ([#73](#73)) ([9d0e0a0](9d0e0a0)) * **tdf:** sdk interface changes ([#123](#123)) ([2aa2422](2aa2422)) * **tdf:** sdk interface cleanup ([#201](#201)) ([6f7d815](6f7d815)) * **tdf:** TDFOption varargs interface ([#235](#235)) ([b3fb720](b3fb720)) ### Bug Fixes * **archive:** remove 10gb zip file test ([#373](#373)) ([6548f55](6548f55)) * attribute missing rpc method for listing attribute values ([#69](#69)) ([1b3a831](1b3a831)) * **attribute value:** fixes attribute value crud ([#86](#86)) ([568df9c](568df9c)) * **issue 90:** remove duplicate attribute_id from attribute value create/update, and consumes schema setup changes in namespaces that were introduced for integration testing ([#100](#100)) ([e0f6d07](e0f6d07)) * **issue-124:** SDK kas registry import name mismatch ([#125](#125)) ([112638b](112638b)), closes [#124](#124) * **proto/acre:** fix resource encoding service typo ([#30](#30)) ([fe709d2](fe709d2)) * remove padding when b64 encoding ([#437](#437)) ([d40e94a](d40e94a)) * SDK Quickstart ([#628](#628)) ([f27ab98](f27ab98)) * **sdk:** change unwrapper creation ([#346](#346)) ([9206435](9206435)) * **sdk:** double bearer token in auth config ([#350](#350)) ([1bf4699](1bf4699)) * **sdk:** fixes Manifests JSONs with OIDC ([#140](#140)) ([a4b6937](a4b6937)) * **sdk:** handle err ([#548](#548)) ([ebabb6c](ebabb6c)) * **sdk:** make KasInfo fields public ([#320](#320)) ([9a70498](9a70498)) * **sdk:** shutdown conn ([#352](#352)) ([3def038](3def038)) * **sdk:** temporarily move unwrapper creation into options func. ([#309](#309)) ([b34c2fe](b34c2fe)) * **sdk:** use the dialoptions even with no client credentials ([#400](#400)) ([a7f1908](a7f1908)) * **security:** add a new encryption keypair different from dpop keypair ([#461](#461)) ([7deb51e](7deb51e)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please). Co-authored-by: opentdf-automation[bot] <149537512+opentdf-automation[bot]@users.noreply.github.com>

🤖 I have created a release *beep* *boop* --- ## [0.1.0](opentdf/platform@sdk-v0.1.0...sdk/v0.1.0) (2024-04-22) ### Features * add structured schema policy config ([#51](opentdf/platform#51)) ([8a6b876](opentdf/platform@8a6b876)) * **auth:** add authorization via casbin ([#417](opentdf/platform#417)) ([292f2bd](opentdf/platform@292f2bd)) * in-process service to service communication ([#311](opentdf/platform#311)) ([ec5eb76](opentdf/platform@ec5eb76)) * **kas:** support HSM and standard crypto ([#497](opentdf/platform#497)) ([f0cbe03](opentdf/platform@f0cbe03)) * key access server assignments ([#111](opentdf/platform#111)) ([a48d686](opentdf/platform@a48d686)), closes [#117](opentdf/platform#117) * key access server registry impl ([#66](opentdf/platform#66)) ([cf6b3c6](opentdf/platform@cf6b3c6)) * **namespaces CRUD:** protos, generated SDK, db interactivity for namespaces table ([#54](opentdf/platform#54)) ([b3f32b1](opentdf/platform@b3f32b1)) * **PLAT-3112:** Initial consumption of ec_key_pair functions by nanotdf ([#586](opentdf/platform#586)) ([5e2cba0](opentdf/platform@5e2cba0)) * **policy:** add FQN pivot table ([#208](opentdf/platform#208)) ([abb734c](opentdf/platform@abb734c)) * **policy:** add soft-delete/deactivation to namespaces, attribute definitions, attribute values [#96](opentdf/platform#96) [#108](opentdf/platform#108) ([#191](opentdf/platform#191)) ([02e92a6](opentdf/platform@02e92a6)) * **resourcemapping:** resource mapping implementation ([#83](opentdf/platform#83)) ([c144db1](opentdf/platform@c144db1)) * **sdk:** BACK-1966 get auth wired up to SDK using `Options` ([#271](opentdf/platform#271)) ([f1bacab](opentdf/platform@f1bacab)) * **sdk:** BACK-1966 implement fetching a DPoP token ([#45](opentdf/platform#45)) ([dbd3cf9](opentdf/platform@dbd3cf9)) * **sdk:** BACK-1966 make the unwrapper retrieve public keys as well ([#260](opentdf/platform#260)) ([7d051a1](opentdf/platform@7d051a1)) * **sdk:** BACK-1966 pull rewrap into auth config ([#252](opentdf/platform#252)) ([84017aa](opentdf/platform@84017aa)) * **sdk:** Include auth token in grpc ([#367](opentdf/platform#367)) ([75cb5cd](opentdf/platform@75cb5cd)) * **sdk:** normalize token exchange ([#546](opentdf/platform#546)) ([9059dff](opentdf/platform@9059dff)) * **sdk:** Pass dpop key through to `rewrap` ([#435](opentdf/platform#435)) ([2d283de](opentdf/platform@2d283de)) * **sdk:** read `expires_in` from token response and use it to refresh access tokens ([#445](opentdf/platform#445)) ([8ecbe79](opentdf/platform@8ecbe79)) * **sdk:** sdk stub ([#10](opentdf/platform#10)) ([8dfca6a](opentdf/platform@8dfca6a)) * **sdk:** take a function so that callers can use this the way that they want ([#340](opentdf/platform#340)) ([72059cb](opentdf/platform@72059cb)) * **subject-mappings:** refactor to meet db schema ([#59](opentdf/platform#59)) ([59a073b](opentdf/platform@59a073b)) * **tdf:** implement tdf3 encrypt and decrypt ([#73](opentdf/platform#73)) ([9d0e0a0](opentdf/platform@9d0e0a0)) * **tdf:** sdk interface changes ([#123](opentdf/platform#123)) ([2aa2422](opentdf/platform@2aa2422)) * **tdf:** sdk interface cleanup ([#201](opentdf/platform#201)) ([6f7d815](opentdf/platform@6f7d815)) * **tdf:** TDFOption varargs interface ([#235](opentdf/platform#235)) ([b3fb720](opentdf/platform@b3fb720)) ### Bug Fixes * **archive:** remove 10gb zip file test ([#373](opentdf/platform#373)) ([6548f55](opentdf/platform@6548f55)) * attribute missing rpc method for listing attribute values ([#69](opentdf/platform#69)) ([1b3a831](opentdf/platform@1b3a831)) * **attribute value:** fixes attribute value crud ([#86](opentdf/platform#86)) ([568df9c](opentdf/platform@568df9c)) * **issue 90:** remove duplicate attribute_id from attribute value create/update, and consumes schema setup changes in namespaces that were introduced for integration testing ([#100](opentdf/platform#100)) ([e0f6d07](opentdf/platform@e0f6d07)) * **issue-124:** SDK kas registry import name mismatch ([#125](opentdf/platform#125)) ([112638b](opentdf/platform@112638b)), closes [#124](opentdf/platform#124) * **proto/acre:** fix resource encoding service typo ([#30](opentdf/platform#30)) ([fe709d2](opentdf/platform@fe709d2)) * remove padding when b64 encoding ([#437](opentdf/platform#437)) ([d40e94a](opentdf/platform@d40e94a)) * SDK Quickstart ([#628](opentdf/platform#628)) ([f27ab98](opentdf/platform@f27ab98)) * **sdk:** change unwrapper creation ([#346](opentdf/platform#346)) ([9206435](opentdf/platform@9206435)) * **sdk:** double bearer token in auth config ([#350](opentdf/platform#350)) ([1bf4699](opentdf/platform@1bf4699)) * **sdk:** fixes Manifests JSONs with OIDC ([#140](opentdf/platform#140)) ([a4b6937](opentdf/platform@a4b6937)) * **sdk:** handle err ([#548](opentdf/platform#548)) ([ebabb6c](opentdf/platform@ebabb6c)) * **sdk:** make KasInfo fields public ([#320](opentdf/platform#320)) ([9a70498](opentdf/platform@9a70498)) * **sdk:** shutdown conn ([#352](opentdf/platform#352)) ([3def038](opentdf/platform@3def038)) * **sdk:** temporarily move unwrapper creation into options func. ([#309](opentdf/platform#309)) ([b34c2fe](opentdf/platform@b34c2fe)) * **sdk:** use the dialoptions even with no client credentials ([#400](opentdf/platform#400)) ([a7f1908](opentdf/platform@a7f1908)) * **security:** add a new encryption keypair different from dpop keypair ([#461](opentdf/platform#461)) ([7deb51e](opentdf/platform@7deb51e)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please). Co-authored-by: opentdf-automation[bot] <149537512+opentdf-automation[bot]@users.noreply.github.com>

jakedoublev added 14 commits February 12, 2024 13:54

add state to ERD and add new migration

e840509

add real uuids to fixtures for readability

0f36899

fix namespaces GET and LIST queries to only return id and name but no…

971d063

…t new state column value

add indexes on state column

9b4f973

Merge branch 'main' into issue-96/state

0e7de50

add state to namespaces fixtures

6b2e38c

add function and trigger definitions to new migration sql

60ef44a

add soft-delete to namespaces, then support list filtering based on s…

a866acc

…tate in sql queries

prove out triggers working as expected in update cascade from namespaces

b6e71ef

demonstrate it can still be found when inactive

312cd40

add state enum to namespaces, attributes, attribute values protos and…

091bd29

… rename delete to deactivate which is its current purpose

generated sdk code from protos

b762b6e

add support for state in the db functions that message handlers call

77a2d4e

add debug logs, clean up naming from soft-delete to deactivation, mak…

c9f3a76

…e sure to log db-interpreted state and not the proto enum value which can be different than the db (i.e. 'any' which queries both 'active' and 'inactive')

jakedoublev changed the title ~~feat(issue-96): add soft-delete to namespaces, attribute definitions, attribute values~~ feat(issue-96): add soft-delete/"deactivation" to namespaces, attribute definitions, attribute values Feb 14, 2024

cleanup

faf14f5

jakedoublev marked this pull request as ready for review February 15, 2024 15:02

jakedoublev requested a review from a team as a code owner February 15, 2024 15:02

This was linked to issues Feb 15, 2024

Add state to the attributes, namespaces, and attribute values #96

Closed

Policy API: Add soft delete feature #108

Closed

jakedoublev changed the title ~~feat(issue-96): add soft-delete/"deactivation" to namespaces, attribute definitions, attribute values~~ feat(issue-96, issue-108): add soft-delete/"deactivation" to namespaces, attribute definitions, attribute values Feb 15, 2024

jakedoublev added 2 commits February 15, 2024 10:20

cleanup

f4dbba3

rename schema ERD files and preserve timestamped schemas per migration

bbee6b9

dmihalcik-virtru self-requested a review February 15, 2024 16:24

dmihalcik-virtru previously approved these changes Feb 15, 2024

View reviewed changes

proto/common/common.proto Outdated Show resolved Hide resolved

proto/namespaces/namespaces.proto Show resolved Hide resolved

internal/db/attributes.go Outdated Show resolved Hide resolved

migrations/20240212000000_add_active_state_enums.sql Outdated Show resolved Hide resolved

jrschumacher changed the title ~~feat(issue-96, issue-108): add soft-delete/"deactivation" to namespaces, attribute definitions, attribute values~~ feat(policy): add soft-delete/"deactivation" to namespaces, attribute definitions, attribute values #96 #108 Feb 15, 2024

jrschumacher reviewed Feb 16, 2024

View reviewed changes

internal/db/attributes.go Outdated Show resolved Hide resolved

jrschumacher reviewed Feb 16, 2024

View reviewed changes

internal/db/attributes.go Outdated Show resolved Hide resolved

jrschumacher reviewed Feb 16, 2024

View reviewed changes

internal/db/attributes.go Outdated Show resolved Hide resolved

jrschumacher reviewed Feb 16, 2024

View reviewed changes

internal/db/attributes.go Outdated Show resolved Hide resolved

jakedoublev added 3 commits February 16, 2024 09:04

Merge branch 'main' into issue-96/state

09cc5df

use Tables struct and fields provided instead of tableField func for …

cf2981c

…clarity

remove unused tableField func

9e8238e

jakedoublev dismissed dmihalcik-virtru’s stale review via 9e8238e February 16, 2024 14:31

jakedoublev added 6 commits February 16, 2024 10:21

Merge branch 'main' into issue-96/state

fe83e1a

change state enum to active bool column in schema migration/erd

de308d6

allow clients to use a string enum active/inactive/any/unspecified in…

45263de

… proto messages but keep database column for active state a boolean

add more integration tests, update tests, and add deactivated state t…

5c07e8c

…o fixtures

Merge branch 'main' into issue-96/state

22af346

Merge branch 'main' into issue-96/state

28a6092

jakedoublev changed the title ~~feat(policy): add soft-delete/"deactivation" to namespaces, attribute definitions, attribute values #96 #108~~ feat(policy): add soft-delete/deactivation to namespaces, attribute definitions, attribute values #96 #108 Feb 16, 2024

jakedoublev requested review from dmihalcik-virtru and jrschumacher February 16, 2024 17:06

jrschumacher reviewed Feb 16, 2024

View reviewed changes

proto/attributes/attributes.proto Outdated Show resolved Hide resolved

jakedoublev added 6 commits February 16, 2024 15:22

update protos and generated code

18cf018

handle active response as boolean

4e334e6

update example grpcurls to reflect new response active field instead …

c080c4e

…of status enum

introduced google protobuf boolean wrapper to handle marshaling false…

44a0af1

… as zero value and real value of boolean fields in some policy messages (see issue: golang/protobuf#438)

consume wrapper type for active state

388d6eb

fix tests to account for new bool wrapper and make sure to scan into …

2905053

…bool variable and not the wrapper type

Merge branch 'main' into issue-96/state

7ca58bc

jrschumacher approved these changes Feb 17, 2024

View reviewed changes

Merge branch 'main' into issue-96/state

d80f19b

jrschumacher merged commit 02e92a6 into main Feb 17, 2024

jrschumacher deleted the issue-96/state branch February 17, 2024 04:52

opentdf-automation bot mentioned this pull request Apr 18, 2024

chore(main): release sdk 0.1.0 #612

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat(policy): add soft-delete/deactivation to namespaces, attribute definitions, attribute values #96 #108 #191

feat(policy): add soft-delete/deactivation to namespaces, attribute definitions, attribute values #96 #108 #191

Uh oh!

jakedoublev commented Feb 13, 2024 •

edited

Loading

Uh oh!

dmihalcik-virtru left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jakedoublev commented Feb 16, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

feat(policy): add soft-delete/deactivation to namespaces, attribute definitions, attribute values #96 #108 #191

feat(policy): add soft-delete/deactivation to namespaces, attribute definitions, attribute values #96 #108 #191

Uh oh!

Conversation

jakedoublev commented Feb 13, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dmihalcik-virtru left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jakedoublev commented Feb 16, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

jakedoublev commented Feb 13, 2024 •

edited

Loading