Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
132 changes: 132 additions & 0 deletions docs/cosmos-resource-deletion.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,132 @@
# Cosmos Resource Deletion via Soft-Delete

## Problem

Cosmos DB's "latest version" change feed mode does not surface hard deletions.
When a resource is deleted via `DeleteItem`, the change feed never sees that event, which means
controllers watching through `ChangeFeedListWatcher` cannot react to deletions until the next
full relist (up to 30 minutes later). This breaks correctness guarantees for controllers that
need to act promptly on resource removal.

## Solution: DeletionTimestamp + TTL

Instead of hard-deleting documents with `DeleteItem`, we implement a two-phase soft-delete:

1. **Mark phase** (immediate): Set `DeletionTimestamp` on the document and set the Cosmos TTL
to 30 seconds. This is a Replace operation, which the change feed **does** surface.
2. **Cleanup phase** (automatic): After the TTL expires, Cosmos DB automatically removes the
document from storage.

### Document Schema Change

`TypedDocument` gains a new field:

```go
type TypedDocument struct {
BaseDocument
PartitionKey string `json:"partitionKey"`
ResourceID *azcorearm.ResourceID `json:"resourceID"`
ResourceType string `json:"resourceType"`
DeletionTimestamp *metav1.Time `json:"deletionTimestamp,omitempty"`
Properties json.RawMessage `json:"properties"`
}
```

When `DeletionTimestamp` is nil (or absent from the JSON), the document is live. When set, the
document is logically deleted and awaiting Cosmos TTL cleanup.

### CRUD Behavior Changes

#### Delete

When `Delete` is called, the CRUD helper:

1. Reads the document from Cosmos (`ReadItem`).
2. Sets `deletionTimestamp` to the current time.
3. Sets `ttl` to 30 (seconds after the write timestamp `_ts`).
4. Increments `properties.cosmosMetadata.instanceVersion` so the change feed recognizes the update.
5. Replaces the document with an etag precondition (optimistic concurrency).

If the document is already not found, the delete is a no-op (idempotent).

#### Get / GetByID

After reading a document, the CRUD layer checks for `DeletionTimestamp`. If it is non-nil,
the method returns a 404 Not Found error as if the document does not exist. This ensures
callers never see a logically-deleted document through the typed CRUD layer.

#### List (including global listers)

All list queries gain an additional WHERE clause:

```sql
AND (NOT IS_DEFINED(c.deletionTimestamp))
```

Since `deletionTimestamp` uses `omitempty` in its JSON tag, live documents have no
`deletionTimestamp` field in their JSON at all. The `IS_DEFINED` check is both correct
and index-friendly. This filter applies to:

- Per-partition scoped lists (`nestedCosmosResourceCRUD.List`)
- Cross-partition global listers (`cosmosGlobalLister.List`)
- Active operations lister (`cosmosActiveOperationsGlobalLister.List`)

### Change Feed Behavior

The `processDocument` method in `ChangeFeedWatcher` checks `DeletionTimestamp` on every
document surfaced by the change feed:

- **DeletionTimestamp is non-nil AND the resource was previously seen** (in the list snapshot
or a prior change feed event): Deliver a `watch.Deleted` event with the current document
content.
- **DeletionTimestamp is non-nil AND the resource was NOT previously seen**: Skip silently.
This handles the case where a resource was created and deleted between relists.

This means controllers and informers receive timely Delete notifications without waiting for
a relist cycle.

### Mock Implementation

The in-memory mock (`MockResourcesDBClient`) mirrors production behavior:

- `Delete` performs a read-modify-write that sets `DeletionTimestamp` and `TTL`, then calls
`StoreDocument` (which records to the mock change feed).
- `GetDocument` returns the raw document regardless of deletion status (the CRUD layer does
the filtering).
- `ListDocuments` filters out documents with `DeletionTimestamp` set.
- The mock change feed sees the soft-delete as a normal mutation (since it goes through
`StoreDocument`), so changefeed-based tests work identically.

### Timeline

```
T+0s Delete() called
├─ ReadItem (get current doc + etag)
├─ Set deletionTimestamp = now, ttl = 30
├─ Increment instanceVersion
└─ ReplaceItem (conditional on etag)

T+0-1s Change feed surfaces the updated document
└─ ChangeFeedWatcher delivers watch.Deleted event

T+30s Cosmos TTL expires
└─ Document physically removed from storage
```

## Integration Tests

The following test scenarios verify the end-to-end behavior:

1. **Changefeed delivers Delete event**: Create a cluster, start list/watch, delete the
cluster, verify a `watch.Deleted` event arrives with the document content.

2. **Changefeed + lister reports not-found**: Create a cluster, start an informer, delete
the cluster, verify the informer delivers a Delete event and the lister returns not-found.

3. **Changefeed + relist has no extra records**: Create a cluster, delete it, then perform a
fresh list/watch cycle. Verify no events are delivered for the deleted cluster (it should
not appear in either the list result or as a change feed event).

4. **Cosmos TTL cleanup**: Create a cluster, delete it (sets 30s TTL), wait for TTL to expire,
then verify the document is physically absent from Cosmos. This test runs only against the
real Cosmos emulator.
1 change: 1 addition & 0 deletions internal/database/crud_hcpcluster.go
Original file line number Diff line number Diff line change
Expand Up @@ -69,6 +69,7 @@ func (d *operationCRUD) ListActiveOperations(options *ResourcesDBClientListActiv
query := fmt.Sprintf(
"SELECT * FROM c WHERE STRINGEQUALS(c.resourceType, %q, true) "+
"AND LENGTH(c.resourceID) > 0 "+
"AND (NOT IS_DEFINED(c.deletionTimestamp)) "+
"AND NOT ARRAYCONTAINS([%q, %q, %q], c.properties.status)",
api.OperationStatusResourceType.String(),
arm.ProvisioningStateSucceeded,
Expand Down
86 changes: 79 additions & 7 deletions internal/database/crud_helpers.go
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,10 @@ import (
"errors"
"fmt"
"strings"
"time"

metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"k8s.io/apimachinery/pkg/apis/meta/v1/unstructured"
"k8s.io/utils/ptr"

azcorearm "github.com/Azure/azure-sdk-for-go/sdk/azcore/arm"
Expand All @@ -30,6 +33,8 @@ import (
"github.com/Azure/ARO-HCP/internal/utils"
)

const SoftDeleteTTLSeconds = 30

func OldResourceIDToCosmosID(resourceID *azcorearm.ResourceID) (string, error) {
if resourceID == nil {
return "", errors.New("resource ID is nil")
Expand Down Expand Up @@ -76,9 +81,21 @@ func getByItemID[InternalAPIType, CosmosAPIType any](ctx context.Context, contai
return nil, utils.TrackError(err)
}

if IsSoftDeleted(responseItem.Value) {
return nil, NewNotFoundError()
}

return responseItemToInternalObj[InternalAPIType, CosmosAPIType](ctx, cosmosID, responseItem)
}

func IsSoftDeleted(data []byte) bool {
var doc TypedDocument
if err := json.Unmarshal(data, &doc); err != nil {
return false
}
return doc.DeletionTimestamp != nil
}

func get[InternalAPIType, CosmosAPIType any](ctx context.Context, containerClient *azcosmos.ContainerClient, partitionKeyString string, completeResourceID *azcorearm.ResourceID) (*InternalAPIType, error) {
// try to see if the cosmosID we've passed is also the exact resource ID. If so, then return the value we got.
newExactCosmosID, err := arm.ResourceIDToCosmosID(completeResourceID)
Expand All @@ -101,9 +118,9 @@ func list[InternalAPIType, CosmosAPIType any](ctx context.Context, containerClie
PageSizeHint: -1,
}
if prefix == nil {
query = "SELECT * FROM c WHERE LENGTH(c.resourceID) > 0"
query = "SELECT * FROM c WHERE LENGTH(c.resourceID) > 0 AND (NOT IS_DEFINED(c.deletionTimestamp))"
} else {
query = "SELECT * FROM c WHERE STARTSWITH(c.resourceID, @prefix, true)"
query = "SELECT * FROM c WHERE STARTSWITH(c.resourceID, @prefix, true) AND (NOT IS_DEFINED(c.deletionTimestamp))"
queryOptions = azcosmos.QueryOptions{
PageSizeHint: -1,
QueryParameters: []azcosmos.QueryParameter{
Expand Down Expand Up @@ -347,20 +364,75 @@ func deleteResource(ctx context.Context, containerClient *azcosmos.ContainerClie
return utils.TrackError(err)
}

_, err = containerClient.DeleteItem(ctx, azcosmos.NewPartitionKeyString(partitionKeyString), cosmosID, nil)
return softDeleteByCosmosID(ctx, containerClient, partitionKeyString, cosmosID)
}

func deleteByCosmosID(ctx context.Context, containerClient *azcosmos.ContainerClient, partitionKeyString, cosmosID string) error {
return softDeleteByCosmosID(ctx, containerClient, partitionKeyString, cosmosID)
}

func softDeleteByCosmosID(ctx context.Context, containerClient *azcosmos.ContainerClient, partitionKeyString, cosmosID string) error {
pk := azcosmos.NewPartitionKeyString(partitionKeyString)

responseItem, err := containerClient.ReadItem(ctx, pk, cosmosID, nil)
if IsNotFoundError(err) {
return nil
}
if err != nil {
return utils.TrackError(err)
}
return nil
}

func deleteByCosmosID(ctx context.Context, containerClient *azcosmos.ContainerClient, partitionKeyString, cosmosID string) error {
_, err := containerClient.DeleteItem(ctx, azcosmos.NewPartitionKeyString(partitionKeyString), cosmosID, nil)
var doc GenericDocument[map[string]interface{}]
if err := json.Unmarshal(responseItem.Value, &doc); err != nil {
return fmt.Errorf("failed to unmarshal document for soft delete: %w", err)
}

if doc.DeletionTimestamp != nil {
return nil
}

SetSoftDeleteFields(&doc, time.Now())

modified, err := json.Marshal(doc)
if err != nil {
return fmt.Errorf("failed to marshal soft-deleted document: %w", err)
}
Comment on lines +394 to +399

opts := &azcosmos.ItemOptions{
IfMatchEtag: &responseItem.ETag,
}
_, err = containerClient.ReplaceItem(ctx, pk, cosmosID, modified, opts)
if err != nil {
return utils.TrackError(err)
Comment thread
deads2k marked this conversation as resolved.
Resolved
}
return nil
}

// SetSoftDeleteFields marks a document as soft-deleted and increments its instanceVersion.
// We use map[string]interface{} instead of unstructured.Unstructured because
// Unstructured.UnmarshalJSON requires kind/apiVersion fields and fails with
// "Object 'Kind' is missing in '<json>'" on Cosmos documents.
func SetSoftDeleteFields(doc *GenericDocument[map[string]interface{}], now time.Time) {
if doc.DeletionTimestamp != nil {
return
}

ts := metav1.NewTime(now)
doc.DeletionTimestamp = &ts
doc.TimeToLive = SoftDeleteTTLSeconds

Comment on lines +420 to +423
// allow deleting legacy content
if doc.Content == nil {
return
}

fv, found, err := unstructured.NestedFloat64(doc.Content, "cosmosMetadata", "instanceVersion")
current := int64(1)
if err == nil && found {
current = int64(fv)
if current < 1 {
current = 1
}
}
_ = unstructured.SetNestedField(doc.Content, current+1, "cosmosMetadata", "instanceVersion")
}
3 changes: 2 additions & 1 deletion internal/database/global_lister.go
Original file line number Diff line number Diff line change
Expand Up @@ -150,6 +150,7 @@ func (l *cosmosActiveOperationsGlobalLister) List(ctx context.Context, options *
query := fmt.Sprintf(
"SELECT * FROM c WHERE STRINGEQUALS(c.resourceType, %q, true) "+
"AND LENGTH(c.resourceID) > 0 "+
"AND (NOT IS_DEFINED(c.deletionTimestamp)) "+
"AND NOT ARRAYCONTAINS([%q, %q, %q], c.properties.status)",
api.OperationStatusResourceType.String(),
arm.ProvisioningStateSucceeded,
Expand Down Expand Up @@ -192,7 +193,7 @@ func (l *cosmosGlobalLister[InternalAPIType, CosmosAPIType]) List(ctx context.Co
resourceTypeConditions = append(resourceTypeConditions, fmt.Sprintf("STRINGEQUALS(c.resourceType, %q, true)", resourceType.String()))
}
whereClause := strings.Join(resourceTypeConditions, " OR ")
query := fmt.Sprintf("SELECT * FROM c WHERE LENGTH(c.resourceID) > 0 AND (%s)", whereClause)
query := fmt.Sprintf("SELECT * FROM c WHERE LENGTH(c.resourceID) > 0 AND (NOT IS_DEFINED(c.deletionTimestamp)) AND (%s)", whereClause)

queryOptions := azcosmos.QueryOptions{
PageSizeHint: -1,
Expand Down
10 changes: 8 additions & 2 deletions internal/database/informers/changefeed_list_watch.go
Original file line number Diff line number Diff line change
Expand Up @@ -404,10 +404,16 @@ func (c *ChangeFeedWatcher[InternalAPIType, InternalAPITypePointer, CosmosAPITyp
}

objDeleted := false
if c.shouldDeliverItemFn != nil && !c.shouldDeliverItemFn(internalObj) {
if objAsTypedDocument.DeletionTimestamp != nil {
if objPreviouslySeen {
objDeleted = true
} else {
logger.Info("skipping soft-deleted document not previously seen", "content", cosmosObj)
return nil
}
} else if c.shouldDeliverItemFn != nil && !c.shouldDeliverItemFn(internalObj) {
if objPreviouslySeen {
objDeleted = true
// we need to deliver a delete, so fall through
} else {
logger.Info("should not deliver document", "content", cosmosObj)
return nil
Expand Down
11 changes: 7 additions & 4 deletions internal/database/types_typeddocument.go
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,8 @@ package database
import (
"encoding/json"

metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"

azcorearm "github.com/Azure/azure-sdk-for-go/sdk/azcore/arm"
)

Expand All @@ -26,8 +28,9 @@ import (
// implements the DocumentProperties interface.
type TypedDocument struct {
BaseDocument
PartitionKey string `json:"partitionKey"`
ResourceID *azcorearm.ResourceID `json:"resourceID"`
ResourceType string `json:"resourceType"`
Properties json.RawMessage `json:"properties"`
PartitionKey string `json:"partitionKey"`
ResourceID *azcorearm.ResourceID `json:"resourceID"`
ResourceType string `json:"resourceType"`
DeletionTimestamp *metav1.Time `json:"deletionTimestamp,omitempty"`
Properties json.RawMessage `json:"properties"`
}
7 changes: 4 additions & 3 deletions internal/databasetesting/mock_resources_changefeed.go
Original file line number Diff line number Diff line change
Expand Up @@ -27,9 +27,10 @@ import (
// mockChangeFeed is an in-memory log of mutations to the Resources
// container. Every successful StoreDocument call (Create or Replace)
// appends a snapshot of the stored document; reads return everything
// past the consumer's continuation position. Deletes are intentionally
// NOT recorded — that mirrors "latest version" change feed mode in
// production Cosmos DB, which does not surface deletions.
// past the consumer's continuation position. Hard deletes are
// intentionally NOT recorded — that mirrors "latest version" change
// feed mode in production Cosmos DB. Soft deletes (which are modeled
// as Replace operations) ARE recorded via StoreDocument.
type mockChangeFeed struct {
mu sync.Mutex
events []json.RawMessage
Expand Down
Loading