Skip to content

Unmarshal and compare user objects in CompareAndSwapUser#37660

Merged
espadolini merged 3 commits intomasterfrom
espadolini/compareandswapuser
Feb 2, 2024
Merged

Unmarshal and compare user objects in CompareAndSwapUser#37660
espadolini merged 3 commits intomasterfrom
espadolini/compareandswapuser

Conversation

@espadolini
Copy link
Copy Markdown
Contributor

This PR makes it so that CompareAndSwapUser does a comparison between unmarshaled user objects rather than comparing the JSON serialization through a backend CompareAndSwap. Seeing as this requires fetching the object anyway, the new implementation uses a ConditionalUpdate operation, which is generally cheaper.

This fixes the failure condition reported at #35325 (comment) , but doesn't implement broader changes (to replace all the CompareAndSwapUser calls that almost always already get the current value anyway with calls to UpdateUser).

changelog: fix conditional user modifications (used by certain Teleport subsystems such as Device Trust) on users that have previously been locked out due to repeated recovery attempts

@espadolini
Copy link
Copy Markdown
Contributor Author

This is barely changelog-worthy, since any modification to the user params item that doesn't go through CompareAndSwapUser will also fix the issue.

Copy link
Copy Markdown
Contributor

@codingllama codingllama left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for tackling this, Edoardo!

Comment thread lib/services/local/users.go Outdated
Comment thread lib/services/local/users.go Outdated
Comment thread lib/services/local/users.go Outdated
Comment thread lib/services/local/users.go Outdated
Key: backend.Key(webPrefix, usersPrefix, existing.GetName(), paramsPrefix),
Value: existingValue,
}
const iterationLimit = 5
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is the rationale behind the retries? Are all ConditionalUpdate callers expected to retry? Why 5 times exactly?

Copy link
Copy Markdown
Contributor Author

@espadolini espadolini Feb 2, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Revisions change because something else modified the item - but from the point of view of a compare-and-swap operation we don't care about that, we only care if the value matches or not, and a concurrent modification that leaves the value identical should still result in a successful compare-and-swap. In practice this isn't actually followed in some of our CompareAndSwapFoo implementations, unfortunately: CompareAndSwapCertAuthority can fail even though it should succeed if the existing value is concurrently overwritten with a different one that is equivalent after deserialization (in that specific case it's not a big deal because all cases of CASing a cert authority are already on various timed loops to begin with).

Whether or not revisions can also spuriously change is still not fully defined - with the more recent changes to firestorebk (to do transactions and store a hard revision value in the actual document rather than rely on lastModifiedTime), the only backend for which that's a possibility is etcdbk, where the backend revision of an item is the modrevision of the underlying etcd item, and that would change if a backup is restored while Teleport is running (which would be crazy, admittedly).

Oh, and I picked 5 because humans generally have 5 fingers on one hand, although this is such a fringe scenario that I'm going to change it to 3: the initial attempt, one retry because spurious failures could actually happen, and another one because a single retry just feels weird.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

and I picked 5 because humans generally have 5 fingers on one hand (...)

That's rock-solid logic, can't argue with that.

Thanks for the explanation. So what I'm gathering is that looped ConditionalUpdates are a practice we'd like to encourage, which is good to know.

Do you think it makes sense to capture the number of retries in a more generic, reusable constant, like ConditionalUpdateDefaultRetries (or something to that effect)?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't know if there's a one-size-fits-all value for such a constant, I'm only doing a retry loop very close to the backend here because I want to match the slightly stronger semantics of compare-and-swap.

Comment thread lib/services/local/users.go Outdated
Comment thread lib/services/local/users.go
espadolini and others added 2 commits February 2, 2024 11:29
Co-authored-by: Alan Parra <alan.parra@goteleport.com>
Comment thread lib/services/local/users.go
@espadolini espadolini added this pull request to the merge queue Feb 2, 2024
Merged via the queue into master with commit e4043cf Feb 2, 2024
@espadolini espadolini deleted the espadolini/compareandswapuser branch February 2, 2024 14:32
@public-teleport-github-review-bot
Copy link
Copy Markdown

@espadolini See the table below for backport results.

Branch Result
branch/v15 Create PR

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants