[1.18] Add workflow history signing documentation - #5082
Conversation
Add documentation for the workflow history signing feature, which provides cryptographic tamper detection for workflow execution histories using the sidecar's mTLS X.509 SPIFFE identity. Implemented in dapr/dapr#9666 Signed-off-by: joshvanl <me@joshvanl.dev>
Signed-off-by: joshvanl <me@joshvanl.dev>
There was a problem hiding this comment.
Pull request overview
Adds documentation for the “workflow history signing” feature, explaining how Dapr workflows sign and verify workflow history using the sidecar’s mTLS X.509 identity, and surfaces operational guidance around certificate/CA rotation.
Changes:
- Adds a new concept doc detailing workflow history signing, verification, failure modes, and state layout.
- Links the new doc from the workflow overview and the preview features list.
- Adds mTLS certificate rotation warnings specific to signed workflow history.
Reviewed changes
Copilot reviewed 5 out of 5 changed files in this pull request and generated 3 comments.
| File | Description |
|---|---|
| daprdocs/content/en/operations/support/support-preview-features.md | Adds “Workflow History Signing” to the preview features table and links to the new doc. |
| daprdocs/content/en/operations/security/mtls.md | Adds warnings about CA rotation impact on signed workflow histories. |
| daprdocs/content/en/developing-applications/building-blocks/workflow/workflow-overview.md | Adds a “History signing” overview section linking to the detailed doc. |
| daprdocs/content/en/developing-applications/building-blocks/workflow/workflow-history-signing.md | New, comprehensive documentation page for workflow history signing. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Signed-off-by: joshvanl <me@joshvanl.dev>
Signed-off-by: joshvanl <me@joshvanl.dev>
Signed-off-by: joshvanl <me@joshvanl.dev>
Signed-off-by: joshvanl <me@joshvanl.dev>
Add coverage for the v1.18 follow-up PRs on top of #9778: - dapr/dapr#9825 Tamper response: instead of just deleting reminders, the orchestrator now appends a single unsigned terminal ExecutionCompleted event tagged DAPR_WORKFLOW_HISTORY_TAMPERED on in-flight workflows, leaves already-completed workflows untouched, and routes signing-configuration mismatches through a separate ConfigurationError path that does not invoke tamper recovery. - dapr/dapr#9831 Child workflow and activity attestation: cross-identity completion events now carry a signed ChildCompletionAttestation / ActivityCompletionAttestation that commits to parent instance id, parent task scheduled id, a canonical ioDigest, the signer cert digest, and terminal status; receivers verify before the event enters the inbox and absorb foreign certs into ext-sigcert-NNNNNN. - dapr/dapr#9890 Lineage propagation: each PropagatedHistoryChunk now carries its own producer-signed signature plus DER cert chain; receivers verify chain-of-trust, SPIFFE app match, contiguity, and reject malformed variants explicitly. Also expand the failure-causes table, the state-store layout, the verification flow diagram, and the security-properties table. Signed-off-by: joshvanl <me@joshvanl.dev>
Signed-off-by: Marc Duiker <marcduiker@users.noreply.github.com>
marcduiker
left a comment
There was a problem hiding this comment.
LGTM, thanks for adding all the diagrams!
msfussell
left a comment
There was a problem hiding this comment.
Love the details on this article Thank you. Mostly my feedback in on the operational guidance side how to manage Dapr's default root cert when there are long running workflows that you would not be able to purge.
|
|
||
| Dapr workflow history signing provides cryptographic tamper detection for | ||
| workflow execution histories. Every history event produced during a workflow's | ||
| lifetime is signed using the sidecar's mTLS identity (X.509 SVID), creating an |
There was a problem hiding this comment.
| lifetime is signed using the sidecar's mTLS identity (X.509 SVID), creating an | |
| lifetime is signed using the sidecar's mTLS identity (X.509 SPIFFE Verifiable Identity Document (SVID)), creating an |
| 1. Deterministically marshals each new history event. | ||
| 2. Computes a SHA-256 digest over the batch of events. | ||
| 3. Chains the new digest to the previous signature's digest. | ||
| 4. Signs the combined input using the sidecar's [SPIFFE](https://spiffe.io/) X.509 private key. |
There was a problem hiding this comment.
| 4. Signs the combined input using the sidecar's [SPIFFE](https://spiffe.io/) X.509 private key. | |
| 4. Signs the combined input using the sidecar's [SPIFFE](https://spiffe.io/) X.509 private key (SVID) |
|
|
||
| | Condition | How to check | | ||
| |-----------|-------------| | ||
| | mTLS is enabled | Sentry is running and the sidecar has a valid SVID | |
There was a problem hiding this comment.
| | mTLS is enabled | Sentry is running and the sidecar has a valid SVID | | |
| | mTLS is enabled | Sentry service is running and the sidecar has a valid SVID | |
| ### Does signing add latency to workflow execution? | ||
|
|
||
| The signing operation (SHA-256 hashing and ECDSA/Ed25519 signing) is fast and | ||
| adds negligible latency. The main cost is the additional state store writes for |
There was a problem hiding this comment.
So does it add latency due to the writes? Not sure you answered the question.
There was a problem hiding this comment.
Yes, however is negligible compared to the existing latency profile. Expanded
| If you signed the new cert root with a **different private key**, you must restart the Dapr Sentry service, followed by the remainder of the Dapr control plane service. | ||
|
|
||
| {{% alert title="Workflow history signing" color="warning" %}} | ||
| If you rotate to a completely new root CA (different private key), any running workflows with [signed history]({{% ref "workflow-history-signing.md" %}}) will fail signature verification because their signing certificates were issued by the old CA. Those workflows will be reported as FAILED with error type `SignatureVerificationFailed`. To avoid this, complete or purge in-flight workflows before performing a full CA rotation. |
There was a problem hiding this comment.
This last sentence needs to be more discoverable in the FAQ above and in the overall article.
Is there anyway that you could overlap these root certs, so that you do not have to have downtime in affect? Say that your workflows run for 3 weeks on average, how to you purge these and update the root cert? That would seem to be a quite difficult task. Can you provide guidance here?
| *Note: If you signed the cert root with a different private key, restart the Dapr instances.* | ||
|
|
||
| {{% alert title="Workflow history signing" color="warning" %}} | ||
| If you rotate to a completely new root CA (different private key), any running workflows with [signed history]({{% ref "workflow-history-signing.md" %}}) will fail signature verification. Complete or purge in-flight workflows before performing a full CA rotation. |
There was a problem hiding this comment.
Same here. Needs specific guidance for long running workflows.
Signed-off-by: joshvanl <me@joshvanl.dev>
|
Thanks @msfussell, please can you take another look? |
Signed-off-by: joshvanl <me@joshvanl.dev>
Add documentation for the workflow history signing feature, which provides cryptographic tamper detection for workflow execution histories using the sidecar's mTLS X.509 SPIFFE identity.
Implemented in dapr/dapr#9778