Proposal: Building block API for cryptography

[ItalyPaleAle]

Note: Crypto as in cryptography, NOT cryptocurrency!!

Update: API Design proposal below: #4508 (comment)

In what area(s)?

/area runtime

Describe the proposal

This is a proposal for a new building block for Dapr to allow developers to leverage cryptography in a SAFE and consistent way. Goal is to expose an API that allows developers to ask Dapr to perform operations such as encrypting and decrypting messages, and calculating and verifying digital signatures.

Modern applications make extensive use of cryptography, which, when implemented correctly, can make solutions safer even in case data is compromised. Even more, in certain cases the use of crypto is required to comply with industry regulations (think banking) or even with legal requirements (GDPR). However, leveraging cryptography is hard: developers need to pick the right algorithms and options, and need to learn the proper way to manage and protect keys. Additionally, there are operational complexities when teams want to limit who has access to cryptographic key material.

Organizations have been increasingly started to leverage tools and services to perform crypto outside of applications. Examples include services such as Azure Key Vault, AWS KMS, Google Cloud KMS, etc. Customers may also use on-prem HSM products like Thales Luna. While those products/services perform the same or very similar operations, their APIs are very different.

This is an area where Dapr can help. Just like we're offering an abstraction on top of secret stores, we can offer an abstraction layer on top key vaults.

Using this building block, developers would be able to perform cryptographic operations without having to access raw key material. We would also offer a selection of algorithms that are configured correctly and forbid the usage of unsafe algorithms and operations. Algorithms available will depend on what the backend vaults support, but in general developers should always find AES (encrypt/decrypt only) and RSA; when supported, we can offer also ChaCha20-Poly1305 (encrypt/decrypt only) and ECC with ECDSA or EdDSA (sign/verify only).

Benefits would allow:

• Making it easier for developers to perform cryptographic operations in a safe way. Dapr provides safeguards against using unsafe algorithms, or using algorithms with unsafe options.
• Keeping keys outside of applications. Applications never see key material, but can request the vault to perform operations with the keys.
• Allowing greater separation of concerns. By using external vaults, only authorized teams can access private/shared key materials.
• Simplify key management and key rotation. Keys are managed in the vault and outside of the application, and they can be rotated without needing the developers to be involved (or even without restarting the apps).
• Enabling better audit logging to monitor when operations are performed with keys in the vault.

The new building block would feature 7 APIs:

• /encrypt: encrypts arbitrary data using a key stored in the vault. It supports symmetric and asymmetric ciphers, depending on the type of key in use (and the types of keys supported by the vault).
• /decrypt: decrypts arbitrary data, performing the opposite of what /encrypt does.
• /wrapkey: wraps keys using other keys stored in the vault. This is exactly like encrypting data, but it expects inputs to be formatted as keys (for example formatted as JSON Web Key) and it exposes additional algorithms not available when encrypting general data (like AES-KW)
• /unwrapkey: un-wraps (decrypts) keys, performing the opposite of what /wrap does
• /sign: signs an arbitrary message using an asymmetric key stored in the vault (we could also consider offering HMAC here, using symmetric keys, although not widely supported by the vault services)
• /verify: verifies a digital signature over an arbitrary message, using an asymmetric key stored in the vault (same: we may be able to offer HMAC too)
• /getkey: this can be used only with asymmetric keys stored in the vault, and returns the public part of the key

Different components would be developed to perform those operations on supported backends such as the products/services listed above. Dapr would "translate" these calls into whatever format the backends require. Dapr never sees the private/shared keys, which remain safely stored inside the vaults.

Additionally, we could offer a "local" crypto component where keys are stored as Kubernetes secrets and cryptographic operations are performed within the Dapr sidecar. Although this is not as secure as using an external key vault, it still offers some benefits such as using standardized APIs and separation of concerns/roles with regards to key management.

[yaron2]

I support adding a cryptography API as it will hide a lot of complexity and provide best practices in an area where they are very needed.

I particularly find useful the "local" version of the component.

[georgestevens99]

I support a "Building block for cryptography". And I would like to add a feature to your list of features that seeks to make end-to-end message encryption really, really easy to use, for both the sender and receiver. Namely, hide all the details of message encryption from the developer to have the kind of end-to-end message encryption necessary in highly secure situations (used even with a locked down cluster).

The Defense In Depth security best practice now widely recommended demands multiple levels of security. Just like putting your jewels in a safe within your locked house. Just a locked house is not good enough to protect really high value items. You need more layers of security -- a safe, how about a guard dog, an alarm on your front door, an alarm on your safe, etc. -- to have a good Defense In Depth. Note that this is key when communicating from a service to a Dapr side car. There is no mTLS on that network hop and to compensate for that there needs to be a good end-to-end message encryption that can easily span this potential message security gap. Plus it needs to be really easy to use so that a developer does not have to understand all about public and private keys, etc., etc. to effectively do highly secure, failsafe, end-to-end message encryption on both the sending and receiving ends.

When using encrypted messages flowing between 2 services (that may be located who knows where, and not just locally, but it applies to a local cluster/host as well) end to end message encryption is highly desireable, especially for portions of the message that are seriously private. In order for the sender and receiver to encrypt/decrypt the message (or portions of it, like PII) they need to share encryption keys, aka the public key -- this the the hardest part of end-to-end encryption.

Please implement an end-to-end message encryption feature that uses a "Secure Conversation" concept that allows the sender and receiver to securely communicate to exchange info containing the encryption key for each message. There is a spec defined for this called WS-SecureConversation. Described as "Following a pattern similar to TLS, WS-SecureConversation establishes a kind of session key." See https://en.wikipedia.org/wiki/WS-SecureConversation and, for a complete spec http://docs.oasis-open.org/ws-sx/ws-secureconversation/v1.4/ws-secureconversation.html.

Plus, please make it really easy to use through a simple API which I will leave to others to specify in detail. It would be nice to have a single really high level operation like EndToEndEncrypt(someMessage) that takes care of all the grubby secure conversation/"session key" details, plus the encryption, behind the scenes.

The service would call EndToEndEncrypt() to encrypt the message before it is sent to message through Dapr via Service Invocation AND/OR Pubsub. Then Dapr would take care of all the rest of the details. Then when an encrypted message is received by the receiver Dapr does the decryption behind the scenes as well.

I believe that such a "session key" may be able to be obtained from Sentry, or based on things that Sentry already does. Thus, this feature, especially the "session key" part of it, may not be all that difficult to implement, leveraging existing code. (I hope).

Thanks for the highly useful idea!

[ItalyPaleAle]

Hey @georgestevens99 thanks for sharing your thoughts!

I would like to understand more about your requirements for E2E encryption in this case.

As you know, communication between Dapr-ized applications already uses TLS. Between Dapr sidecars, communications use gRPC with mTLS, which offers network-level encryption and mutual authentication. Communication between applications and the Dapr sidecar isn't encrypted, but that should happen locally.

Because of that, I don't know if the requirement for providing an API for E2EE communication would fit in the scope of this proposal, which is more focused on lower-level cryptographic primitives than higher-level, opinionated protocols.

If E2EE is necessary, however, I would consider implementing that by performing an ECDH key agreement between two applications and deriving that way a symmetric key that will be used with AES (or if hardware-accelerated AES isn't available, ChaCha20-Poly1305). I don't know if this necessarily needs to be implemented in Dapr itself, however: I feel that for true E2EE you would want to implement this in your applications: each service would expose an endpoint that is invoked to perform the ECDH agreement, and that allows obtaining the shared secret. This would happen with a regular service-to-service invocation between your applications, which would transparently pass through Dapr (ECDH is designed to be used over insecure channels so it's perfectly safe even if no encryption is present). If this were implemented within Dapr, additionally, then the sidecars would be able to sniff the shared encryption key, thus violating the E2EE requirement.

Perhaps we could provide an example of how to do this design (ECDH to derive a shared encryption key) even with current versions of Dapr :)

[georgestevens99]

E2EE for a message (or parts of it) is suggested by the dapr documents for highly confidential info. But dapr currently supplies no way to do such encryption.

While E2EE can certainly be done at a low level as is your idea, the fact of the matter is that there are vast numbers of developers that do not have the experience, knowledge, nor skills to use the "lower level crypto privatives" you are proposing. Couple that with the fact that security is growing more and more challenging each year. Thus in addition to having lower level primitives for those who have the skills to use them, we (our industry) also need a really, really simple to use and highly effective and highly secure E2EE for messages sent from a Service to a Dapr sidecar and beyond. This will provide a solid, highly productive solution for the encryption for this network hop (and beyond) that is suggested in the Dapr docs.

In my experience, most developers haven't a clue as to what you said means, i.e. "If E2EE is necessary, however, I would consider implementing that by performing an ECDH key agreement between two applications and deriving that way a symmetric key that will be used with AES (or if hardware-accelerated AES isn't available, ChaCha20-Poly1305). ". And learning how to effectively use this low level stuff is non trivial and error prone and takes time.

So we need E2EE for developers that have little or no background in security. So think of it as a higher level addition to your excellent proposal. This high level E2EE should be as easy to use as is Dapr Service Invocation or Pubsub or State Management is when done via one of the SDKs. Then, anyone who can write code with Dapr can also do near-world class E2EE of their messages and PII without climbing a steep security learning curve. That is the overall goal of my feature proposal.

[ItalyPaleAle]

A fully-transparent E2EE would only be possible between Dapr sidecars, but Dapr sidecars already talk to each other using mTLS (which is enabled by default), which should provide equal levels of security and authentication.

Perhaps a middle-ground would be to implement an API that performs the ECDH between two services (technically, between two sidecars) and gives the application a shared secret they can then use as they see fit, including using the /encrypt endpoint of this proposal. However it should be clear that in this case the sidecars do have visibility over the encryption keys, even if for just an instant, so it's not true E2EE between the applications. I am not sure if that would void some of the security requirements you're talking about.

Or, we could just create some samples that demonstrate how to do what I described above: creating an endpoint in a service so that they can perform an ECDH agreement, then use that shared secret with the /encrypt endpoint.

Also worth pointing out that in all the cases above, both with things that are integrated with Dapr or implemented in the applications, there's always the problem of authenticating the other party. Dapr doesn't authenticate application IDs, so anyone could create an application with any ID: just because you're performing an ECDH with an application called foo, you don't have any guarantee that it's the foo you want to talk to.

[georgestevens99]

@ItalyPaleAle -- Thanks so much for taking the time to write up the various alternatives that aim at fulfilling my requrements. I really appreciate it. It forced me to think through my requirements and also the limitations inherent in Dapr as it now stands.

I was hoping there was a way to use Dapr's future capabilities to implement industrial strength E2EE for messages.
However, given the caveats you pointed out in your text (many thanks for those!) I am abandoning my attempts to use Dapr for this. Instead I am "biting the bullet" and planning on developing a small Security Utility service to at least deal with the encryption keys for the sender and receiver, and likely more. That way I can use identity based security to authenticate the sender and receiver to the Security Utility to setup encryption keys and avoid the potential identity ambiguity you described in your last paragraph, plus also avoid a tiny "instant" you described where "the sidecars do have visibility over the encryption keys", thus resulting in better overall security.

Thus, I am withdrawing my request for extending your "Building block for cryptography" to support E2EE as well.

Based on our conversation thus far, it seems to me that Dapr could offer such a standalone, un-daprized (no sidecar) Security Utility for E2EE. This would be highly useful to those of us who need to deal with very high levels of security. Any thoughts on the possibility of a such a Dapr Security Utility?

Thanks again for your contribution to this conversation!

[ItalyPaleAle]

Thanks for the detailed writeup!

I think what you wrote could be interesting. I don't know if this would be part of the "official" Dapr or some "community-contributed" extension.

The biggest problem about making this official is that there may not be a one-size-fits-all. So we'd have to be opinionated and with that we'd have to make tradeoffs.

Perhaps we should take this to another issue (or on discord). It would be useful to know your requirements too. Establishing a secure communication channel is "easy" (I described above how it could just be as "simple" as using ECIES, by means of performing an ECDH over Dapr's service invocation APIs). But if the requirements include authenticating services, then you start getting into needing a PKI or something like that (how 'bout a blockchain? 😂 just kidding). Things can spiral into getting very complicated, very fast.

[georgestevens99]

@ItalyPaleAle --Good idea. I am OK on being opinionated and making tradeoffs. I am mainly interested in a E2EE security utility that works with .NET, especially ASP.NET Core gRPC services using Protobuf-net code first, and using either certificate or AAD identity based authentication with each service having its own identity. But within that package there is core functionality that could probably be used by various languages, etc. And I am interested in preventing complexity spirals as well.

What Dapr Repo would be appropriate for a "Standalone Dapr E2EE Security Utility"? Lets start a new issue for that in the appropriate repo and take it from there!

[olitomlinson]

+1 for this proposal.

Particularly interested in this building block, as we want to offer a Bring Your Own Keys feature to our customers.

It would be super neat if a tenant in our multi-tenant system could share with us their key (one-time, super locked down SDLC for this part of the app which receives keys) and then we never get to see that key ever again because it is tucked away in a cloud vendor HSM/daprd, and our developers never get access to that key as they use an API something like this :

I would see a dapr API looking something like this

POST http://localhost:3500/v1.0/crypto/{tenant_id}/encrypt

REQ

{
  "unencrypted_payload" : "I am super secret tenant data. Its encrypted state will be used downstream in dapr State Management & dapr Pub Sub components"
}

RESPONSE

{
  "encrypted_payload" : "9yo&*Ti6TEo437togeg43qow347yga4pw3ogh7GOGo7gGWPW473y48p3T8Y"
}

downstream....

POST http://localhost:3500/v1.0/state/someStateComponent

[
  {
    "key": "weapon",
    "value": "9yo&*Ti6TEo437togeg43qow347yga4pw3ogh7GOGo7gGWPW473y48p3T8Y",
    "etag": "1234"
  }
]

[dapr-bot]

This issue has been automatically marked as stale because it has not had activity in the last 60 days. It will be closed in the next 7 days unless it is tagged (pinned, good first issue, help wanted or triaged/resolved) or other activity occurs. Thank you for your contributions.