feat(crypto): add encrypt and decrypt APIs

[ItalyPaleAle]

Description

This PR implements the crypto.encrypt and crypto.decrypt methods in the JS SDK. These allow using the crypto APIs in Dapr 1.11 to encrypt and decrypt data.

• The APIs work with streams: they accept a readable stream and return a readable stream with the result
• Only gRPC is supported. Trying to invoke these methods using the HTTP client will throw an exception. This is by design, because the high-level crypto APIs should only be used with gRPC (a HTTP method is available but is only meant for development/testing).
• Added examples

TODO

1. I would like to work a bit more on the APIs. Right now, they accept a Readable stream and return another Readable stream. I would like to explore if using a Duplex stream (as a transformer) would be better.
2. Not sure how I would go about writing tests for this, besides the example.

Issue reference

Fixes #474

Checklist

Please make sure you've completed the relevant tasks for this PR, out of the following list:

• Code compiles correctly
• Created/updated tests
• Extended the documentation

[codecov]

Codecov Report

Merging #491 (27fde9b) into main (1114591) will decrease coverage by 0.31%.
The diff coverage is 10.83%.

@@            Coverage Diff             @@
##             main     #491      +/-   ##
==========================================
- Coverage   36.14%   35.84%   -0.31%     
==========================================
  Files          82       85       +3     
  Lines        9927    10047     +120     
  Branches      371      394      +23     
==========================================
+ Hits         3588     3601      +13     
- Misses       6280     6387     +107     
  Partials       59       59              

Impacted Files	Coverage Δ
src/implementation/Client/GRPCClient/crypto.ts	6.06% <6.06%> (ø)
src/utils/Streams.util.ts	6.52% <6.52%> (ø)
src/implementation/Client/HTTPClient/crypto.ts	50.00% <50.00%> (ø)
src/implementation/Client/DaprClient.ts	84.26% <100.00%> (+0.74%)	⬆️

[XavierGeerinck]

Hi @ItalyPaleAle ! First of all, thank you so much for contributing this, it is shaping up to be an amazing PR!!

As for my thoughts, I do have some questions:

• The Crypto API supports both streams (e.g., file encryption) as well as small messages (through the Subtle API if I read this correctly?). Do you plan on adding the latter as well?
• I am thinking if we have streams and messages, I would be fond of using encryptStream and encrypt where the latter is the Subtle API implementation. The first would make it clearer that a stream is required.
• Could you walk me through what it is you are proposing here? Will this just encrypt a file itself by sending the payload over a gRPC stream and return the encrypted one that it then saves to a file? What use cases did you have in mind for the streaming API (to help me make sense of how we could improve the API spec)

(adding your example below to illustrate what you proposed in the PR here more easily)

await pipeline(
  await client.crypto.encrypt(createReadStream("plaintext.txt"), {
    componentName: "crypto-local",
    keyName: "symmetric256",
    keyWrapAlgorithm: "A256KW",
  }),
  createWriteStream("ciphertext.out"),
);

// Decrypt the message
await pipeline(
  await client.crypto.decrypt(createReadStream("ciphertext.out"), {
    componentName: "crypto-local",
  }),
  createWriteStream("plaintext. Out"),
);

[ItalyPaleAle]

Hi @XavierGeerinck thanks for the initial review.

On subtle APIs:

• First, please note Subtle Crypto APIs are not available in 1.11. Although fully-implemented, they are not included in the binary for now. So no need to implement a SDK right now.
• On the difference between subtle and high-level, I would suggest reading Add proposal for crypto building block proposals#3 to understand how they work with each other. The TLDR is that yes, both offer an "encrypt" method, but they're very different (subtle APIs offer other methods too). The subtle APIs are low-level, and offer direct access to the crypto primitives. The difference isn't just that the high-level APIs allow encrypting larger messages (although depending on how they're used, the low-level APIs allow encrypting "large" messages too): the high-level APIs implement an encryption scheme that is optimized for encrypting data as a stream, safely while using state-of-the-art crypto.

Because of the above, my suggestion would be to keep the subtle crypto APIs (when they are implemented for Dapr 1.12 or later) into a separate object, for example crypto.subtle.encrypt (just like in Web Crypto!)

Could you walk me through what it is you are proposing here? Will this just encrypt a file itself by sending the payload over a gRPC stream and return the encrypted one that it then saves to a file?

Yes, this is how the high-level encrypt/decrypt methods are meant to use. Except they're not just for files: anything that is a Readable stream can be encrypted, including Response.body from the Fetch API and a host more.

The current API design mirrors what's implemented with the Go SDK: first parameter is a Readable stream with the input, and the result is a (Promise that resolves with) a Readable stream as output. This requires using something like the above:

await pipeline(
  await client.crypto.encrypt(createReadStream("plaintext.txt"), {
    componentName: "crypto-local",
    keyName: "symmetric256",
    keyWrapAlgorithm: "A256KW",
  }),
  createWriteStream("ciphertext.out"),
);

Where the pipeline thing is just syntactic sugar over:

const cryptoStream = await client.crypto.encrypt(createReadStream("plaintext.txt"), {
  componentName: "crypto-local",
  keyName: "symmetric256",
  keyWrapAlgorithm: "A256KW",
})
const outStream = createWriteStream("ciphertext.out")
cryptoStream.pipe(outStream)
// Plus code to `await` on cryptoStream's end

I am thinking that for JS, maybe this could be changed to use a Duplex stream, where data is written into the writable part of the stream, and the result is read from the readable part. This would allow for something like this:

await pipeline(
  createReadStream("plaintext.txt"),
  // Note "await" is missing here; errors are returned into the stream
  client.crypto.encrypt({
    componentName: "crypto-local",
    keyName: "symmetric256",
    keyWrapAlgorithm: "A256KW",
  }),
  createWriteStream("ciphertext.out"),
);

This seems nicer when used with pipeline, but it's a bit more complex if users have to do things by hand.

Another thing to consider specifically for JS is perhaps we could offer a wrapper around client.crypto.encrypt to allow encrypting a message already in-memory (like a Buffer or ArrayBuffer or an ArrayBuffer view), and retrieve the data as a Buffer/ArrayBuffer too. Something like:

const message: = Buffer.from(...)
const encrypted: Buffer = client.crypto.encrypt(
  message,
  {
    componentName: "crypto-local",
    keyName: "symmetric256",
    keyWrapAlgorithm: "A256KW",
  }),

(In this case I'm suggesting the use of overloading rather than a separate method, but a separate method could be fine too)

In this case, internally we still use streams, but users don't need to worry about them. Of course this is not efficient especially when encrypting large files, but it removes the pain that it is to work with streams in JS.

[XavierGeerinck]

Thanks for adding your extensive remarks! I had a read on the proposal document as well to try to grasp better on what is happening here.

Your latest remarks is what sums it up for me personal (looking at @shubham1172 and @DeepanshuA their opinion as well): "user's shouldn't worry about how things work". This is the vision of the JS SDK specifically as well, providing value over the normal Dapr APIs to ensure that everything is done for the user behind the scenes.

Looking at your last example, I think this is shaping up to be an API I would love to expose to the users. As I fear that the "streaming" implementation that is being exposed to users will be either to complex or difficult to explain in documentation.

Boiling it down in use cases, this is what I could think of (please feel free to add more use cases):

• Encrypting a Message: Use a cryptographic API to encrypt a message before sending it over the network.
• Verifying Data Integrity: Utilize a cryptographic API to check if the received data has been tampered with during transmission.
• Storing Secure Passwords: Securely store user passwords in a database using a cryptographic API's hash function.
• Generating Digital Signatures: Sign a document or message using a cryptographic API to verify its authenticity.
• Implementing Two-Factor Authentication: Use a cryptographic API to generate and validate time-based one-time passwords (TOTPs) for two-factor authentication.
• Securely Storing User Secrets: Encrypt and store sensitive user information, such as credit card numbers, using a cryptographic API.
• Implementing Secure Communication Channels: Set up secure communication channels (e.g., SSL/TLS) between clients and servers using cryptographic APIs.
• Complying with Data Protection Regulations: Ensure compliance with data protection regulations by encrypting sensitive data at rest using a cryptographic API.
• Implementing Secure Token-based Authentication: Use cryptographic APIs to generate and verify JSON Web Tokens (JWTs) for secure token-based authentication.
• Securing API Communication: Apply cryptographic APIs to secure the communication between different components of an API system, ensuring data privacy and integrity.

Using those, I would like to refrain from having the stream API as the main entry point for users. Thus your example below:

const message: = Buffer.from(...)
const encrypted: Buffer = client.crypto.encrypt(
  message,
  {
    componentName: "crypto-local",
    keyName: "symmetric256",
    keyWrapAlgorithm: "A256KW",
  }),

would definitely be the most interesting for me! Could we add some tests around how this API could be used to support:

1. Small message encryption / decryption
2. File encryption / decryption
3. Stream encryption / decryption

Let me know what you think!

[ItalyPaleAle]

Thanks for your thoughts @XavierGeerinck

Yes, what you list there is correct, but many (most?) of those things require the subtle APIs, which are currently not available.

However, what you say about designing this to be "non-stream" first makes sense. I have a complicated relationship with streams myself, as I both (came to) love them and hate them as they're not very easy to use :)

I can update the APIs. What do you think about making this overloaded? I'm using encrypt as an example but decrypt would mirror this:

// This variant uses buffers
// Buffer = Node.js buffer
// BufferSource = ArrayBuffer or a view (like Uint8Array)
// string = if passed, will be interpreted as UTF-8
encrypt(data: Buffer | BufferSource | string, opts: EncryptRequest): Promise<Buffer>;

// This variant uses streams
// Duplex = duplex streams
encrypt(opts: EncryptRequest): Promise<Duplex>;

The version with duplex streams can be used as:

await pipeline(
  createReadStream("plaintext.txt"),
  // Note "await" is missing here; errors are returned into the stream
  client.crypto.encrypt({
    componentName: "crypto-local",
    keyName: "symmetric256",
    keyWrapAlgorithm: "A256KW",
  }),
  createWriteStream("ciphertext.out"),
);

[XavierGeerinck]

What would you think about making an API spec that can as well future wise merge in the Subtle API?

I.e., based on encrypt and what we pass to it, it will use the correct underlying API

P.S. Let's be careful of method overloading in Typescript as it's not natively as supported as we want it to

So what do we have then?

1. Streaming

Q: Do we need to offer stream in / out? Couldn't we just offer "file-in" and "file-out" or "buffer-in" / "buffer-out"?

2. Messages
3. Files

What about something like the below?

const bufferOut = await client.crypto.encrypt(componentName, msg: string | Buffer, {
    keyName: "symmetric256",
    keyWrapAlgorithm: "A256KW",
    type: "file | message" // <-- default to message, if file, it will take the message string as file name and read from the file
})

Under the hood we then translate to the correct Dapr API and initially don't use the Subtle API

This would ensure that in the future we can adapt it without breaking changes, while still adopting all the use cases

[ItalyPaleAle]

What would you think about making an API spec that can as well future wise merge in the Subtle API?

I.e., based on encrypt and what we pass to it, it will use the correct underlying API

No, the subtle encrypt and high-level encrypt are very different.

• High-level: uses the Dapr Encryption Scheme v1, which is opinionated. Among the various things, it chunks the data and encrypt it chunk-by-chunk (that's how we get streaming support safely), it adds a header, etc. When using this with a key stored in a vault, only the key wrapping is performed in the vault (it uses "envelope encryption" essentially)
• Subtle encrypt: sends the entire message to the vault to be encrypted as-is with the key stored in the vault. If this is a 1GB message, for example, it sends 1GB to the vault (which is inefficient, even when allowed). If the message is larger than a few bytes, this can't always be used (e.g. when using RSA, the max message size is usually a few dozen bytes).

The subtle API is also very low level. Users must know what they're doing. For example, with the high-level API, they just need to tell Dapr "encrypt this, using this key, and the key type is RSA". With the low-level API they need to worry about generating their own nonces/IVs if needed, they need to deal with authentication tags, etc.

I strongly recommend keeping the subtle APIs, when available, in its own namespace. For example crypto.subtle (like JavaScript's own WebCrypto does).

What about something like the below?

I'm ok with passing a readable stream explicitly. But if you don't use a duplex stream, you can't do like:

await pipeline(
  createReadStream("plaintext.txt"),
  client.crypto.encrypt({
    componentName: "crypto-local",
    keyName: "symmetric256",
    keyWrapAlgorithm: "A256KW",
  }),
  createWriteStream("ciphertext.out"),
);

Instead you must do this which is a bit more awkward:

await pipeline(
  client.crypto.encrypt(createReadStream("plaintext.txt"), {
    componentName: "crypto-local",
    keyName: "symmetric256",
    keyWrapAlgorithm: "A256KW",
  }),
  createWriteStream("ciphertext.out"),
);

[XavierGeerinck]

The following API spec might make sense then:

const streamIn = createReadStream("plaintext.txt");
const streamOut = createWriteStream("encrypted.txt");

const out = await client.crypto.encrypt(streamIn, { ...config });

// If stream
streamOut.pipe(out);

// If string or Buffer
const res = out; 

Which then would translate to

async function encrypt(componentName: string, in: string | Readable | Buffer, config): Promise<string | Writable> {
    // Handle each of the datatypes
}

For the subtle API, as agreed on, we will publish this under a separate namespace if it - ever - gets released on the Dapr core encryptSubtle will become available then

Note: updated after syn call with submitter

[ItalyPaleAle]

Code has been updated. The new interfaces are:

interface IClientCrypto {
  encrypt(opts: EncryptRequest): Promise<Duplex>;
  encrypt(inData: Buffer | ArrayBuffer | ArrayBufferView | string, opts: EncryptRequest): Promise<Buffer>;

  decrypt(opts: DecryptRequest): Promise<Duplex>;
  decrypt(inData: Buffer | ArrayBuffer | ArrayBufferView, opts: DecryptRequest): Promise<Buffer>;
}

An example of the usage (using encrypt as example only)

// Using streams
// Here `encrypt` returns a `Duplex` stream that can be used in a pipeline
await pipeline(
  createReadStream("plaintext.txt"),
  await client.crypto.encrypt({
    componentName: "crypto-local",
    keyName: "symmetric256",
    keyWrapAlgorithm: "A256KW",
  }),
  createWriteStream("ciphertext.out"),
);

// Using buffers (or strings)
// Here `ciphertext` is a `Buffer`
const ciphertext = await client.crypto.encrypt(plaintext, {
  componentName: "crypto-local",
  keyName: "symmetric256",
  keyWrapAlgorithm: "A256KW",
});

[XavierGeerinck]

Awesome! Just checking something as well, the componentName is always mandatory right? Would it make sense to have that as the first parameter? We do this for the state store as well (https://docs.dapr.io/developing-applications/sdks/js/js-client/#save-get-and-delete-application-state)

[XavierGeerinck]

Hi @ItalyPaleAle can you check for the tests? Would LOVE to get this in! :)

[ItalyPaleAle]

@XavierGeerinck sorry, i was on vacation till yesterday. I think I fixed the CI failures. I need to look into adding E2E tests, for now there's only a sample

[ItalyPaleAle]

@XavierGeerinck added E2E tests too

[shubham1172]

Ye, just one last thing - could you please add docs as well that go in here https://v1-11.docs.dapr.io/developing-applications/sdks/js/js-client ?

[XavierGeerinck]

Small comments

[XavierGeerinck]

Shouldn't we use !opts?

[ItalyPaleAle]

I like explicitly checking for "undefined" in these cases as it's not ambiguous.

[XavierGeerinck]

Amazing! Merged it!! Thank you for this amazing contribution!

Merge issue