Support parsing pem key only once for crypto api

[tlbdk]

Current API parses the PEM files on every crypto operation, fx:

• https://nodejs.org/api/crypto.html#crypto_class_sign

• node/src/node_crypto.cc

  Line 4168 in 0d22858

  pkey = PEM_read_bio_PrivateKey(bp,

I would be nice if this could be done once and a ref to the openssl key could be retained:

const crypto = require('crypto');

// Read and parse key once
const privateKey = crypto.readPrivateKey(getPrivateKeySomehow());
// privateKey would be a ref to the PEM_read_bio_PrivateKey() pointer

for(let i = 0; i < 10000; i++) {
  const sign = crypto.createSign('RSA-SHA256');
  sign.write('some data to sign' + i);
  sign.end();
  console.log(sign.sign(privateKey, 'hex'));
}

[seishun]

I don't like passing an opaque object to the sign method, and introducing a "parsed key" class would increase complexity. I think it would be a cleaner API to allow passing the PEM key to createSign and allow reusing the resulting Sign object.

[tlbdk]

@seishun This get a bit problematic to use when you are streaming data as what it seems the current API is build for. I'm guessing the hash is being incrementally calculated when doing the 'write' calls and the 'sign' call then encrypts that to provide the signature.

The example below shows a sample where using a shared sign object would give an unexpected result:

const crypto = require('crypto');

// Create shared sign object
const sign = crypto.createSign('RSA-SHA256', getPrivateKeySomehow());

// Start taking requests.
const server = http.createServer((req, res) => {
  req.on('data', (chunk) => {
     // Here we are mixing data between the requests
     sign.write(chunk);
  }).on('end', () => {
     sign.end();
     response.end(sign.sign(privateKey, 'hex'))
});
...

Most other languages have a "parsed key" class as there are many different ways signing can be done, fx. both C# and Java can use different underlying signing implementations fx. when using HSM devices or running on other platforms where the key material is not a available to application. Fx. C# uses windows certificate store on windows, keychain on mac and openssl on Linux. Java use a hardware backed keystore on Android, etc.

Having someone kind of abstraction would make the whole key handling a lot more flexible, fx. imagine using a hardware backed keystore for storage of the private key.

For inspiration here is how the API looks in Java, it's similar to Node with the exception that you pass a key object so it's only constructed once.

PrivateKey privatekey =  customPemToPrivateKey(getPrivateKeySomehow())
for(let i = 0; i < 10000; i++) {
  Signature signature = Signature.getInstance("SHA256withRSA");
  signature.initSign(privatekey);
  signature.update(String.getBytes("some data to sign" + i, StandardCharsets.UTF_8));
  byte[] signatureBytes = signature.sign();
}

And here in C# where the "key" in the focal point of the interface, so the "key" object has a sign function.

SHA256 sha256 = SHA256.Create();
RSACryptoServiceProvider privatekey = customPemToCryptoProvider(getPrivateKeySomehow())
for(let i = 0; i < 10000; i++) {
  byte[] signatureBytes = privatekey.SignData(Encoding.UTF8.GetBytes("some data to sign" + i), sha256);
}

[tniessen]

cc @nodejs/crypto

I'd like to revive the idea. After having worked on the crypto module a lot recently, I think having key objects could actually be advantageous for us:

• Once a key has been created, it is guaranteed to be valid and not malformed.
• Currently, users need to keep the passphrase for encrypted PKCS#8 keys or the unencrypted key in JS-managed memory which isn't optimal from a security point of view. Having key objects would allow us to protect the memory, see crypto.alloc() for encryption key memory management #18896.
• We would finally be able to work with keys stored in engines! (see Support parsing pem key only once for crypto api #15113 (comment))
• A crypto key API would allow users to seamlessly convert between PKCS#1 / SPKI and PKCS#1 / PKCS#8 / SEC1 and even between PEM / DER.
• All existing APIs can be adapted to support key objects.
• The way we currently handle keys is a mess, to be honest, and having key objects could greatly simplify that.
• Key objects would play nicely with the API being designed in crypto: add key pair generation #22660.
• A carefully designed key API would allow users to implement the JWK key format rather easily compared to the current situation.

I am currently trying to assess some API drafts, I'll let you know when I have a proposal.

[tlbdk]

@tniessen this would also enable us to work with keys stored outside the nodejs process, fx. in keychain on mac, certificate store on Windows, GNOME Keyring on linux, TPM (Trusted Platform Module) or HSM's (Hardware Security Module) like Yubikey, TouchID, smartcards, etc.

Fx. if we stick with openssl it can be extended with an engine config: https://developers.yubico.com/YubiHSM2/Usage_Guides/OpenSSL_with_pkcs11_engine.html

[tniessen]

this would also enable us to work with keys stored outside the nodejs process, fx. in keychain on mac, certificate store on Windows, GNOME Keyring on linux, TPM (Trusted Platform Module) or HSM's (Hardware Security Module) like Yubikey, TouchID, smartcards, etc.

@tlbdk Will that require support for custom key objects (user-defined classes)? That would make a secure implementation more difficult.

[tlbdk]

@tniessen No, not if the implementation detail is left to OpenSSL fx using its engine API to add the different HSM backends, it already has the needed abstraction.

There is already API for setting the engine:

https://nodejs.org/api/crypto.html#crypto_crypto_setengine_engine_flags

But we need an API to load keys from engines:

• https://www.openssl.org/docs/man1.1.1/man3/ENGINE_load_private_key.html
• https://github.com/OpenSC/libp11/blob/09b4b9c6253ca5d2868f0685675d91ad98a512c1/tests/rsa-pss-sign.c#L137

fx.

const privateKey = crypto.readEnginePrivateKey("keyid"); // This would just be a wrapper around the openssl key ref 

The key object class could have the same the suggested conversion methods but would just return null when the engine does give access to the underlying key data.

[tniessen]

@tlbdk Awesome, thanks for the clarification.

[tniessen]

@tlbdk I proposed an API in #24234, for now, it does not support engine key objects, but we can definitely add support later once the API has been established.

[tlbdk]

@tniessen Looks really good, thanks for working on this :)

[tniessen]

Key objects have landed. Please let us know whether there is a feature you would like to see that is still missing. (Loading keys from engines is not supported yet.)

[tlbdk]

Would be nice when it's available for the key object and should be fairly simple with the EVP_PKEY_get* methods:
https://www.openssl.org/docs/man1.0.2/crypto/EVP_PKEY_get1_RSA.html
https://www.openssl.org/docs/man1.0.2/crypto/rsa.html#DESCRIPTION

I guess an api to work directly with public and private exponents, etc. by fx creating the RSA object directly with EVP_PKEY_set would also make some things simpler:

EVP_PKEY* pRsaKey = EVP_PKEY_new();
RSA* rsa = RSA_new();
rsa->e = e;
rsa->n = n;
EVP_PKEY_assign_RSA(pRsaKey, rsa);

At the moment I have done the code to convert a JWK to PEM in zero dependency javascript, but parsing PEM without an ASN.1 library would be a lot more work:
https://github.com/connectedcars/node-jwtutils/blob/master/src/jwkutils.js

[tlbdk]

Btw did a quick test with the new key object API for an RSA-SHA256 verification and it about twice as fast as parsing the PEM key on every run, nice work @tniessen :)

Verify key with key load x 21,283 ops/sec ±1.02% (85 runs sampled)
Verify key with key object x 41,119 ops/sec ±0.92% (82 runs sampled)

[panva]

@tniessen Brilliant work, thank you!

Please let us know whether there is a feature you would like to see that is still missing.

At the moment, after obtaining a PrivateKeyObject/PublicKeyObject, apart from knowing if it's public/private and rsa/ec/dsa we know little about it. Having access to the key's components we'd be able to determine its supported sign/verify/key derivation algorithms.

Hence i'd like to see the ability to

• export/access the different key components such as public and private exponents, prime values, and the like for EC keys (curve name) etc. Essentially access to the parsed asn.1 tags and values.
• get a PublicKeyObject instance out of PrivateKeyObject one

This would enable native JOSE (JWK, JWS, JWA, etc) implementation with no dependencies and the possibility to create queryable key sets to be able to get a key from an imported key set (such as authorization server's or OAuth2.0 client's jwks_uri) with specific parameters and key operations available.

Libraries capable of this, e.g. node-jose rely heavily on pure JS implementation like forge to do key format conversion in order to pass the keys to asn.1 parsers (again pure JS) to get this information, making the whole process heavy on dependencies and slow as well.

(sorry for re-posting my reply)

[davidgtonge]

I 100% agree with @panva.
Getting access to the raw asn.1 tags and values would be massively helpful and enable a big reduction in the number of dependencies needed to support JOSE (JWK, JWS, JWA, JWT) and OpenID Connect implementations.

[tniessen]

Thank you for your kind feedback! I have been discussing ways to access the individual properties with @sam-github. I believe our preferred way of doing this would be via a fields property of the key object which would expose the fields of the key. A similar construct could be used to create key objects from these fields. However, there are some challenges in integrating this into Node.js (e.g., due to API differences between crypto and TLS keys), so it won't happen over night.

[tlbdk]

@panva @davidgtonge, I don't think exposing the undelaying ASN.1 tags would be the best option as this is very format dependent, fx if the key is loaded from an engine or if support for loading JWK keys directly is added in the future. Having a fields property fx with generic names inspired by JWK would be a nicer solution.

@tniessen Would the OpenSSL key API not help with this, it seems to be fairly little code? And why would it be linked to TLS certificate handing(note I have not looked at the code so I might just be missing something obvious)?

That node-jose is dependencies heavy and slow is not because of limitations in Node or a missing key object API, fx. we are doing all the needed JOSE stuff to integrate with Google and Microsofts OpenID Connect providers down to Node.js 6.x with zero dependencies, our IdP only has two extra dependencies to convert PEM keys to JWK, but if we were using JWK's as base it would be zero.

For PEM to JWK:
https://github.com/dannycoates/pem-jwk
https://github.com/PeculiarVentures/asn1.js (only dependency for pem-jwk)

We have some sample code that shows how to do this with very little code:

SP for Azure AD that loads the keys from an openid configuration endpoint:
https://github.com/connectedcars/node-jwtutils/tree/master/sample/azureadauth

SP for Google that loads keys from googles JWK endpoint:
https://github.com/connectedcars/node-jwtutils/tree/master/sample/googleoauth2v2

POC IdP that's missing the key export parts, but that would not be very much code:
https://github.com/connectedcars/node-jwtutils/tree/master/sample/simpleidp

[panva]

@tlbdk if JWK format was native I'd agree, to some extent. But it isn't and is likely further out than being able to access the key's underlying asn.1 - which at least internally will be needed to support pem to JWK conversion anyway, won't it?

Looking even at the samples i don't see any of them being zero dependency, the pem > jwk and vice versa route is there as dependency, this brings a considerable amount of transitive dependencies that we're looking to avoid. (while pem-jwk only has one direct (more transitive), there's very limited encoding and key type support in that one, so something like @trust/keyto which is as small as it gets, with wider encoding and key type support, you're looking at 3 direct, 8 more transitive)

we are doing all the needed JOSE stuff to integrate with Google and Microsofts OpenID Connect

"all needed" to do simple client integration is just a small portion of JOSE

The moment you step out of a simple one way integration, e.g. clients that produce signed assertions, encrypt request content for the provider, decrypt content from the provider, or providers needing to nail down a key to use from a loaded keystore with specific RS bitsize or EC curve, having access to keyobject's properties is a godsend.

The big target here is indeed native JWK support, so let's maybe shoot for that one

• crypto.createPrivateKey accepting a JWK
• crypto.createPublicKey accepting a JWK
• keyObject.export supporting JWK format

but at the same time being able to get a PublicKeyObject instance out of PrivateKeyObject one.

[tlbdk]

@panva Right now Node.js is using OpenSSL to do the PEM parsing and that internally is doing the ASN.1 decoding to a struct that has the key components, the struct can also manually be populated or read into for any other format/engines OpenSSL supports. The API's I linked gives access to the struct so it should be fairly simple to expose that in the key object. But I have not seen an API's exposing the ASN.1 parts without having to decode the same key twice, I did not look that hard so it might be there. My point was just that ASN.1 is an encoding detail better to not expose in an API.

Generating PEM without dependencies is fairly easy if you have the key components. Adding support for converting JWK to PEM in crypto.createPrivateKey or crypto.createPublicKey would be super easy. Would be happy to do a pull request that does exactly that if it would be accepted, @tniessen ? The pretty option would be populating the OpenSSL key struct directly in C world.

jwkutils.js should support all the key formats in JWK, am I missing any?

That would sort:

• crypto.createPrivateKey accepting a JWK
• crypto.createPublicKey accepting a JWK

For exporting to JWK I guess with a bit of cleanup we could get it down to 2-3 dependencies with an ASN.1 library or zero with only doing enough of the ASN.1 decoding to support PEM, but it would be quite a bit of work. Key components from the OpenSSL struct would make this so much easier.

It's a very small portion of JOSE I have actually seen used in the wild so that was my definition of "needed" for now :).

Deriving a PublicKeyObject from PrivateKeyObject would be useful and also calculating the public key fingerprint would also be nice.

[tniessen]

Would the OpenSSL key API not help with this, it seems to be fairly little code?

It certainly does, the implementation itself is quite simple. The API design is what we should consider carefully.

And why would it be linked to TLS certificate handing(note I have not looked at the code so I might just be missing something obvious)?

TLS already provides ways to access certain properties of the key, but I think @sam-github and I agreed that the property names being used there are not necessarily elegant choices, so I'd be okay with breaking compatibility there.

get a PublicKeyObject instance out of PrivateKeyObject one

This is on my list of upcoming features, I am trying to split them into small incremental changes to make our internal processes easier. Eventually, createPublicKey will accept a key object with type private.

[sam-github]

I think if we attach the key properties to a .fields sub-object as @tniessen suggests, that we can apply that to TLS certificates as well as to crypto key objects. The top-level field names in TLS certificate objects would become effectively legacy.

Its worth bike-shedding the name since it will live a long time. I mildly lean towards .info instead of .fields (as in SubjectKeyInfo from the ASN.1). Maybe the property names can also come from the ASN.1, I think they are already camelCase.

[davidgtonge]

So I agree with @panva that in an ideal world it would be great to get core JWK support. I think node.js and JWK go well together.

For the majority of use-cases having the features mentioned above:

crypto.createPrivateKey accepting a JWK
crypto.createPublicKey accepting a JWK
keyObject.export supporting JWK format

Would remove the need to even access a .fields sub-object.

Having the key properties in a .fields sub-object would make exporting to a JWK fairly easy to do in userland, but I still think that it should belong in core.

I mildly lean towards .info instead of .fields

.info gets my vote as well.

Having a fields property fx with generic names inspired by JWK would be a nicer solution.

I have to say I agree with @tlbdk . At the keyObject level I shouldn't care whether the key was encoded with pkcs1 or spki. The different encodings result in different ASN.1 sequences.

[panva]

I have to say I agree with @tlbdk . At the keyObject level I shouldn't care whether the key was encoded with pkcs1 or spki. The different encodings result in different ASN.1 sequences.

Agreed, generic field names make more sense.

[tniessen]

@sam-github @panva @tlbdk Would you expect BIGNUM fields to be exposed as hexadecimal strings or as JS bigints?

[tlbdk]

@tniessen I would say Bigint for output and Bigint and Buffer for input

[panva]

@tlbdk @tniessen

I would say Bigint for output and Bigint and Buffer for input

Agreed

[sam-github]

Using JWK names is insteresting, if possible. They are described in rfc7518 if anyone is looking (took me a while to find them).

If we go that way, wouldn't it make sense to have the property values be JSON compatible as they are in JWK, so they would be hex strings on output, and bigint/buffer/hexstrings on input?

[panva]

@sam-github if the properties follow jwk names then the same format would indeed make more sense.

[sam-github]

@tniessen IIRC you thought there was some difficulty with JWK support, was it around the key usage fields? They all appear optional to output, and node would have to ignore them on input.

[panva]

Key usage fields (key_ops and use) are optional use by a given application. Node just needs to ignore all fields that aren’t the actual key components.

[tniessen]

IIRC you thought there was some difficulty with JWK support, was it around the key usage fields?

That and the problem of Bigint support in JWK.

[panva]

@tniessen what issue with bigint support? AFAIK all key material is encoded as base64url with no padding Base64urlUInt

[tniessen]

@panva I mean that Node.js should optimally output bigints, but JWK does not seem to support that (since JSON does not).