[Pal/Linux-SGX] RFC: Support for Microsoft Azure Attestation

[dimakuv]

Microsoft Azure Attestation (MAA)

Microsoft Azure cloud provides its own service to attest enclaves, called Microsoft Azure Attestation (MAA), or simply Azure Attestation. MAA is a generic service that works on different TEEs, but we concentrate only on Intel SGX enclaves here.

Concepts and definitions

MAA contains several concepts. For purposes of integration with Gramine, we need to understand the following concepts:

• Attestation provider -- actual web-based REST service that has several endpoint APIs: to attest the SGX enclave (attest/), to get/set signing certificates for attestation tokens (certs/), to get/set/reset attestation policies (policies/), to get/set signing certificates for attestation policies (certificates/). We only care about the first two endpoint APIs (attest/ and certs/) here.
  • Regional shared provider -- an existing default attestation provider located in one particular region and shared between all users. It is convinient for testing, as it provides default policies and doesn't need any configuration.
  • Custom provider -- user-created attestation provider that has user-defined policies, certificates, etc. More powerful than the regional shared provider, but requires careful configuration.
• Attestation request -- JSON string sent by Gramine to attestation provider. Embeds the SGX Quote + EnclaveHeldData. EnclaveHeldData is basically the sgx_quote.sgx_report.user_report_data.
• Attestation token / attestation response -- JSON string containing the signed-by-MAA JSON Web Token (JWT). This JWT contains the claims about the attested SGX enclave, for example "x-ms-sgx-is-debuggable": true and "x-ms-sgx-mrsigner": <SGX enclave msrigner value>.
  • The JWT is signed by the attestation signing certificate. This is a self-signed certificate with subject name matching the AttestUri element of the attestation provider. This certificate can be found in the set of certificates returned via the REST endpoint certs/, see here for details.
  • Some context on JWTs: a JWT is a string in the format xxx.yyy.zzz where xxx is the JWT header, yyy is the JWT payload and zzz is the JWT signature. All three parts are encoded in Base64Url. For the case of RS256 algorithm, the signature looks like this: RSA(SHA256(base64UrlEncode(header) + "." + base64UrlEncode(payload)), rsa-private-key) and the signature verification looks like this: RSA(SHA256(base64UrlEncode(header) + "." + base64UrlEncode(payload)), rsa-public-key).

And we can ignore the following concepts:

• Attestation policy -- users can load their own policies into their own Attestation Provider instances. An example of a policy is "I want to allow only non-debug SGX enclaves that have an SVN number greater than 5; if the SVN number is equal to 4 or 5, I want the output SGX attestation token to contain a claim 'super-secure version'".
• Attestation policy signing -- attestation policies may be unsigned, but then they can be tampered by anyone who has access to the Attestation Provider where the policy is loaded into. To prevent this, users can restrict who can modify/update the policy by supplying certificates and signing policies with these certificates' public keys.
• Claims and claim rules -- these are the specifications on the language in which attestation policies are written. Since we ignore attestation policies in this description, we ignore the claims as well.

We ignore attestation policies because they are set up by users before deploying Gramine SGX enclaves. This setup happens through other means -- typically through the web-based Azure Portal or CLI-based az tool. In other words, management of attestation policies is orthogonal to Gramine execution.

MAA flows

Here is a diagram that illustrates the flows, taken from the Intel SGX enclave validation workflow docs of MAA:

One important point here is that the Client and the Relying Party can be the same entity (located on the trusted remote-user machine). This is the "deployment" that we will assume for Gramine, because it is similar to how Gramine currently performs IAS EPID attestation: the enclave machine sends the SGX quote to the relying party/client, and the latter sends this SGX quote in the attestation request to IAS and receives back the attestation response.

Another important point here is the OpenID Metadata Endpoint in step 4. This is actually the certs/ REST endpoint of the attestation provider. What happens in this step is actually:

1. The relying party/client sends the GET https://mytenant.attest.azure.net/certs HTTPS request. Note that the relying party/client must start a TLS (HTTPS) connection with the attestation provider, and verify that the X.509 cert chain of the attestation provider ends up in the root Microsoft Azure certificate.
2. The response to this HTTPS request contains a JSON string with a set of keys. Each key has a field x5c -- a JSON array of base64url-encoded DER certificates. In the end, one of the returned certificates must be the attestation signing certificate (that signed the JWT).

Comparison with Intel Attestation Service (IAS)

The MAA attestation flow is similar to the EPID (IAS) attestation flow:

1. The SGX enclave generates the evidence locally. In both the EPID and the MAA cases, the evidence is the SGX quote. The evidence is then sent to the remote relying party/client. The latter initiates the validation process by talking to the internet service, see the next three steps.
2. The relying party/client sends the evidence (the SGX quote) to the internet service:
   • In the EPID case, the internet service is IAS. The interface is REST API, using the /attestation/v3/report endpoint. See the details here.
   • In the MAA case, the internet service is the attestation provider. The interface is REST API, using the attest/ endpoint. See the details here.
3. The internet service replies to the relying party/client with the attestation response:
   • In the EPID case, IAS replies with the Attestation Verification Report -- a JSON string in a special format. See the details here.
   • In the MAA case, the attestation provider replies with the attestation token (JWT). See the details about it above.
4. The relying party/client must verify the received attestation response, by getting the public key of the root certificate of the certificate chain that signed the attestation response (somehow off-band):
   • In the EPID case, the Attestation Verification Report is sent as part of the TLS connection with the IAS, which root certificate's public key is hard-coded in the relying party/client, see Gramine example.
   • In the MAA case, the situation is more complicated:
     • The JWT's signature must be verified by the public key contained in one of the attestation signing certificates obtained through the certs/ REST endpoint.
     • The certs/ endpoint is accessed via the GET request and returns a set of keys.
     • Each of these keys has a string field kid. The JWT also has a string field kid. The JWT is signed with the key that has the matching kid.
     • When the matching key is found among keys, the first certificate in the x5c array of certificates should be extracted, base64-decoded, and it will be a properly formatted self-signed X.509 certificate.
     • We should extract the public key from this self-signed X.509 certificate and use this key to verify the signature of the JWT.

Example from Edgeless EGo

Edgeless EGo has a nice example that shows yet another deployment for MAA:

I do not like this deployment: it requires that the SGX enclave not only generates the SGX Quote, but also that it establishes an HTTPS connection to the MS Azure attestation provider. Implementing this case in Gramine would add many dependencies: curl with HTTPS, a JSON parser, maybe a JWT/JWK parser. This unnecessarily increases the TCB.

Proposed changes for Gramine

This is my vision for MAA integration:

                                                      MAA attestation provider
                                                   +------------------------------------------------+
                                                   |                                                |
                                                   |  +--------------------+        +-------------+ |
                                                   |  | `certs/` endpoint  |        |  `attest/`  | |
                                                   |  |                    |        |  endpoint   | |
                                                   |  +---------------+----+        +---------+---+ |
                                                   |                  ^                       ^     |
                                                   +------------------------------------------------+
                                                                      |                       |
   SGX enclave                                                        +------------------+    |
+------------------------------------------+                              set of JWKs    |    |
|                                          |                                             |    | JWT
|              +-----------------------+   |                         +--------------+    |    |
|              |                       |   |                         |              |    |    |
|              |      Application      |   |                         |  Client app  |    |    |
|              |                       |   |                         |              |    |    |
|              +-----------+-----------+   |                         +-------+------+    |    |
|                          |               |                                 |           |    |
|                          v               |                                 v           v    v
|              +-----------+-----------+   |     RA+TLS X.509        +-------+-----------+----+--+
|              |  libra_tls_attest.so  |   |     certificate         |  libra_tls_verify_maa.so  |
|         +--->+     (unmodified)      +---------------------------->+          (new)            |
|         |    +-----------+-----------+   |                         +---------------------------+
|         |                |               |
|    SGX  |                v               |                         New RA+TLS env vars:
|   quote |    +-----------+-----------+   |
|         |    |                       |   |                         + RA_TLS_MAA_PROVIDER_URL
|         |    |       Gramine         |   |
|         +----+     (unmodified)      |   |                         + RA_TLS_MAA_PROVIDER_API_VERSION
|              |                       |   |
|              +-----------------------+   |
|                                          |
+------------------------------------------+

My proposal shifts all burden of communication with Microsoft Azure Attestation (with the attestation provider's attest/ and certs/ endpoints) to the relying party/client.

This means that Gramine itself (the SGX enclave) only does the same thing as it did before: generates the SGX quote.

Similarly, the RA-TLS "attest" library inside the SGX enclave does the same thing as it did before: stashes the generated SGX quote into the generated self-signed X.509 certificate, and uses this certificate for the TLS handshake with the relying party/client.

Therefore, we don't need any changes to the core Gramine, and we add a new "verify" library to RA-TLS. For the context, this is how the RA-TLS libraries look today:

gramine/built-release/lib/x86_64-linux-gnu$ ll libra_tls*
libra_tls_attest.so*              # runs inside SGX enclave, generates the SGX quote + X.509 RA-TLS cert
libra_tls_verify_epid.so*         # runs in relying party (no SGX), talks to IAS to verify the SGX quote
libra_tls_verify_dcap.so*         # runs in relying party (no SGX), talks to the Datacenter Caching Service to verify the SGX quote
libra_tls_verify_dcap_gramine.so* # same as above, but can run in SGX enclave

And with my proposed changes, we will add a new library:

libra_tls_verify_maa.so*          # runs in relying party (no SGX), talks to MAA's attestation provider

Implementation steps

• Replace sgx.remote_attestation = true with sgx.remote_attestation = "none" | "epid" | "dcap"
  • (don't forget to support the deprecated version!)
  • Done in Introduce sgx.remote_attestation = "[none|epid|dcap]" #638
• In LibOS, introduce a pseudo-file /dev/attestation/type that will return one of none, epid or dcap
  • Currently the app has no way to figure out the attestation type
  • The type can be a new string field in g_pal_public_state, populated during SGX PAL init and read by LibOS
  • Done in Introduce sgx.remote_attestation = "[none|epid|dcap]" #638
• Modify RA-TLS:
  • Add a new library libra_tls_verify_maa.so to RA-TLS
    • (note that the libra_tls_attest.so library that executes inside the SGX enclave is left unmodified)
    • Done in [Pal/Linux-SGX] Add MAA to RA-TLS and SecretProv libs #652
  • Add new envvars:
    • RA_TLS_MAA_PROVIDER_URL (no default, must be provided)
    • RA_TLS_MAA_PROVIDER_API_VERSION (default is 2020-10-01)
    • these two envvars are used in POST attest/ request
    • the first envvar is also used in GET certs/ request
    • Done in [Pal/Linux-SGX] Add MAA to RA-TLS and SecretProv libs #652
• Expand the example ra-tls-mbedtls to support MAA mode
  • Done in [Pal/Linux-SGX] Add MAA to RA-TLS and SecretProv libs #652
• Expand the example ra-tls-secretprov to support MAA mode
  • Done in [Pal/Linux-SGX] Add MAA to RA-TLS and SecretProv libs #652
• Add documentation on MAA and its implementation in RA-TLS into https://gramine.readthedocs.io/en/latest/attestation.html
  • Done in [Pal/Linux-SGX] Add MAA to RA-TLS and SecretProv libs #652

Testing on MS Azure

To test MAA, we need a proper MS Azure subscription. After signing in with this subscription, we should prepare the SGX environment like this:

1. Install the SGX CC VM with Ubuntu 18.04, see this tutorial. If inside the Intel network, follow a special internal guide to create such a CC VM.
   • Note that there is no need to install Azure DCAP client (az-dcap-client) manually, it will be installed together with OpenEnclave in the next step.
2. Install OpenEnclave, see this tutorial.
3. Try classic DCAP Attestation examples in OpenEnclave, see this README.
4. Try the official MAA example with OpenEnclave, see this README. This will require the installation of .NET 5.0 SDK.
5. Install latest Go (Ubuntu 18.04 has a too-old version); this is required for the EGo example below. See this web-page.
6. Install EGo from Edgeless, see this tutorial.
7. Try the MAA example with EGo, see this README.

I tested this on Azure CC VM and Ubuntu 18.04. For the first step, I followed the Intel internal guide to create CC VMs. For step 5, I had to slightly change the PATH setting: export PATH=/usr/local/go/bin:$PATH, otherwise Ubuntu picked the system-wide installed old version of Go. The rest was easy.

• The example from OpenEnclave uses the sharedcus.cus.attest.azure.net regional shared attestation provider.
• The example from EGo uses the shareduks.uks.attest.azure.net regional shared attestation provider.
• There are also examples from SCONE and Occlum, but I didn't like them and didn't test them.

Useful links

Official Microsoft documentation:

• Microsoft Azure Attestation -- more of a marketing page
• Microsoft Azure Attestation is now generally available -- blog post with MAA announcement
• Microsoft Azure Attestation docs -- official documentation
• Microsoft Azure Attestation REST API reference -- the exact details on the "submit SGX evidence and receive SGX token" MAA endpoint

MAA example code:

• Sample code for Intel® SGX Attestation using Microsoft Azure Attestation -- oficial examples from Microsoft Azure
• Integrations to Microsoft Azure built in SCONE platform -- SCONE integration tutorial, stale and not well-written
• Go remote attestation sample using Microsoft Azure Attestation -- Edgeless integration code, really nice, can serve as a code reference

Previous attempts

I experimented with MAA almost two years ago, but that effort was abandoned for several reasons:

• We had higher priority tasks to work on,
• MAA was still in its infancy, and a lot of details changed since then.

Here are the relevant issues/PRs from those times:

• [Pal/Linux-SGX] Add interoperability with Microsoft Azure Attestation graphene#1791
• [Pal/Linux-SGX] Add support for Microsoft Azure Attestation to RA-TLS graphene#1793

Some other related issues:

• Quote verification with Microsoft Azure Attestation Service #537

[monavij]

I like this proposal as it keeps attestation library preloaded in Gramine very simple where its sole job is to send the attestation evidence (quote) offer an RA-TLS channel to trusted Relying party. For full solution we will likely need to integrate with AKV as I can see that the next step after verification is to extract the secret from AKV using the token and sending to Gramine over RA-TLS channel. I guess that can be a new sample secret provisioning library.

While this works, I wonder if MSFT plans to provide a sample relying party/client implementation? Otherwise each of the LibOS implementation will have their own sample Relying party code.

[thomasten]

For the RA-TLS use case, it makes perfect sense to me to have both the Client and the Relying Party role in the verify lib. All assumptions taken here seem correct to me.

If you ever want Gramine to directly provide the JWT attestation token (e.g., developers should be able to write Relying Parties without requiring your libs), you can probably move the Client code to the enclave host, so that you don't have to add dependencies within the enclave or increase its TCB.

[dimakuv]

Some updates

• I added a section to the main issue on Testing on MS Azure. Testing was successful, I didn't encounter any particular issues.

Reply to @monavij

For full solution we will likely need to integrate with AKV as I can see that the next step after verification is to extract the secret from AKV using the token and sending to Gramine over RA-TLS channel. I guess that can be a new sample secret provisioning library.

Yes, this is exactly my plan. It actually fits nicely with the current Gramine libs:

• RA-TLS library will incorporate MS Azure Attestation (MAA)
• Secret Prov library will incorporate MS Azure Key Vault (AKV)

While this works, I wonder if MSFT plans to provide a sample relying party/client implementation? Otherwise each of the LibOS implementation will have their own sample Relying party code.

There is this sample implementation: https://github.com/azure-samples/microsoft-azure-attestation/. It has the samples for OpenEnclave and Intel SGX SDK. However, these samples are written in a rather ad-hoc way and in C# language.

Reply to @thomasten

If you ever want Gramine to directly provide the JWT attestation token (e.g., developers should be able to write Relying Parties without requiring your libs), you can probably move the Client code to the enclave host, so that you don't have to add dependencies within the enclave or increase its TCB.

This sounds cumbersome to implement for these reasons:

1. Even if we move the Client code to the untrusted SGX PAL (which communicates with the attestation provider and parses the received JWT attestation token), we still need to link the untrusted SGX PAL against libcurl (for HTTPS) and libjson (for JSON parser). This doesn't sound good, especially the Curl dependency.
2. This will require "piercing through" all layers of Gramine: the LibOS layer will need to provide a new pseudo-file /dev/attestation/jwt, the PAL layer will need to provide a new API DkAttestationToken(), the trusted PAL will need to provide a new OCALL ocall_get_attestation_token(). Too many changes to support a single cloud-vendor-specific feature.

We could add this in the future, when JWT becomes a real standard. But for now we will try the path of no-modifications to core Gramine, when we add support for each new cloud-vendor-specific attestation scheme. After all, the application on top of Gramine can implement all this logic -- get the SGX quote via /dev/attestation/quote, wrap it into the MAA attestation request, do the communication with MAA, parse the resulting JWT token.

[dimakuv]

Now that we have three different schemes, replace sgx.remote_attestation = true with sgx.remote_attestation = "none" | "epid" | "dcap" | "maa"

This is wrong. We don't need to add maa here. MAA is a higher-level flow that doesn't need to be known by the core Gramine. Only RA-TLS (which executes on top of Gramine) needs to be aware of the MAA flows.

TODO: edit the root comment here. UPDATE: done.

[dimakuv]

TODO: I was wrong about the certs/ API endpoint. Edit the root comment to reflect how and why it should be used.

UPDATE: done.

[aneessahib]

@dimakuv
So the client side would perform the regular RA-TLS flow, correct? And will the client side ever know (or need to know) what the attesting backend is? (MAA or DCAP) .

[dimakuv]

So the client side would perform the regular RA-TLS flow, correct?

No, the client side performs an MAA flow, as part of the RA-TLS flow. So, during RA-TLS handshake, the client side extracts the SGX quote and sends it to MAA for verification and gets back from MAA the attestation token.

And will the client side ever know (or need to know) what the attesting backend is? (MAA or DCAP) .

The attesting backend (the Gramine SGX enclave) is never an "MAA type". There is no such thing as an "MAA SGX quote". The attesting backend generates the classic DCAP SGX quote -- this is what the MAA attestation scheme uses (it was always like that). So, the attesting backend has no changes at all, it just generates the classic DCAP SGX quote, as it used to do. The attestating backend knows nothing about MAA at all.

[aneessahib]

Apologies- i mixed up the terminology. I meant to say - the host enclave - performs a regular RA-TLS flow unaware of what flow the verifier is going to take (MAA or directly use DCAP libs).

[dimakuv]

Apologies- i mixed up the terminology. I meant to say - the host enclave - performs a regular RA-TLS flow unaware of what flow the verifier is going to take (MAA or directly use DCAP libs).

Yes, this is exactly correct.

[dimakuv]

I added a P0 label simply because the PR is ready (gramineproject/contrib#38) and should be merged for the next release of Gramine.