SSL/TLS handshake failure in CI-Example (ra-tls-mbedtls)

[ishaangandhi]

I am trying to run the example in gramine/CI-Examples/ra-tls-mbedtls.

Per the README, I ran the following:

make clean
make app dcap RA_TYPE=dcap

gramine-sgx ./server &

/usr/local/bin/gramine-sgx-sigstruct-view server.sig

Based on the output of the sigstruct, I ran the following:

RA_TLS_ALLOW_DEBUG_ENCLAVE_INSECURE=1 \
RA_TLS_ALLOW_OUTDATED_TCB_INSECURE=1 \
RA_TLS_MRSIGNER=5d99138bf73a9adea36ee583ae2b5e27d8076a854e4923824a91067f45aacc8f \
RA_TLS_MRENCLAVE=ee7ac98721908dc071d1c5bcd35ae94623d4b09512706eed1bd6691b36c0acda \
RA_TLS_ISV_PROD_ID=0 \
RA_TLS_ISV_SVN=0 \
./client dcap

However, I get the following error:

  . Performing the SSL/TLS handshake...ra_tls_verify_callback: extract_quote_and_verify_pubkey failed: -9472
 failed
  ! mbedtls_ssl_handshake returned -0x2500
  ! ra_tls_verify_callback_results:
    attestation_scheme=2, err_loc=2, 
    dcap.func_verify_quote_result=0x0, dcap.quote_verification_result=0x0

Last error was: -9472 - X509 - The extension tag or value is invalid

I see that err_loc=2 means quote verification failed and that -0x2500 means that the x.509 certificate had an invalid extension, but I'm unsure how to debug further from here. The Gramine version was pulled from GitHub and built this morning, and I am running on Ubuntu 20.04.

Any help would be appreciated. Thanks!

[ishaangandhi]

Is something wrong with ssl/ca.crt, which gets generated?

[dimakuv]

That's weird. I'm not sure what happens, because you seem to be using the same version of Gramine for both server and client, and the error you see seems to indicate the mismatch in versions...

If you want to look at the nitty-gritty details, check the code:

• gramine/tools/sgx/ra-tls/ra_tls_verify_dcap.c

  Line 129 in 73c8e93

  ret = extract_quote_and_verify_pubkey(crt, &quote, &quote_size);

• gramine/tools/sgx/ra-tls/ra_tls_verify_common.c

  Line 212 in ce7556d

  int extract_quote_and_verify_pubkey(mbedtls_x509_crt* crt, sgx_quote_t** out_quote,

The best way forward for you would be to start the server and the client in two terminals (so that their outputs are not intermingled) and run the commands again, and carefully observe what the server and the client print out. Ideally also add the trace Gramine log level.

I don't think this problem has anything to do with the ssl/ca.crt certificate. The problem seems to be with the ephemeral certificate that's create on-the-fly inside the SGX enclave.

[ishaangandhi]

the same version of Gramine for both server and client

Also, per the README, the client isn't supposed to be run in gramine, you just run ./client. Is that not correct?

See README:

gramine/CI-Examples/ra-tls-mbedtls/README.md

Lines 109 to 125 in 73c8e93

	```sh
	make clean
	make app dcap RA_TYPE=dcap
	gramine-sgx ./server &
	RA_TLS_ALLOW_DEBUG_ENCLAVE_INSECURE=1 \
	RA_TLS_ALLOW_OUTDATED_TCB_INSECURE=1 \
	RA_TLS_MRENCLAVE=<MRENCLAVE of the server enclave> \
	RA_TLS_MRSIGNER=<MRSIGNER of the server enclave> \
	RA_TLS_ISV_PROD_ID=<ISV_PROD_ID of the server enclave> \
	RA_TLS_ISV_SVN=<ISV_SVN of the server enclave> \
	./client dcap
	# client will successfully connect to the server via RA-TLS/DCAP flows
	kill %%
	```

[ishaangandhi]

Thanks, dimakuv! Based on your recommendations, I did the following:

• Set GRAMINE_LOG_LEVEL = trace
• Ran in terminal window 1:

make clean && DEBUG=1 make app client dcap RA_TYPE=dcap && gramine-sgx ./server

• Ran in terminal window 2:

RA_TLS_ALLOW_DEBUG_ENCLAVE_INSECURE=1 RA_TLS_ALLOW_OUTDATED_TCB_INSECURE=1 RA_TLS_MRSIGNER=5d99138bf73a9adea36ee583ae2b5e27d8076a854e4923824a91067f45aacc8f RA_TLS_MRENCLAVE=1f119dcea98f4365f3985911bbc02135b3f1929667f3d26a272d563429aacea2 RA_TLS_ISV_PROD_ID=0 RA_TLS_ISV_SVN=0 ./client dcap

The SGX trace doesn't look super informative to me, but I attached it, anyways: sgx-trace.txt

The message on the client side changed slightly, even though I made no changes to the program:

[ using default SGX-measurement verification callback (via RA_TLS_* environment variables) ]

  . Seeding the random number generator... ok
  . Connecting to tcp/localhost/4433... ok
  . Setting up the SSL/TLS structure... ok
  . Loading the CA root certificate ... ok
  . Installing RA-TLS callback ... ok
  . Performing the SSL/TLS handshake...ra_tls_verify_callback: sgx_qv_verify_quote failed: 57372
 failed
  ! mbedtls_ssl_handshake returned -0x3000
  ! ra_tls_verify_callback_results:
    attestation_scheme=2, err_loc=3, 
    dcap.func_verify_quote_result=0xe01c, dcap.quote_verification_result=0xa006

Last error was: -12288 - X509 - A fatal error occurred, eg the chain is too long or the vrfy callback failed

Tracing that error number in the code, I see it could either occur because sgx_qv_get_quote_supplemental_data_size returned non-zero, or the time() system call returned -1. I'm not sure why either would happen.

[dimakuv]

The 0xe01c error in DCAP stands for SGX_QL_QUOTE_CERTIFICATION_DATA_UNSUPPORTED. See this file: https://download.01.org/intel-sgx/latest/dcap-latest/linux/docs/Intel_SGX_ECDSA_QuoteLibReference_DCAP_API.pdf

Unfortunately, the description of this error code is pretty vague:

The quote verifier doesn’t support the certification data in the Quote. Currently, the Intel QVE only supported CertType = 5.

What is the platform you're trying this example on? If it's your local machine, did you install the PCCS and the whole SGX DCAP infrastructure correctly? If it's Azure VM, did you install the Azure-specific libraries correctly?

[ishaangandhi]

I am running on an Azure VM, and yes, I think I installed the Azure-specific libraries correctly (az-dcap-client). Does that error code indicate only an issue with PCCS and DCAP?

[dimakuv]

@ishaangandhi Please note that Azure VM deployments and bare-metal deployments have different requirements.

In Azure VM deployments, the PCCS server is already installed (by MS Azure) and reachable, so you don't need to install anything PCCS-related. The only requirement is to install the az-dcap-client package.

For some more context, please read the README here carefully: https://hub.docker.com/r/gramineproject/gramine

You could also ask the MS Azure team what this error could mean. Unfortunately, I don't recall seeing such errors at all.

[ishaangandhi]

Gotcha, thank you. For some reason, when I re-ran the instructions, they worked. I wonder if I just needed a new shell window or something like that. Either way, thank you, and I'll close the issue now!