Gramine runtime does not build dcap on debian #213

szymek156 · 2024-08-27T22:13:31Z

Any particular reason for enabling dcap only for Ubuntu?

https://github.com/gramineproject/gsc/blob/master/templates%2FDockerfile.common.compile.template#L33

I removed if statement locally and graminized debian image. Building worked, but during execution I get the error that some Intel sgx library is missing (don't remember exactly the name, I am on mobile now).

I assume in Dockerfile, the build step of installing Intel's sgx packages, misses attestation verification package.

From git blame, seems like the if statement I removed for my local experiment should be more like:
'if not centos then dcap=enable'??

szymek156 · 2024-08-27T22:18:08Z

I think this one:
https://github.com/gramineproject/gsc/blob/master/templates%2Fdebian%2FDockerfile.compile.template#L45

Is missing in BUILDING docker template.
Graminized container should give access to this package if dcap is enabled. I think 🤔?

dimakuv · 2024-08-28T08:16:06Z

Any particular reason for enabling dcap only for Ubuntu?

From git blame, seems like the if statement I removed for my local experiment should be more like: 'if not centos then dcap=enable'??

Yes, I agree, there is no reason to not enable DCAP on Debian. I don't think there is any explanation for Debian omission, just a historical bug.

I removed if statement locally and graminized debian image. Building worked, but during execution I get the error that some Intel sgx library is missing (don't remember exactly the name, I am on mobile now).

I assume in Dockerfile, the build step of installing Intel's sgx packages, misses attestation verification package.

Could you tell us the exact error and what exactly you ran?

In Dockerfiles, we actually have the installation of the required SGX packages, you found it yourself:

gsc/templates/debian/Dockerfile.compile.template

Line 45 in fcf9654

    
           && env DEBIAN_FRONTEND=noninteractive apt-get install -y libsgx-dcap-quote-verify-dev

. The packages installed in this Dockerfile.compile should have propagated to the final Docker image, but now I think -- maybe they don't? I agree that those SGX packages should be (also) installed in Dockerfile.build.

@anjalirai-intel @jinengandhi-intel @jkr0103 Is there anything more you could add?

szymek156 · 2024-08-28T12:51:20Z

The error I get:

error while loading shared libraries: libsgx_dcap_quoteverify.so.1: cannot open shared object file: No such file or directory

The binary I try to launch in gramine links to these (ignore not found, the LD_LIBRARY_PATH was not set on this shell):

root@d3498d9e573e:/# ldd my-binary
/bin/bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)
	linux-vdso.so.1 (0x00007ffd9b7f3000)
	libra_tls_verify_dcap.so => not found
	libra_tls_attest.so => not found
	libssl.so.3 => /lib/x86_64-linux-gnu/libssl.so.3 (0x00007fb46773c000)
	libcrypto.so.3 => /lib/x86_64-linux-gnu/libcrypto.so.3 (0x00007fb4672b9000)
	libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007fb467299000)
	libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fb4671b8000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fb466fd7000)
	/lib64/ld-linux-x86-64.so.2 (0x00007fb468a76000)

~~Wonder why libsgx_dcap_quoteverify.so is not present here, probably it's loaded dynamically during runtime?~~
The libsgx_dcap_quoteverify.so is used by libra_tls_verify_dcap.so:

root@be547385a161:/gramine/meson_build_output/lib/x86_64-linux-gnu# ldd libra_tls_verify_dcap.so
/bin/bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)
	linux-vdso.so.1 (0x00007ffd981f5000)
	libsgx_dcap_quoteverify.so.1 => not found
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fd5626d9000)
	/lib64/ld-linux-x86-64.so.2 (0x00007fd56293d000)

On graminized container there is dcap lib (when I remove the if from template):

root@243b219091fd:/# find . -iname *dcap*
./gramine/meson_build_output/lib/x86_64-linux-gnu/gramine/runtime/glibc/libra_tls_verify_dcap_gramine.so
./gramine/meson_build_output/lib/x86_64-linux-gnu/gramine/runtime/glibc/libsecret_prov_verify_dcap.so
./gramine/meson_build_output/lib/x86_64-linux-gnu/gramine/runtime/glibc/libra_tls_verify_dcap.so
./gramine/meson_build_output/lib/x86_64-linux-gnu/libra_tls_verify_dcap_gramine.so
./gramine/meson_build_output/lib/x86_64-linux-gnu/libsecret_prov_verify_dcap.so
./gramine/meson_build_output/lib/x86_64-linux-gnu/libra_tls_verify_dcap.so

But I don't see any of Intel's sgx library:

root@243b219091fd:/# find . -iname *sgx*
./dev/sgx_enclave
./sys/class/misc/sgx_vepc
./sys/class/misc/sgx_provision
./sys/class/misc/sgx_enclave
./sys/devices/virtual/misc/sgx_vepc
./sys/devices/virtual/misc/sgx_provision
./sys/devices/virtual/misc/sgx_enclave
./sys/module/kvm_intel/parameters/sgx
./gramine/app_files/entrypoint.manifest.sgx
./gramine/meson_build_output/lib/x86_64-linux-gnu/gramine/sgx
./gramine/meson_build_output/lib/x86_64-linux-gnu/libsgx_util.a
./gramine/meson_build_output/lib/python3.11/site-packages/graminelibos/sgx_get_token.py
./gramine/meson_build_output/lib/python3.11/site-packages/graminelibos/sgx_sign.py
./gramine/meson_build_output/include/gramine/sgx_attest.h
./gramine/meson_build_output/include/gramine/sgx_arch.h
./gramine/meson_build_output/bin/gramine-sgx-ias-verify-report
./gramine/meson_build_output/bin/gramine-sgx-pf-tamper
./gramine/meson_build_output/bin/gramine-sgx-pf-crypt
./gramine/meson_build_output/bin/gramine-sgx-sign
./gramine/meson_build_output/bin/gramine-sgx-gen-private-key
./gramine/meson_build_output/bin/gramine-sgx-quote-view
./gramine/meson_build_output/bin/gramine-sgx-sigstruct-view
./gramine/meson_build_output/bin/gramine-sgx-get-token
./gramine/meson_build_output/bin/gramine-sgx-ias-request
./gramine/meson_build_output/bin/is-sgx-available
./gramine/meson_build_output/bin/gramine-sgx

dimakuv · 2024-08-29T06:44:32Z

@szymek156 Thanks for this exploration!

Can you try adding these lines:

gsc/templates/debian/Dockerfile.compile.template

Lines 44 to 45 in fcf9654

    
           RUN env DEBIAN_FRONTEND=noninteractive apt-get update \ 
        
               && env DEBIAN_FRONTEND=noninteractive apt-get install -y libsgx-dcap-quote-verify-dev

To the Dockerfile.build file, somewhere after here:

gsc/templates/debian/Dockerfile.build.template

Line 47 in fcf9654

I think your original suspicion is correct, and we miss the installation of the SGX DCAP library in Dockerfile.build. But then I don't understand how our tests work for Ubuntu ones, if they fail for Debian as you found out...

szymek156 · 2024-09-19T22:41:15Z

Hey @dimakuv I was busy with other stuff, but this one strike me again.

Here you can see the changes I made in order to finally build and run verification:
master...szymek156:gsc:master

I had to add libsgx-dcap-default-qpl-dev package because of the following error:

libdcap_quoteprov.so: cannot open shared object file: No such file or directory[sgx_dcap_load_qpl config.cpp:161] Couldn't find the Quote Provider library libdcap_quoteprov.so.1 or libdcap_quoteprov.so

This package provides libdcap_quoteprov library.

Calling ra_tls_verify_callback_extended_der works, however verification result is following:

RA-TLS certificate verification results: VerifyError {
    return_code: -9984,
    callback_results: ra_tls_verify_callback_results {
        attestation_scheme: RA_TLS_ATTESTATION_SCHEME_DCAP,
        err_loc: AT_VERIFY_EXTERNAL,
        dcap: dcap {
            func_verify_quote_result: 57369,
            quote_verification_result: "UNSPECIFIED",
        },
    },
}

That turns out is because libsgx-dcap-default-qpl-dev introduces /etc/sgx_default_qcnl.conf with default values. In particular this entry in config file:

  //PCCS server address
  "pccs_url": "https://localhost:8081/sgx/certification/v4/"

Is problematic, it needs to point to valid PCCS server address.

What I found out is to mount the file while running container from the host machine: -v /etc/sgx_default_qcnl.conf:/etc/sgx_default_qcnl.conf that contains correct configuration.

And in manifest file add these:

sgx.allowed_files = [
    "file:/etc/sgx_default_qcnl.conf"
]

This works, verification goes as expected. However now gramine yells at me that allowed_files are insecure...

I wonder if there is a better way of doing this? Another approach I tried:
In app Dockerfile (the one that I want to graminize) I added:

COPY --from=builder /src/deploy/sgx/sgx_default_qcnl.conf /etc/sgx_default_qcnl.conf

So my image contains valid sgx_default_qcnl.conf file from the beginning.
Updated my manifest:

sgx.trusted_files = [
    "file:/etc/sgx_default_qcnl.conf"
]

In order to include that file into final graminized image. However this not works because, during building unsigned graminized image I get following error:

Setting up libsgx-dcap-default-qpl (1.18.100.1-bionic1) ...

Configuration file '/etc/sgx_default_qcnl.conf'
 ==> File on system created by you or by a script.
 ==> File also in package provided by package maintainer.
   What would you like to do about it ?  Your options are:
    Y or I  : install the package maintainer's version
    N or O  : keep your currently-installed version
      D     : show the differences between the versions
      Z     : start a shell to examine the situation
 The default action is to keep your current version.
*** sgx_default_qcnl.conf (Y/I/N/O/D/Z) [default=N] ? 
dpkg: error processing package libsgx-dcap-default-qpl (--configure):
 end of file on stdin at conffile prompt
dpkg: dependency problems prevent configuration of libsgx-dcap-default-qpl-dev:
 libsgx-dcap-default-qpl-dev depends on libsgx-dcap-default-qpl (= 1.18.100.1-bionic1); however:
  Package libsgx-dcap-default-qpl is not configured yet.

dpkg: error processing package libsgx-dcap-default-qpl-dev (--configure):
 dependency problems - leaving unconfigured
Processing triggers for libc-bin (2.36-9+deb12u8) ...
Errors were encountered while processing:
 libsgx-dcap-default-qpl
 libsgx-dcap-default-qpl-dev
ESC[91mE: Sub-process /usr/bin/dpkg returned an error code (1)
ESC[0m
Successfully built an unsigned graminized Docker image `gsc-operator-sgx:latest-unsigned` from original application image `operator-sgx:latest`.

The file already exists (as expected) and dpkg expects user interaction, but will not get it, since it docker image build.

What do you think, is it possible to have valid qcnl.conf, and not rely on host mount to that file? Or I did something completely wrong? Maybe installing libsgx-dcap-default-qpl-dev was a mistake on a first place?

I would like to avoid string substitution in that file during startup of my enclave. I prefer clean solution where file is somehow provided.

dimakuv · 2024-09-20T09:27:10Z

Maybe installing libsgx-dcap-default-qpl-dev was a mistake on a first place?

No, this sounds correct.

The file already exists (as expected) and dpkg expects user interaction, but will not get it, since it docker image build.

I think the default choice (N) is a correct one for your case, but the subsequent error is weird. You may ask in DCAP developers at https://github.com/intel/SGXDataCenterAttestationPrimitives/issues. The error seems to me an incorrect implementation of DCAP packaging.

I would like to avoid string substitution in that file during startup of my enclave. I prefer clean solution where file is somehow provided.

The simplest thing you can do is to learn the hash of your config file ($ sha256sum /src/deploy/sgx/sgx_default_qcnl.conf) and add this hash to the Gramine manifest file:

sgx.trusted_files = [
    { uri = "file:/etc/sgx_default_qcnl.conf", sha256 = "<sha256sum output>" },
]

Now you can run Docker container with -v /etc/sgx_default_qcnl.conf:/etc/sgx_default_qcnl.conf.

szymek156 · 2024-09-20T16:37:06Z

@dimakuv trusted_files approach works like a charm! Thank you!
What about this: master...szymek156:gsc:master

Should I prepare pull request?

dimakuv · 2024-09-23T07:07:14Z

Hm, isn't it enough to add libsgx-dcap-default-qpl-dev to this line:

gsc/templates/debian/Dockerfile.compile.template

Line 45 in 3168b08

    
           && env DEBIAN_FRONTEND=noninteractive apt-get install -y libsgx-dcap-quote-verify-dev

dimakuv · 2024-09-23T07:12:42Z

Btw, officially Intel SGX SDK/PSW/DCAP is not supported on Debian, at least I don't see the corresponding packages here: https://download.01.org/intel-sgx/sgx_repo

So I'm not sure it would be correct for GSC to "support" DCAP libraries from Ubuntu that run in a Debian Docker container. I think that's the main reason why we have {% if template_path(Distro) == 'ubuntu' %}-Ddcap=enabled{% endif %} condition.

szymek156 · 2024-09-27T08:39:07Z

True, debian distro should not include ubuntu packages.

I changed my base Docker image to use ubuntu:22.04, graminized one still returns error
error while loading shared libraries: libra_tls_verify_dcap.so: cannot open shared object file: No such file or directory

I am using gsc on commit 3168b08 (latest master)

Seems like

       {% if template_path(Distro) == 'ubuntu' %}-Ddcap=enabled{% endif %} \

Is not working

I am using the config.yaml that looks like so

# Specify the OS distro that is used to build Gramine, i.e., the distro from where the Gramine build
# gets all tools and dependencies from. This distro should match the distro underlying the
# application's Docker image; otherwise the results may be unpredictable (if you specify `"auto"`,
# which is recommended, you don't need to worry about the mismatch).
#
# Currently supported distros are:
# - ubuntu:20.04, ubuntu:21.04, ubuntu:22.04, ubuntu:23.04
# - debian:10, debian:11, debian:12
# - centos:8
# - redhat/ubi8:8.8
# - redhat/ubi8-minimal:8.8

# If Distro is set to "auto", GSC detects the distro automatically by examining the supplied
# Docker image. Alternatively, Distro can be set to one of the supported distros mentioned above.
Distro: "ubuntu:22.04"

# If the image has a specific registry, define it here.
# Empty by default; example value: "registry.access.redhat.com/ubi8".
Registry: ""

# If you're using your own fork and branch of Gramine, specify the GitHub link and the branch name
# below; typically, you want to keep the default values though.
#
# It is also possible to specify the prebuilt Gramine Docker image (that was built previously via
# the `gsc build-gramine` command). For this, remove Repository and Branch and instead write:
#   Image:      "<prebuilt Gramine Docker image>"
#
# GSC releases are guaranteed to work with corresponding Gramine releases (and GSC `master`
# branch is guaranteed to work with current Gramine `master` branch).
Gramine:
    Repository: "https://github.com/gramineproject/gramine.git"
    Branch:     "v1.7"

# Specify the Intel SGX driver installed on your machine (more specifically, on the machine where
# the graminized Docker container will run); there are several variants of the SGX driver:
#
#   - upstream (in-kernel) driver: use empty values like below
#         Repository: ""
#         Branch:     ""
#
#   - DCAP out-of-tree driver: same as above, use empty values
#         Repository: ""
#         Branch:     ""
#
#   - legacy out-of-tree driver: use something like the below values, but adjust the branch name
#         Repository: "https://github.com/01org/linux-sgx-driver.git"
#         Branch:     "sgx_driver_1.9"
#
SGXDriver:
    Repository: ""
    Branch:     ""

call the command:

./gsc build-gramine --no-cache  --config_file config.yaml gramine-self-build-image

the Dockerfile.compile looks like so


FROM ubuntu:22.04 AS gramine


# Install distro-specific packages to build Gramine (e.g., python3, protobuf, tomli, etc.)


RUN env DEBIAN_FRONTEND=noninteractive apt-get update \
    && env DEBIAN_FRONTEND=noninteractive apt-get install -y \
        autoconf \
        bison \
        build-essential \
        cmake \
        coreutils \
        curl \
        gawk \
        git \
        libprotobuf-c-dev \
        linux-headers-generic \
        nasm \
        ninja-build \
        pkg-config \
        protobuf-c-compiler \
        protobuf-compiler \
        python3 \
        python3-cryptography \
        python3-protobuf \
        wget \
# please see the comment in Dockerfile.build.template for explanation why this condition is needed
           python3-pip \
           && /usr/bin/python3 -B -m pip install 'tomli>=1.1.0' 'tomli-w>=0.4.0' 'meson>=0.56,!=1.2.*'

COPY intel-sgx-deb.key /

RUN echo 'deb [arch=amd64] https://download.01.org/intel-sgx/sgx_repo/ubuntu bionic main' \
    > /etc/apt/sources.list.d/intel-sgx.list \
    && apt-key add /intel-sgx-deb.key

RUN env DEBIAN_FRONTEND=noninteractive apt-get update \
    && env DEBIAN_FRONTEND=noninteractive apt-get install -y libsgx-dcap-quote-verify-dev


RUN git clone https://github.com/gramineproject/gramine.git /gramine

RUN cd /gramine \
    && git fetch origin v1.7 \
    && git checkout v1.7


RUN mkdir -p /gramine/driver/asm \
    && cd /gramine/driver/asm \
    && wget --timeout=10 -O sgx.h \
        https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/plain/arch/x86/include/uapi/asm/sgx.h?h=v5.11 \
    && sha256sum sgx.h | grep -q a34a997ade42b61376b1c5d3d50f839fd28f2253fa047cb9c0e68a1b00477956


RUN cd /gramine \
    && meson setup build/ --prefix="/gramine/meson_build_output" \
       --buildtype=release \
       -Ddirect=enabled -Dsgx=enabled \
        \
        \
       -Dsgx_driver=upstream -Dsgx_driver_include_path=/gramine/driver \
        \
    && ninja -C build \
    && ninja -C build install

Note lack of -Ddcap=enabled in rendered Dockerfile.

dimakuv · 2024-09-27T09:49:20Z

@szymek156 Looks like you found a recently introduced regression bug in GSC.

I created PR #216 to fix it. Can you try it?

szymek156 · 2024-09-27T10:27:18Z

I tried, and it works, however I do not understand at which stage the libsgx libs (libsgx-dcap-default-qpl-dev) goes to the final image?

dimakuv · 2024-09-27T10:30:44Z

@szymek156 I'm also not sure :)

Could it be that this line that installs the DCAP QuoteVerification package also installs the QPL-dev package?

gsc/templates/debian/Dockerfile.compile.template

Line 45 in 3168b08

    
           && env DEBIAN_FRONTEND=noninteractive apt-get install -y libsgx-dcap-quote-verify-dev

szymek156 · 2024-09-27T11:08:41Z

Nah mistake in my environment, build scripts were still using my forked repo, sorry for confusion, now building using yours #216

it breaks as expected:

error while loading shared libraries: libsgx_dcap_quoteverify.so.1

adding
apt install libsgx-dcap-default-qpl-dev fixes it but then:

Couldn't find the Quote Provider library libdcap_quoteprov.so.1 or libdcap_quoteprov.so

so adding to that: apt install libsgx-dcap-default-qpl-dev
finally resolves the problem

szymek156 · 2024-10-02T06:50:45Z

@dimakuv I think adding libsgx-dcap-default-qpl-dev, libsgx-dcap-default-qpl-dev packages should be part of https://github.com/gramineproject/gsc/blob/master/templates/ubuntu/Dockerfile.build.template

dimakuv linked a pull request Sep 27, 2024 that will close this issue

Fix installation of Gramine DCAP support on Ubuntu #216

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Gramine runtime does not build dcap on debian #213

Gramine runtime does not build dcap on debian #213

szymek156 commented Aug 27, 2024

szymek156 commented Aug 27, 2024

dimakuv commented Aug 28, 2024

szymek156 commented Aug 28, 2024 •

edited

Loading

dimakuv commented Aug 29, 2024

szymek156 commented Sep 19, 2024

dimakuv commented Sep 20, 2024

szymek156 commented Sep 20, 2024 •

edited

Loading

dimakuv commented Sep 23, 2024

dimakuv commented Sep 23, 2024

szymek156 commented Sep 27, 2024

dimakuv commented Sep 27, 2024

szymek156 commented Sep 27, 2024

dimakuv commented Sep 27, 2024

szymek156 commented Sep 27, 2024

szymek156 commented Oct 2, 2024

Gramine runtime does not build dcap on debian #213

Gramine runtime does not build dcap on debian #213

Comments

szymek156 commented Aug 27, 2024

szymek156 commented Aug 27, 2024

dimakuv commented Aug 28, 2024

szymek156 commented Aug 28, 2024 • edited Loading

dimakuv commented Aug 29, 2024

szymek156 commented Sep 19, 2024

dimakuv commented Sep 20, 2024

szymek156 commented Sep 20, 2024 • edited Loading

dimakuv commented Sep 23, 2024

dimakuv commented Sep 23, 2024

szymek156 commented Sep 27, 2024

dimakuv commented Sep 27, 2024

szymek156 commented Sep 27, 2024

dimakuv commented Sep 27, 2024

szymek156 commented Sep 27, 2024

szymek156 commented Oct 2, 2024

szymek156 commented Aug 28, 2024 •

edited

Loading

szymek156 commented Sep 20, 2024 •

edited

Loading