Performance degradations observed in few workloads when EDMM is enabled

[vasanth-intel]

Description of the problem

When EDMM is enabled, few tests in the following workloads show more degradation with Gramine-SGX execution when compared with the actual baseline performance numbers. The baseline performance numbers are with Gramine v1.5 release. The details of performance degradation data are as attached in Gramine_SGX_EDMM_Degradation.xlsx.

Following are the workloads.
Nginx
OVMS (OpenVINO Model Server)
MySQL
MariaDB

Note:
The above observation was seen with commit fa437cb, but the issue is also seen with Gramine v1.5; i.e. when EDMM is enabled the above perf degradations are visible even with Gramine v1.5.

Gramine_SGX_EDMM_Degradation.xlsx

Steps to reproduce

Nginx Workload:

1. Install Nginx dependencies with below command:
   sudo apt-get install -y build-essential libssl-dev zlib1g-dev
2. Clone gramine repo.
3. cd gramine/CI-Examples/nginx/
4. make SGX=1

Execute Nginx workload in Native mode against wrk https benchmark.

1. Launch Nginx server in Native mode.
   ./install/sbin/nginx -c conf/nginx-gramine.conf &
2. Invoke wrk https benchmarking command.
   ./wrk/wrk -t1 -c300 -d180s https://127.0.0.1:8444/random/10K.1.html

Execute Nginx workload in Gramine SGX mode against wrk https benchmark.

1. Launch Nginx server in Gramine SGX mode.
   gramine-sgx ./nginx &
2. Invoke wrk https benchmarking command.
   ./wrk/wrk -t1 -c300 -d180s https://127.0.0.1:8444/random/10K.1.html

Note:
The wrk benchmarking tool is invoked for 1 thread and 10k file size. These parameters need to be changed for higher thread count and file sizes.

MySQL Workload:

1. Install sysbench benchmarking tool using below command:
   sudo apt install -y sysbench
2. Execute the 5 steps mentioned in section Build a Gramine-protected MySQL image within https://github.com/gramineproject/contrib/tree/master/Intel-Confidential-Compute-for-X/workloads/mysql.
3. To initialize and run MySQL server container on Linux-Native, run the below command:
   mkdir workloads/mysql/test_db
docker run --rm --net=host --name init_test_db --user $(id -u):$(id -g) \ -v $PWD/workloads/mysql/test_db:/test_db \ -e MYSQL_ALLOW_EMPTY_PASSWORD=true -e MYSQL_DATABASE=test_db mysql:8.0.32-debian \ --datadir /test_db &
4. To initialize and run MySQL server container with GSC (With Encryption), run the below command:
   mkdir workloads/mysql/test_db
docker run --net=host --name init_test_db --device=/dev/sgx/enclave -v /var/run/test_db_encrypted:/var/run/test_db_encrypted -it gsc-mysql:8.0.32-debian --datadir /var/run/test_db_encrypted

Note:
The below Prepare and Cleanup commands need to be run only once per test. Run command can be run for 10 iterations.

Read only operation:
sysbench --db-driver=mysql --mysql-host=127.0.0.1 --mysql-port=3306 --mysql-user=root --mysql-db=test_db --time=40 --report-interval=5 oltp_read_only --tables=16 --table_size=100000 --threads=8 prepare
sysbench --db-driver=mysql --mysql-host=127.0.0.1 --mysql-port=3306 --mysql-user=root --mysql-db=test_db --time=40 --report-interval=5 oltp_read_only --tables=16 --table_size=100000 --threads=8 run
sysbench --db-driver=mysql --mysql-host=127.0.0.1 --mysql-port=3306 --mysql-user=root --mysql-db=test_db --time=40 --report-interval=5 oltp_read_only --tables=16 --table_size=100000 --threads=8 cleanup

Write only operation:
sysbench --db-driver=mysql --mysql-host=127.0.0.1 --mysql-port=3306 --mysql-user=root --mysql-db=test_db --time=20 --report-interval=5 oltp_write_only --tables=16 --table_size=100000 --threads=8 prepare
sysbench --db-driver=mysql --mysql-host=127.0.0.1 --mysql-port=3306 --mysql-user=root --mysql-db=test_db --time=20 --report-interval=5 oltp_write_only --tables=16 --table_size=100000 --threads=8 run
sysbench --db-driver=mysql --mysql-host=127.0.0.1 --mysql-port=3306 --mysql-user=root --mysql-db=test_db --time=20 --report-interval=5 oltp_write_only --tables=16 --table_size=100000 --threads=8 cleanup

Read-write operation:
sysbench --db-driver=mysql --mysql-host=127.0.0.1 --mysql-port=3306 --mysql-user=root --mysql-db=test_db --time=40 --report-interval=5 oltp_read_write --tables=16 --table_size=100000 --threads=8 prepare
sysbench --db-driver=mysql --mysql-host=127.0.0.1 --mysql-port=3306 --mysql-user=root --mysql-db=test_db --time=40 --report-interval=5 oltp_read_write --tables=16 --table_size=100000 --threads=8 run
sysbench --db-driver=mysql --mysql-host=127.0.0.1 --mysql-port=3306 --mysql-user=root --mysql-db=test_db --time=40 --report-interval=5 oltp_read_write --tables=16 --table_size=100000 --threads=8 cleanup

MariaDB Workload:
For this workload, the execution operations are same as MySQL workload, but for building the workload refer section Build a Gramine-protected MariaDB image within https://github.com/gramineproject/contrib/tree/master/Intel-Confidential-Compute-for-X/workloads/mariadb.

OpenVINO Model Server (OVMS) Workload:

1. Download and build the OVMS CPP benchmark client as mentioned in https://docs.openvino.ai/2023.1/ovms_demo_benchmark_app_cpp.html.
2. Execute the 5 steps mentioned in section Build a Gramine-protected OpenVINO Model Server image within https://github.com/gramineproject/contrib/tree/master/Intel-Confidential-Compute-for-X/workloads/openvino-model-server.

Execute OVMS workload in Native mode against OVMS CPP benchmark client.

1. Launch OVMS in Native mode.
   docker run --net=host --name init_native_test_db -u $(id -u):$(id -g) -v $(pwd)/workloads/openvino-model-server/test_model:/model -p 9001:9001 openvino/model_server:latest --model_path /model --model_name model-name --port 9001 --layout NCHW

2. Invoke OVMS CPP benchmarking client command.
   docker run --rm --network host -e "no_proxy=localhost" ovms_cpp_benchmark \ ./synthetic_client_async_benchmark --model_name=model-name --grpc_port=9001 --iterations=2000 \ --max_parallel_requests=100 --consumers=8 --producers=1

3. Launch OVMS in Gramine-SGX mode:
   docker run --rm --net=host --name init_sgx_enc_test_db -u 0:0 -p 9001:9001 --device=/dev/sgx/enclave -v /mnt/tmpfs/model_encrypted:/mnt/tmpfs/model_encrypted -t gsc-openvino/model_server --model_path /mnt/tmpfs/model_encrypted --model_name model-name --port 9001 --layout NCHW

4. Invoke OVMS CPP benchmarking client command.
   docker run --rm --network host -e "no_proxy=localhost" ovms_cpp_benchmark \ ./synthetic_client_async_benchmark --model_name=model-name --grpc_port=9001 --iterations=2000 \ --max_parallel_requests=100 --consumers=8 --producers=1

Note:
Replace "model-name" strings in above commands with the corresponding model names to be tested for. i.e. 'resnet50-binary-0001' for Resnet model and 'face-detection-retail-0005' for Face detection model.

Expected results

When EDMM is enabled, the performance degradation of the workloads should not exceed the baseline performance numbers.

Actual results

When EDMM is enabled, the performance degradation for few tests of the above mentioned workloads exceed the baseline performance numbers.

Gramine commit hash

fa437cb

[Villain88]

I have a similar problem.
I'm testing the performance of edmm on an empty python application like:

def read_and_print_file(file_path):
    try:
        with open(file_path, 'r') as file:
            file_contents = file.read()
            print(file_contents)
    except FileNotFoundError:
        print(f"File '{file_path}' not found.")
    except Exception as e:
        print(f"Error reading file: {e}")

if __name__ == "__main__":
    print("Hello, EDMM")
    read_and_print_file('/proc/1/status')

Enclave with edmm slower (40s vs 26s for non-edmm enclave)

Application output:

Hello, EDMM
VmPeak:    43084 kB

I thought allocating 43MB of memory shouldn't take that long

Enclave manifest for EDMM:

...
sgx.max_threads = 4
sgx.edmm_enable = true
...

Enclave manifest without EDMM:

...
sgx.enclave_size = "16G"
sgx.max_threads = 128
sgx.edmm_enable = false
...

Dockerfile:

FROM ubuntu:22.04

RUN apt update \
    && apt install -y libnss-mdns libnss-myhostname lsb-release \
    && apt install -y python3.10 python3-pip libcurl4

ENTRYPOINT ["/usr/bin/python3.10", "/entrypoint.py"]

Additional information:
Intel(R) Xeon(R) Gold 6312U CPU @ 2.40GHz (HT off)
Ubuntu 22.04, kernel 6.5.11

[Villain88]

I figured out the problem a little
My enclave contains the python3-pip package (related issue)
If I remove it, the speed increases almost 4 times.

edmm-on, without pip - 10.8s, VmPeak:    30672 kB
edmm-on, with pip - 38.71s, VmPeak:    30672 kB
edmm-off, without pip - 25s, VmPeak:    43084 kB
edmm-off, with pip - 26s, VmPeak:    43084 kB

I don't understand why this package increases memory consumption and slows down enclave startup by 4 times when using edmm

[dimakuv]

@Villain88 You're using GSC, right?

One thing I can imagine is that the Python interpreter (your ENTRYPOINT) looks into some Python3-pip files if it detects its installation on the system. I have no idea why Python would do this. I have no other ideas.

You could enable the Gramine logs and see what it prints. Search for pip in this log, maybe this will give some hints.

Generally, EDMM moves some burden of enclave page first-time accesses from startup to runtime. So if the application (or more specifically, Python interpreter) opens and loads some pip-related files, these will be first-time accesses, and EDMM execution will be slower.

[Villain88]

@Villain88 You're using GSC, right?

One thing I can imagine is that the Python interpreter (your ENTRYPOINT) looks into some Python3-pip files if it detects its installation on the system. I have no idea why Python would do this. I have no other ideas.

I assumed the same thing, I'll try to figure it out and post the result here. In general, even using an enclave without pip, our machine learning solutions lose in speed when EDMM is enabled, despite the fact that they get a head start of ~20 seconds to launch the enclave (16Gb)

[dimakuv]

In general, even using an enclave without pip, our machine learning solutions lose in speed when EDMM is enabled, despite the fact that they get a head start of ~20 seconds to launch the enclave (16Gb)

Yes, this is expected. We have a quick explanation here: https://gramine.readthedocs.io/en/stable/manifest-syntax.html#edmm

We also have many discussions on this EDMM perf issue in our GitHub repo. Please just search for "EDMM + performance", and you can learn more from our Issues and Discussions.

TLDR: The Linux SGX driver is not optimized for EDMM at all. Until the SGX driver has EDMM optimizations, Gramine can do nothing about the EDMM performance.

[Villain88]

You could enable the Gramine logs and see what it prints. Search for pip in this log, maybe this will give some hints.

The python3-pip package does not make any changes to the python application.
Most likely the problem is the size of the manifest:
3,474,035 for signed manifest with pip
1,595,860 for signed manifest without pip

python3-pip package installs a bunch of dependencies:

The following NEW packages will be installed:
  build-essential bzip2 cpp cpp-11 dirmngr dpkg-dev fakeroot fontconfig-config fonts-dejavu-core g++ g++-11 gcc gcc-11 gcc-11-base gnupg gnupg-l10n gnupg-utils gpg gpg-agent gpg-wks-client gpg-wks-server gpgconf gpgsm javascript-common libalgorithm-diff-perl libalgorithm-diff-xs-perl libalgorithm-merge-perl
  libasan6 libassuan0 libatomic1 libbsd0 libc-dev-bin libc-devtools libc6-dev libcc1-0 libcrypt-dev libdeflate0 libdpkg-perl libexpat1-dev libfakeroot libfile-fcntllock-perl libfontconfig1 libfreetype6 libgcc-11-dev libgd3 libgdbm-compat4 libgdbm6 libgomp1 libisl23 libitm1 libjbig0 libjpeg-turbo8 libjpeg8
  libjs-jquery libjs-sphinxdoc libjs-underscore libksba8 liblocale-gettext-perl liblsan0 libmd0 libmpc3 libmpfr6 libnpth0 libnsl-dev libperl5.34 libpng16-16 libpython3-dev libpython3.10 libpython3.10-dev libquadmath0 libstdc++-11-dev libtiff5 libtirpc-dev libtsan0 libubsan1 libwebp7 libx11-6 libx11-data libxau6
  libxcb1 libxdmcp6 libxpm4 linux-libc-dev lto-disabled-list make manpages manpages-dev netbase patch perl perl-modules-5.34 pinentry-curses python3-dev python3-distutils python3-lib2to3 python3-pip python3-setuptools python3-wheel python3.10-dev rpcsvc-proto ucf xz-utils zlib1g-dev

that's why the size of the manifest is so large

By my logic it shouldn't affect the loading speed of the enclave that much, surely it's not a gramine issue?

[dimakuv]

I would be very surprised if loading the manifest of 3.5MB (rather than 1.5MB) would significantly affect the speed of the enclave.

But on the other side, maybe? We haven't done such experiments lately. Maybe we have a terrible bottleneck somewhere.

[vasanth-intel]

Adding Openvino to the above degradation list of workloads when we enable EDMM. Attaching the updated Gramine_SGX_EDMM_Degradation.xlsx which is same as earlier data, but appended with Openvino perf data. Degradation observed in Openvino workload is compared with respect to Gramine v1.7.

Gramine_SGX_EDMM_Degradation.xlsx