Multi-Threading Performance Issue with TBB in Gramine-SGX

[BaoSKY]

I am attempting to use multi-threading with the TBB library to optimize the performance of my algorithm. The implementation works well in both REE and Gramine-direct modes, but I am encountering a significant slowdown in Gramine-SGX.

Here are the results of my experiments:

• Max threads set to 1:
  The performance is comparable across REE, Gramine-direct, and Gramine-SGX.

• Max threads set to 16:
  The performance remains comparable between REE and Gramine-direct. However, in Gramine-SGX, the performance slows down by approximately 5x.

Notably, the algorithm does not involve any I/O operations, so I suspect this is not an issue related to Ocall/Ecall overhead.

Questions:

• Does Gramine-SGX have limited support for TBB or multi-threading in general?

• Could this slowdown be inherent to the SGX architecture's multi-threading support?

I would greatly appreciate any insights or guidance on this issue. Thank you!

[kailun-qin]

Max threads set to 1:
Max threads set to 16:

Do you mean the setting in the TBB library? Have you adjusted the sgx.max_threads Gramine manifest option accordingly? Pls see https://gramine.readthedocs.io/en/latest/manifest-syntax.html#number-of-threads for details.

[BaoSKY]

Max threads set to 1:
Max threads set to 16:

Yes, this refers to the configuration in the TBB library, set using:
tbb::global_control c(tbb::global_control::max_allowed_parallelism, MAX_THREAD_NUM);
Here, MAX_THREAD_NUM is either 1 or 16.

Additionally, the sgx.max_threads option in the Gramine manifest is set to 20.

If I set sgx.max_threads to a value smaller than MAX_THREAD_NUM in the TBB configuration (e.g., sgx.max_threads is 8 while MAX_THREAD_NUM is 16), I encounter the following error:

error: There are no available TCS pages left for a new thread. Please try to increase sgx.max_threads in the manifest. The current value is 8

On the other hand, increasing sgx.max_threads beyond 20 (e.g., setting it to 32) results in only slight changes in performance.

[kailun-qin]

Thanks for the clarification.

May I know the EPC size of your platform (can be obtained via is-sgx-available) and your sgx.enlave_size and sgx.edmm_enable settings for TBB MAX_THREAD_NUM=16 in Gramine manifest?

[BaoSKY]

Sure!

Here is is-sgx-available:

SGX supported by CPU: true
SGX1 (ECREATE, EENTER, ...): true
SGX2 (EAUG, EACCEPT, EMODPR, ...): true
Flexible Launch Control (IA32_SGXPUBKEYHASH{0..3} MSRs): true
SGX extensions for virtualizers (EINCVIRTCHILD, EDECVIRTCHILD, ESETCONTEXT): false
Extensions for concurrent memory management (ETRACKC, ELDBC, ELDUC, ERDINFO): false
CET enclave attributes support (See Table 37-5 in the SDM): false
Key separation and sharing (KSS) support (CONFIGID, CONFIGSVN, ISVEXTPRODID, ISVFAMILYID report fields): true
Max enclave size (32-bit): 0x80000000
Max enclave size (64-bit): 0x100000000000000
EPC size: 0x1fcf800000
SGX driver loaded: true
AESMD installed: true
SGX PSW/libsgx installed: true
#PF/#GP information in EXINFO in MISC region of SSA supported: true
#CP information in EXINFO in MISC region of SSA supported: false

And here is Gramine Manifest:

sgx.debug = false
sgx.edmm_enable = {{ 'true' if env.get('EDMM', '0') == '1' else 'false' }}
sgx.enclave_size = "2G"
sgx.max_threads = 20

And here is build information:

Attributes (required for enclave measurement):
size: 0x80000000
edmm: False
max_threads: 20