Pytorch cannot load libtorch_cpu.so when running in a Gramine Enclave #1293

hop-along-polly · 2023-04-18T16:15:36Z

hop-along-polly
Apr 18, 2023

I'm trying to get a FastAPI app that uses torch to run within Gramine. I've gotten to a point where the API will start and handle requests but the API endpoint that deals with pytorch models fails with this error ImportError: libtorch_cpu.so: failed to map segment from shared object. I'm fairly certain I'm doing something wrong with the fs.mounts or sgx.trusted_files but I can't see what I've done wrong.

I'm not sure what information is all needed so It's probably information overload.

is-sgx-available output

SGX supported by CPU: true
SGX1 (ECREATE, EENTER, ...): true
SGX2 (EAUG, EACCEPT, EMODPR, ...): false
Flexible Launch Control (IA32_SGXPUBKEYHASH{0..3} MSRs): true
SGX extensions for virtualizers (EINCVIRTCHILD, EDECVIRTCHILD, ESETCONTEXT): false
Extensions for concurrent memory management (ETRACKC, ELDBC, ELDUC, ERDINFO): false
CET enclave attributes support (See Table 37-5 in the SDM): false
Key separation and sharing (KSS) support (CONFIGID, CONFIGSVN, ISVEXTPRODID, ISVFAMILYID report fields): false
Max enclave size (32-bit): 0x80000000
Max enclave size (64-bit): 0x1000000000
EPC size: 0x7000000
SGX driver loaded: true
AESMD installed: true
SGX PSW/libsgx installed: true
#PF/#GP information in EXINFO in MISC region of SSA supported: false
#CP information in EXINFO in MISC region of SSA supported: false

Environment

Server: Intel SGX enabled VM running in Azure (I verified this with is-sgx-
OS: Ubuntu 20.04
Python:
- Ver: 3.11
- Location: /usr/bin/python -> /usr/bin/python3.11
Pip:
- Ver: 23.0.1
- Location: /home/modelscan/.local/bin/pip

Directory Structure

/home
   /modelscan
        /api
           app.manifest <- gramine manifest file
           requirements.txt
           modelscan/
                app.py <- this is FastAPI entrypoint
                ... the rest of the app code

Contents of App.py (Redacted for IP protection)

import os
import uvicorn

from fastapi import FastAPI, UploadFile

from modelscan.loader import load_model


app = FastAPI()
log = logging.getLogger(__name__)


@app.post("/api/upload")
async def create_upload_file(file: UploadFile):
        try:
            model = load_model(file) # This is the function that uses torch to load the model
            # propriety code redacted
        except UnknownModelFormatError:
            return {"error": "unable to parse", "filename": file.filename}


@app.get('/api/health')
async def health():
    return { 'status': 'healthy' }


if __name__ == "__main__":
    uvicorn.run("modelscan.app:app", host="0.0.0.0", port=8000, reload=True)

The Gramine Manifest File

[loader]
entrypoint = "file:/usr/lib/x86_64-linux-gnu/gramine/libsysdb.so"
log_level = "error"
insecure__use_cmdline_argv = true # temporarily allow while troubleshooting

[loader.env]
LD_LIBRARY_PATH = "/lib:/lib:/lib/x86_64-linux-gnu:/usr//lib/x86_64-linux-gnu"
OMP_NUM_THREADS = "4"
PYTHONPATH = "/usr/local/lib/python3.11/site-packages" # make sure Python in Gramine enclave can see the locally installed packages. (See mounts below)

[libos]
entrypoint = "/usr/bin/python3.11"

[sys]
enable_sigterm_injection = true
enable_extra_runtime_domain_names_conf = true

[sys.stack]
size = "2M"

[fs]
mounts = [
   { path = "/usr/bin/uvicorn", uri = "file:/home/modelscan/.local/bin/uvicorn" },
   { path = "/modelscan/", uri = "file:/home/modelscan/api/modelscan/" },
   { path = "/lib", uri = "file:/usr/lib/x86_64-linux-gnu/gramine/runtime/glibc" },
   { path = "/lib/x86_64-linux-gnu", uri = "file:/lib/x86_64-linux-gnu" },
   { path = "/usr//lib/x86_64-linux-gnu", uri = "file:/usr//lib/x86_64-linux-gnu" },
   { path = "/usr/lib/python3.11", uri = "file:/usr/lib/python3.11" },
   { path = "/usr/lib/python3.11/lib-dynload", uri = "file:/usr/lib/python3.11/lib-dynload" },
   { path = "/usr/local/lib/python3.11/dist-packages", uri = "file:/usr/local/lib/python3.11/dist-packages" },
   { path = "/usr/lib/python3/dist-packages", uri = "file:/usr/lib/python3/dist-packages" },
   { path = "/usr/bin/python3.11", uri = "file:/usr/bin/python3.11" },
   { path = "/usr/local/lib/python3.11/site-packages", uri = "file:/home/modelscan/.local/lib/python3.11/site-packages" }, # Mounts locally installed pypi packages to a location in Gramine that Python can find. See PYTHONPATH envvar.
   { type = "tmpfs", path = "/tmp" }
]

[sgx]
debug = true
edmm_enable = false
enclave_size = "1G"
max_threads = 32
remote_attestation = "none"
ra_client_spid = ""
ra_client_linkable = false
trusted_files = [
   { uri = "file:/usr/lib/x86_64-linux-gnu/gramine/libsysdb.so" },
   { uri = "file:/usr/bin/python3.11" },
   { uri = "file:/usr/lib/x86_64-linux-gnu/gramine/runtime/glibc/" },
   { uri = "file:/lib/x86_64-linux-gnu/" },
   { uri = "file:/usr//lib/x86_64-linux-gnu/" },
   { uri = "file:/usr/lib/python3.11/" },
   { uri = "file:/usr/lib/python3.11/lib-dynload/" },
   { uri = "file:/usr/local/lib/python3.11/dist-packages/" },
   { uri = "file:/usr/lib/python3/dist-packages/" },
   { uri = "file:/home/modelscan/api/modelscan/" },
   { uri = "file:/home/modelscan/.local/lib/python3.11/site-packages/" },
   { uri = "file:/home/modelscan/.local/bin/uvicorn" }
]
isvprodid = 0
isvsvn = 0
require_avx = false
require_avx512 = false
require_mpx = false
require_pkru = false
require_amx = false
require_exinfo = false
nonpie_binary = true
enable_stats = false

Commands Run

gramine-sgx-sign --manifest app.manifest --output app.manifest.sgx
gramine-sgx app /usr/bin/uvicorn modelscan.app:app --host 0.0.0.0 --port 8000

The Problem

When I use the api/upload endpoint I get the previously mentioned error. Here is the entire stacktrace

**Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/uvicorn/protocols/http/h11_impl.py", line 429, in run_asgi
    result = await app(  # type: ignore[func-returns-value]
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/uvicorn/middleware/proxy_headers.py", line 78, in __call__
    return await self.app(scope, receive, send)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/fastapi/applications.py", line 276, in __call__
    await super().__call__(scope, receive, send)
  File "/usr/local/lib/python3.11/site-packages/starlette/applications.py", line 122, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/usr/local/lib/python3.11/site-packages/starlette/middleware/errors.py", line 184, in __call__
    raise exc
  File "/usr/local/lib/python3.11/site-packages/starlette/middleware/errors.py", line 162, in __call__
    await self.app(scope, receive, _send)
  File "/usr/local/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 79, in __call__
    raise exc
  File "/usr/local/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 68, in __call__
    await self.app(scope, receive, sender)
  File "/usr/local/lib/python3.11/site-packages/fastapi/middleware/asyncexitstack.py", line 21, in __call__
    raise e
  File "/usr/local/lib/python3.11/site-packages/fastapi/middleware/asyncexitstack.py", line 18, in __call__
    await self.app(scope, receive, send)
  File "/usr/local/lib/python3.11/site-packages/starlette/routing.py", line 718, in __call__
    await route.handle(scope, receive, send)
  File "/usr/local/lib/python3.11/site-packages/starlette/routing.py", line 276, in handle
    await self.app(scope, receive, send)
  File "/usr/local/lib/python3.11/site-packages/starlette/routing.py", line 66, in app
    response = await func(request)
               ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/fastapi/routing.py", line 237, in app
    raw_response = await run_endpoint_function(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/fastapi/routing.py", line 163, in run_endpoint_function
    return await dependant.call(**values)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/modelscan/app.py", line 29, in create_upload_file
    scanner = get_scanner_for_model(pathlib.Path(temp.name), log)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/modelscan/torch.py", line 7, in <module>
    import torch
  File "/usr/local/lib/python3.11/site-packages/torch/__init__.py", line 229, in <module>
    from torch._C import *  # noqa: F403
    ^^^^^^^^^^^^^^^^^^^^^^
ImportError: libtorch_cpu.so: failed to map segment from shared object
**

The Question

With all that being said I know that libtorch_cpu.so is on the base Ubuntu system.
find . -name libtorch_cpu.so finds it in 2 locations

./api/venv/lib/python3.11/site-packages/torch/lib/libtorch_cpu.so <- this can be ignored since I'm not using the virtual environment
./.local/lib/python3.11/site-packages/torch/lib/libtorch_cpu.so

The second location is also mounted and trusted in the gramine manifest file so there should be no issues with reading it.

#app.manifest excerpt
[fs]
mounts = [
    ...,
    { path = "/usr/local/lib/python3.11/site-packages", uri = "file:/home/modelscan/.local/lib/python3.11/site-packages" }, # Mounts locally installed pypi packages to a location in Gramine that Python can find. See PYTHONPATH envvar.
]

[sgx]
trusted_files = [
    ....
    { uri = "file:/home/modelscan/.local/lib/python3.11/site-packages/" },
]

Any help is greatly appreciated.

hop-along-polly · 2023-04-18T18:19:52Z

hop-along-polly
Apr 18, 2023
Author

Interestingly enough running the application with gramine-direct instead of gramine-sgx does not encounter that error. I'm not sure what the difference is between gramine-direct and gramine-sgx but I'm guessing gramine-direct is not running in a secure enclave.

2 replies

monavij Apr 19, 2023
Collaborator

Interestingly enough running the application with gramine-direct instead of gramine-sgx does not encounter that error. I'm not sure what the difference is between gramine-direct and gramine-sgx but I'm guessing gramine-direct is not running in a secure enclave.

@hop-along-polly Can you say more why you are not sure on difference between gramine-direct and gramine-sgx ? Do we not state that clearly in our documentation or you did not get to read the basic documentation? BTW your guess is correct. I am just looking for feedback on where we need to fix our documentation, so other people new to gramine don't have the same confusion.

hop-along-polly Apr 19, 2023
Author

I was looking at this page which didn't have a lot of information. Otherwise I was looking at examples and the Gramine Manifest Syntax page

kailun-qin · 2023-04-19T02:38:01Z

kailun-qin
Apr 19, 2023
Maintainer

Hi @hop-along-polly, thanks for your report!

Would you pls kindly try increasing the enclave_size in your Gramine manifest to e.g., 4G and see whether the issue persists? I'm suspecting mmap() failure due to the shortage of memory. Thanks!

2 replies

hop-along-polly Apr 19, 2023
Author

I stumbled across this earlier in the day which got me passed the error I reported in the original post. I'm still getting various issues regarding permissions accessing files. In some cases I have to mount the file/folder into the Gramine instance others I just need to add to trusted_files. Never sure when or why I need to do this.

dimakuv Apr 19, 2023

The logic is like this:

Mounting files is required for any type of Gramine invocation (gramine-direct or gramine-sgx). Technical/security explanation of why this is so can be found here: https://gramine.readthedocs.io/en/latest/devel/features.html#file-systems
Specifying trusted files is required for gramine-sgx (the prefix sgx.trusted_files implies that this option is for SGX only). Technical/security explanation can be found on the same page.

I agree that mastering the art of writing manifest files in Gramine requires some experimentation... Some of the manifest options' names are historical and probably misleading. E.g., I don't like the part that we have so many different prefixes (loader., libos., sys., sgx.) -- they make sense for developers, but not for end users. We'll fix it one day.

hop-along-polly · 2023-04-19T06:37:26Z

hop-along-polly
Apr 19, 2023
Author

So after setting enclave_size to "4G" and adding some additional paths to fs.mounts and sgx.trusted_files I was able to get the app running in Gramine without error.

However I was using insecure__use_cmdline_argv = true which of course is an insecure setting. When I tried to replace that setting with the loader.argv setting which is a secure version of passing cmd line args I get the error uvicorn: can't open file '//modelscan.app:app': [Errno 2] No such file or directory

That error literally makes no sense, like zero sense, because with the insecure__use_cmdline_argv = true setting the command I run is gramine-sgx app /usr/bin/uvicorn modelscan.app:app --host 0.0.0.0 --port 8000. So the loader.argv equivalent of that command should be loader.argv = [ "/usr/bin/uvicorn", "modelscan.app:app", "--host", "0.0.0.0", "--port", "8000" ] but apparently it's not. Literally makes no sense.

3 replies

dimakuv Apr 19, 2023

@hop-along-polly If I understand correctly, the normal (non-Gramine) invocation looks like this:

python3 /usr/bin/uvicorn modelscan.app:app --host 0.0.0.0 --port 8000

Am I right? In this case, you must specify like this in the manifest:

loader.argv = [ "/usr/bin/python3.11", "/usr/bin/uvicorn", "modelscan.app:app", "--host", "0.0.0.0", "--port", "8000" ]

Note that the first argument (argv[0] in C parlance) is the absolute path to the Python3 executable.

Basically, loader.argv is always the same as you type for "normal" execution, plus expanding all paths/binary names to proper absolute paths.

hop-along-polly Apr 19, 2023
Author

The normal execution is uvicorn modelscan.app:app --host 0.0.0.0 --port 8000 The execution when using insecure__use_cmdline_argv = true is gramine-sgx app /usr/bin/uvicorn modelscan.app:app --host 0.0.0.0 --port. So I'm guessing the "insecure cmd line args are appended to the libos.entrypoint but when using loader.argv it overwrites the libos.entrypoint.

dimakuv Apr 19, 2023

So I'm guessing the "insecure cmd line args are appended to the libos.entrypoint but when using loader.argv it overwrites the libos.entrypoint.

Yes, Gramine doesn't overwrite libos.entrypoint -- in the sense that Gramine loads the entrypoint binary, but then passes arguments as specified in loader.argv, not adding anything to these arguments.

The execution when using insecure__use_cmdline_argv = true is gramine-sgx app /usr/bin/uvicorn modelscan.app:app --host 0.0.0.0 --port.

This is confusing to me. What is in the app.manifest.template then? Can you show its contents? What is the loader.entrypoint there?

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Pytorch cannot load libtorch_cpu.so when running in a Gramine Enclave #1293

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 3 comments 7 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

Pytorch cannot load libtorch_cpu.so when running in a Gramine Enclave #1293

hop-along-polly Apr 18, 2023

is-sgx-available output

Environment

Directory Structure

Contents of App.py (Redacted for IP protection)

The Gramine Manifest File

Commands Run

The Problem

The Question

Replies: 3 comments · 7 replies

hop-along-polly Apr 18, 2023 Author

monavij Apr 19, 2023 Collaborator

hop-along-polly Apr 19, 2023 Author

kailun-qin Apr 19, 2023 Maintainer

hop-along-polly Apr 19, 2023 Author

dimakuv Apr 19, 2023

hop-along-polly Apr 19, 2023 Author

dimakuv Apr 19, 2023

hop-along-polly Apr 19, 2023 Author

dimakuv Apr 19, 2023

hop-along-polly
Apr 18, 2023

Replies: 3 comments 7 replies

hop-along-polly
Apr 18, 2023
Author

monavij Apr 19, 2023
Collaborator

hop-along-polly Apr 19, 2023
Author

kailun-qin
Apr 19, 2023
Maintainer

hop-along-polly Apr 19, 2023
Author

hop-along-polly
Apr 19, 2023
Author

hop-along-polly Apr 19, 2023
Author