Skip to content

Commit

Permalink
[LibOS,Pal] Add flexible device-specific IOCTL support
Browse files Browse the repository at this point in the history
The newly added `ioctl()` syscall emulation on device-backed file
descriptors is pass-through. It is insecure by itself since the
emulation only passes the arguments to and from the untrusted memory.
It is the responsibility of the app developer to correctly use ioctls,
with security implications in mind.

On the Linux-SGX PAL, a set of IOCTL requests must be explicitly allowed
in the manifest via the new option `sgx.allowed_ioctls.[id].request`.
Also, the allowed IOCTLs' arguments (typically pointers to complex
nested objects) must be explicitly described in the manifest via the new
options `sgx.ioctl_structs.[id]` and a corresponding reference
`sgx.allowed_ioctls.[id].struct`; see docs for explanation of the IOCTL
struct format.

This commit adds a new LibOS test `device_ioctl` that tests the flexible
IOCTL logic against Gramine dummy device `/dev/gramine_test_dev`. This
device is found in companion repo `gramineproject/device-testing-tools`.

Signed-off-by: Dmitrii Kuvaiskii <[email protected]>
  • Loading branch information
dimakuv committed Jul 25, 2022
1 parent 07b5c62 commit 60c2c64
Show file tree
Hide file tree
Showing 20 changed files with 1,212 additions and 2 deletions.
94 changes: 94 additions & 0 deletions Documentation/manifest-syntax.rst
Original file line number Diff line number Diff line change
Expand Up @@ -717,6 +717,100 @@ they were listed as ``allowed_files``. (However, this policy still does not
allow writing/creating files specified as trusted.) This policy is a convenient
way to determine the set of files that the ported application uses.

Allowed IOCTLs
^^^^^^^^^^^^^^

::

sgx.ioctl_structs.[identifier] = [memory-layout-format]

sgx.allowed_ioctls.[identifier].request = [NUM]
sgx.allowed_ioctls.[identifier].struct = "[identifier-of-ioctl-struct]"

By default, Gramine with SGX disables all device-backed IOCTLs. This syntax
allows to explicitly allow a set of IOCTLs on devices (devices must be
explicitly mounted via ``fs.mounts`` manifest syntax). Only IOCTLs with the
``request`` argument found among the manifest-listed IOCTLs are allowed to
pass-through to the host. Each IOCTL entry must also contain a reference to an
IOCTL struct in its ``struct`` field.

Available IOCTL structs are described via ``sgx.ioctl_structs``. Each IOCTL
struct describes the memory layout of the ``arg`` argument (typically a pointer
to a complex nested object passed to the device). Description of the memory
layout is required for a deep copy of the argument. The memory layout is
described using the TOML syntax of inline arrays (for each new separate memory
region) and inline tables (for each sub-region in one memory region). Each
sub-region is described via the following keys:

- ``name`` is an optional name for this sub-region; mainly used to find
length-specifying fields and nested memory regions.
- ``align`` is an optional alignment of the memory region; may be specified only
in the first sub-region of a memory region (all other sub-regions are
contigious with the first sub-region, so specifying their alignment doesn't
make sense).
- ``size`` is a mandatory size of this sub-region. The ``size`` field may be a
string with the name of another field that contains the size value or an
integer with the constant size measured in ``units`` (default unit is 1 byte;
also see below). For example, ``size = "strlen"`` denotes a size field that
will be calculated dynamically during IOCTL execution based on the sub-region
named ``strlen``, whereas ``size = 16`` denotes a sub-region of size 16B. Note
that for ``ptr`` sub-regions, the ``size`` field has a different meaning: it
denotes the number of adjacent memory regions (in other words, it denotes the
number of items in the ``ptr`` array).
- ``unit`` is an optional unit of measurement for ``size``. It is 1 byte by
default. Unit of measurement must be a constant integer. For example,
``size = "strlen"`` and ``unit = 2`` denote a wide-char string (where each
character is 2B long) of a dynamically calculated length.
- ``adjust`` is an optional integer adjustment for ``size``. It is 0 bytes by
default. This field must be a constant (possibly negative) integer. For
example, ``adjust = -8`` and ``size = 12`` results in a total size of 4B.
- ``type = ["none" | "out" | "in" | "inout"]`` is an optional direction of copy
for this sub-region. For example, ``type = "out"`` denotes a sub-region to be
copied out of the enclave to untrusted memory, i.e., this sub-region is an
input to the host device. The default value is ``none`` which is useful for
e.g. padding of structs. This field may be ommitted if the ``ptr`` field is
specified for this sub-region (pointer sub-regions contain the pointer value
which will be unconditionally rewired to point to untrusted memory).
- ``ptr = [ another memory region ]`` or ``ptr = "another-memory-region"``
specifies a pointer to another, nested memory region. This field is required
when describing complex IOCTL structs. Such pointer memory region always has
the implicit size of 8B, and the pointer value is always rewired to the memory
region in untrusted memory (containing a copied-out nested memory region). If
``ptr`` is specified together with ``size``, it describes not just a pointer
but an array of these memory regions. A special keyword ``ptr = "this"``
specifies a pointer to the memory region of the IOCTL struct's root memory
layout.

Consider this simple example::

sgx.ioctl_structs.st1 = [ { ptr=[ {name="nested_region", align=4096, size=4096, type="out"} ] } ]

The above example specifies a root struct (first memory region) that consists
of a single sub-region that contains an 8-byte pointer value. This pointer
points to another memory region in enclave memory that contains a single
sub-region of size 4KB and that must be 4KB-aligned. This nested sub-region has
a name ``nested_region`` (not used, only for illustrative purposes). Also, this
nested sub-region is copied out of the enclave. The pointer value of the first
memory region is rewired to point to the copied-out second memory region in
untrusted memory. No fields/memory regions are copied back from untrusted memory
inside the enclave after an IOCTL with this struct executes.

If the IOCTL's third argument is simply an integer (or unused at all), then the
syntax must specify the struct as an empty TOML array::

sgx.ioctl_structs.st2 = [ ]

IOCTLs that use these structs are defined like this::

sgx.allowed_ioctls.io1.request = 0x12345678
sgx.allowed_ioctls.io1.struct = "st1"

sgx.allowed_ioctls.io2.request = 0x87654321
sgx.allowed_ioctls.io2.struct = "st1"

sgx.allowed_ioctls.io3.request = 0x43218765 # this IOCTL's arg is passed as-is
sgx.allowed_ioctls.io3.struct = "st2"

Attestation and quotes
^^^^^^^^^^^^^^^^^^^^^^

Expand Down
24 changes: 22 additions & 2 deletions libos/src/sys/libos_ioctl.c
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,7 @@
#include "libos_signal.h"
#include "libos_table.h"
#include "pal.h"
#include "stat.h"

static void signal_io(IDTYPE caller, void* arg) {
__UNUSED(caller);
Expand Down Expand Up @@ -116,9 +117,28 @@ long libos_syscall_ioctl(unsigned int fd, unsigned int cmd, unsigned long arg) {
ret = 0;
break;
}
default:
ret = -ENOSYS;
default: {
lock(&g_dcache_lock);
bool is_host_dev = hdl->type == TYPE_CHROOT && hdl->dentry->inode &&
hdl->dentry->inode->type == S_IFCHR;
unlock(&g_dcache_lock);

if (!is_host_dev) {
ret = -ENOSYS;
break;
}

int cmd_ret;
ret = PalDeviceIoControl(hdl->pal_handle, cmd, arg, &cmd_ret);
if (ret < 0) {
ret = pal_to_unix_errno(ret);
break;
}

assert(ret == 0);
ret = cmd_ret;
break;
}
}

put_handle(hdl);
Expand Down
200 changes: 200 additions & 0 deletions libos/test/regression/device_ioctl.c
Original file line number Diff line number Diff line change
@@ -0,0 +1,200 @@
#include <alloca.h>
#include <err.h>
#include <errno.h>
#include <fcntl.h>
#include <stdint.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/ioctl.h>
#include <sys/types.h>
#include <unistd.h>

#include "rw_file.h"

#define STRING_READWRITE "Hello world via read/write\n"
#define STRING_IOCTL "Hello world via ioctls\n"
#define STRING_IOCTL_REPLACED "He$$0 w0r$d via i0ct$s\n"

struct gramine_test_dev_ioctl_write {
size_t buf_size; /* in */
const char* buf; /* in */
ssize_t off; /* in/out -- updated after write */
ssize_t copied; /* out -- how many bytes were actually written */
};

struct gramine_test_dev_ioctl_read {
size_t buf_size; /* in */
char* buf; /* out */
ssize_t off; /* in/out -- updated after read */
ssize_t copied; /* out -- how many bytes were actually read */
};

struct gramine_test_dev_ioctl_replace_char {
char src; /* in */
char dst; /* in */
char pad[6];
};

struct gramine_test_dev_ioctl_replace_arr {
/* array of replacements, e.g. replacements_cnt == 2 and [`l` -> `$`, `o` -> `0`] */
size_t replacements_cnt;
struct gramine_test_dev_ioctl_replace_char* replacements_arr;
};

struct gramine_test_dev_ioctl_replace_list {
/* list of replacements, e.g. [`l` -> `$`, next points to `o` -> `0`, next points to NULL] */
struct gramine_test_dev_ioctl_replace_char replacement;
struct gramine_test_dev_ioctl_replace_list* next;
};

#define GRAMINE_TEST_DEV_IOCTL_BASE 0x33

#define GRAMINE_TEST_DEV_IOCTL_REWIND _IO(GRAMINE_TEST_DEV_IOCTL_BASE, 0x00)
#define GRAMINE_TEST_DEV_IOCTL_WRITE _IOWR(GRAMINE_TEST_DEV_IOCTL_BASE, 0x01, \
struct gramine_test_dev_ioctl_write)
#define GRAMINE_TEST_DEV_IOCTL_READ _IOWR(GRAMINE_TEST_DEV_IOCTL_BASE, 0x02, \
struct gramine_test_dev_ioctl_read)
#define GRAMINE_TEST_DEV_IOCTL_GETSIZE _IO(GRAMINE_TEST_DEV_IOCTL_BASE, 0x03)
#define GRAMINE_TEST_DEV_IOCTL_CLEAR _IO(GRAMINE_TEST_DEV_IOCTL_BASE, 0x04)
#define GRAMINE_TEST_DEV_IOCTL_REPLACE_ARR _IOW(GRAMINE_TEST_DEV_IOCTL_BASE, 0x05, \
struct gramine_test_dev_ioctl_replace_arr)
#define GRAMINE_TEST_DEV_IOCTL_REPLACE_LIST _IOW(GRAMINE_TEST_DEV_IOCTL_BASE, 0x06, \
struct gramine_test_dev_ioctl_replace_list)

int main(int argc, char* argv[]) {
int ret;
ssize_t bytes;
char buf[64];

int devfd = open("/dev/gramine_test_dev", O_RDWR);
if (devfd < 0)
err(1, "/dev/gramine_test_dev open");

/* test 1 -- use write() and read() syscalls */
bytes = posix_fd_write(devfd, STRING_READWRITE, sizeof(STRING_READWRITE));
if (bytes < 0)
return EXIT_FAILURE;

/* lseek() doesn't work in Gramine because it is fully emulated in LibOS and therefore lseek()
* is not aware of device-specific semantics; instead we use a device-specific ioctl() */
off_t offset = ioctl(devfd, GRAMINE_TEST_DEV_IOCTL_REWIND);
if (offset < 0)
err(1, "/dev/gramine_test_dev ioctl(GRAMINE_TEST_DEV_IOCTL_REWIND)");
if (offset > 0)
errx(1, "/dev/gramine_test_dev ioctl(GRAMINE_TEST_DEV_IOCTL_REWIND) didn't return 0 "
"(returned: %ld)", offset);

memset(&buf, 0, sizeof(buf));
bytes = posix_fd_read(devfd, buf, sizeof(buf) - 1);
if (bytes < 0)
return EXIT_FAILURE;

if (strcmp(buf, STRING_READWRITE))
errx(1, "read `%s` from /dev/gramine_test_dev but expected `%s`", buf, STRING_READWRITE);

ssize_t devfd_size = ioctl(devfd, GRAMINE_TEST_DEV_IOCTL_GETSIZE);
if (devfd_size < 0)
err(1, "/dev/gramine_test_dev ioctl(GRAMINE_TEST_DEV_IOCTL_GETSIZE)");
if (devfd_size != sizeof(STRING_READWRITE))
errx(1, "/dev/gramine_test_dev ioctl(GRAMINE_TEST_DEV_IOCTL_GETSIZE) didn't return %lu "
"(returned: %ld)", sizeof(STRING_READWRITE), devfd_size);

/* test 2 -- use ioctl(GRAMINE_TEST_DEV_IOCTL_WRITE) and ioctl(GRAMINE_TEST_DEV_IOCTL_READ)
* syscalls */
ret = ioctl(devfd, GRAMINE_TEST_DEV_IOCTL_CLEAR);
if (ret < 0)
err(1, "/dev/gramine_test_dev ioctl(GRAMINE_TEST_DEV_IOCTL_CLEAR)");

struct gramine_test_dev_ioctl_write write_arg = {
.buf_size = sizeof(STRING_IOCTL),
.buf = STRING_IOCTL,
.off = 0
};
ret = ioctl(devfd, GRAMINE_TEST_DEV_IOCTL_WRITE, &write_arg);
if (ret < 0)
err(1, "/dev/gramine_test_dev ioctl(GRAMINE_TEST_DEV_IOCTL_WRITE)");
if (write_arg.off != sizeof(STRING_IOCTL))
errx(1, "/dev/gramine_test_dev ioctl(GRAMINE_TEST_DEV_IOCTL_WRITE) didn't update offset "
"to %lu (returned: %ld)", sizeof(STRING_IOCTL), write_arg.off);
if (write_arg.copied != sizeof(STRING_IOCTL))
errx(1, "/dev/gramine_test_dev ioctl(GRAMINE_TEST_DEV_IOCTL_WRITE) didn't copy %lu bytes "
"(returned: %ld)", sizeof(STRING_IOCTL), write_arg.copied);

memset(buf, 0, sizeof(buf));
struct gramine_test_dev_ioctl_read read_arg = {
.buf_size = sizeof(buf) - 1,
.buf = buf,
.off = 0
};
ret = ioctl(devfd, GRAMINE_TEST_DEV_IOCTL_READ, &read_arg);
if (ret < 0)
err(1, "/dev/gramine_test_dev ioctl(GRAMINE_TEST_DEV_IOCTL_READ)");
if (read_arg.off != sizeof(STRING_IOCTL))
errx(1, "/dev/gramine_test_dev ioctl(GRAMINE_TEST_DEV_IOCTL_READ) didn't update offset "
"to %lu (returned: %ld)", sizeof(STRING_IOCTL), read_arg.off);
if (read_arg.copied != sizeof(STRING_IOCTL))
errx(1, "/dev/gramine_test_dev ioctl(GRAMINE_TEST_DEV_IOCTL_READ) didn't copy %lu bytes "
"(returned: %ld)", sizeof(STRING_IOCTL), read_arg.copied);

if (strcmp(buf, STRING_IOCTL))
errx(1, "read `%s` from /dev/gramine_test_dev but expected `%s`", buf, STRING_IOCTL);

devfd_size = ioctl(devfd, GRAMINE_TEST_DEV_IOCTL_GETSIZE);
if (devfd_size < 0)
err(1, "/dev/gramine_test_dev ioctl(GRAMINE_TEST_DEV_IOCTL_GETSIZE)");
if (devfd_size != sizeof(STRING_IOCTL))
errx(1, "/dev/gramine_test_dev ioctl(GRAMINE_TEST_DEV_IOCTL_GETSIZE) didn't return %lu "
"(returned: %ld)", sizeof(STRING_IOCTL), devfd_size);

/* test 3 -- use complex ioctl(GRAMINE_TEST_DEV_IOCTL_REPLACE_ARR) syscall */
struct gramine_test_dev_ioctl_replace_char replace_chars[] = {
{ .src = 'l', .dst = '$' },
{ .src = 'o', .dst = '0' }
};
struct gramine_test_dev_ioctl_replace_arr replace_arr = {
.replacements_cnt = 2,
.replacements_arr = replace_chars
};
ret = ioctl(devfd, GRAMINE_TEST_DEV_IOCTL_REPLACE_ARR, &replace_arr);
if (ret < 0)
err(1, "/dev/gramine_test_dev ioctl(GRAMINE_TEST_DEV_IOCTL_REPLACE_ARR)");

memset(buf, 0, sizeof(buf));
read_arg.off = 0;
ret = ioctl(devfd, GRAMINE_TEST_DEV_IOCTL_READ, &read_arg);
if (ret < 0)
err(1, "/dev/gramine_test_dev ioctl(GRAMINE_TEST_DEV_IOCTL_READ)");
if (strcmp(buf, STRING_IOCTL_REPLACED))
errx(1, "read `%s` from /dev/gramine_test_dev but expected `%s`", buf,
STRING_IOCTL_REPLACED);

/* test 4 -- use complex ioctl(GRAMINE_TEST_DEV_IOCTL_REPLACE_LIST) syscall */
struct gramine_test_dev_ioctl_replace_list replace_list_2 = {
.replacement = { .src = '0', .dst = 'o' },
.next = NULL
};
struct gramine_test_dev_ioctl_replace_list replace_list = {
.replacement = { .src = '$', .dst = 'l' },
.next = &replace_list_2
};

ret = ioctl(devfd, GRAMINE_TEST_DEV_IOCTL_REPLACE_LIST, &replace_list);
if (ret < 0)
err(1, "/dev/gramine_test_dev ioctl(GRAMINE_TEST_DEV_IOCTL_REPLACE_LIST)");

memset(buf, 0, sizeof(buf));
read_arg.off = 0;
ret = ioctl(devfd, GRAMINE_TEST_DEV_IOCTL_READ, &read_arg);
if (ret < 0)
err(1, "/dev/gramine_test_dev ioctl(GRAMINE_TEST_DEV_IOCTL_READ)");
if (strcmp(buf, STRING_IOCTL))
errx(1, "read `%s` from /dev/gramine_test_dev but expected `%s`", buf, STRING_IOCTL);

ret = close(devfd);
if (ret < 0)
err(1, "/dev/gramine_test_dev close");

puts("TEST OK");
return 0;
}
Loading

0 comments on commit 60c2c64

Please sign in to comment.