With Go program, when server program does a read, getting permission denied.

[sudharkrish]

Description of the problem

With Go program, when server program does a read, getting permission denied.

Steps to reproduce

Able to reproduce on a recent graphene pull(Aug 30th, 2021), commit-id-> c321726229eaf0a1b52dc5e2507c9cfab423ea94
Also able to reproduce on https://github.com/oscarlab/graphene/releases/tag/v1.2-rc1

Providing Sample Go program and scripts to reproduce the issue.

In graphene repo, under your /home->/graphene/Examples directory, copy this zip file->(
go_sample.zip) , and then unzip it,
to create go_sample directory under /graphene/Examples/go_sample.

Under /graphene/Examples/go_sample$
Run the script -> ./launch_in_graphene_locally.sh
This will build the sample Go program(in a docker container), and then do a graphene-sgx build, and
then it will launch it locally on your host system.

When the Go Server code does a read, getting permission denied.

Expected results

Output below, when running the same Go program, outside of Graphene.
Examples/go_sample$ ./main
SK_DBG: listening on 172.17.0.1:8805
client: wrote: hello
server: read: hello

Actual results

In Graphene, when the Go Server code does a read, getting permission denied.

[P17460:T1:main] debug: Allocating stack at 0x0 (size = 8388608)
[P17460:T1:main] debug: loading "file:./main"
[P17460:T1:main] debug: adding a library for gdb: file:./main
[P17460:T1:main] debug: Creating pipe: pipe.srv:17460
debug: sock_getopt (fd = 11, sockopt addr = 0x7ffda07f52b0) is not implemented and always returns 0
[P17460:T1:main] debug: Shim process initialized
[P17460:shim] debug: IPC worker started
[P17460:T1:main] debug: Created sigframe for sig: 23 at 0xc4009390 (handler: 0x460be0, restorer: 0x460d20)
debug: sock_getopt (fd = 12, sockopt addr = 0x7ffda07f52b0) is not implemented and always returns 0
[P17460:T1:main] debug: Creating pipe: pipe.srv:8fcf6dd6dc08723b8328139ef1955390c16982da3379e1b7f6c07bc4bdc66514
debug: sock_getopt (fd = 15, sockopt addr = 0x7ffda07f52b0) is not implemented and always returns 0
debug: sock_getopt (fd = 16, sockopt addr = 0x7ffda07f52b0) is not implemented and always returns 0
debug: sock_getopt (fd = 17, sockopt addr = 0x7ffda07f52b0) is not implemented and always returns 0
[P17460:T1:main] debug: add fd 5 (handle 0xfb098610) to epoll handle 0xfb098550
[P17460:T1:main] debug: add fd 3 (handle 0xfb0983c0) to epoll handle 0xfb098550
SK_DBG: listening on 172.17.0.1:8805
Allowing access to an unknown file due to file_check_policy settings: file:/etc/localtime
2021/09/03 18:08:33 read udp 172.17.0.1:8805: read: permission denied

Additional information

Go sample code under gopro2 folder in zip file attached.
When Go Program calls net.ListenUDP, this api invokes 2 syscalls, 1. to create socket, 2. bind
When Go server program calls net.ListenUDP these 2 syscalls are successful.
Go program launches a light-weight thread that runs the Client-function, which tries to connect to the server using
Go's net.DialUDP function, and then does a write. Later, server code tries to do a read, and this is where it fails,
with permission denied error(shown in the log above).

[yanzhichao]

try to config above parameter to manifest

sgx.allowed_files.etchostname = "file:/etc/hostname"
sgx.allowed_files.hosts = "file:/etc/hosts"
sgx.allowed_files.resolv = "file:/etc/resolv.conf"

Allowing access to an unknown file due to file_check_policy settings: file:/etc/localtime

and ensure there have /etc/localtime in your docker images

[dimakuv]

Is it possible to try this Go program in TCP mode (switch from a UDP connection to the TCP connection)? This will give us some more info on which part of Graphene fails.

In particular, the UDP handling code in Graphene is buggy -- not many programs use UDP, so didn't have enough testing in Graphene.

[sudharkrish]

I tested with some sample Go program that launches a TCP server, that listens on a TCP port. Client is able to connect to it, and send a buffer. TCP server is able to read that buffer(sent by client), without any issues, when running in Graphene.

Attaching the test TCP program in Go->(gopro_tcp_testing.zip )

Also here are the logs, when running TCP program in Go(using Graphene-SGX):

//Part of graphene's log(when launching TCP server)
[P58259:T1:main] debug: Allocating stack at 0x0 (size = 8388608)
[P58259:T1:main] debug: loading "file:./main"
[P58259:T1:main] debug: adding a library for gdb: file:./main
[P58259:T1:main] debug: Creating pipe: pipe.srv:58259
debug: sock_getopt (fd = 11, sockopt addr = 0x7ffe798c5ba0) is not implemented and always returns 0
[P58259:T1:main] debug: Shim process initialized
[P58259:shim] debug: IPC worker started
[P58259:T1:main] debug: Created sigframe for sig: 23 at 0x38009390 (handler: 0x460b80, restorer: 0x460cc0)
debug: sock_getopt (fd = 12, sockopt addr = 0x7ffe798c5ba0) is not implemented and always returns 0
[P58259:T1:main] debug: Creating pipe: pipe.srv:8f97688268634d0d902b396daf799f4cf21356ff4d71f2eedd0be36fafe83fa5
debug: sock_getopt (fd = 15, sockopt addr = 0x7ffe798c5ba0) is not implemented and always returns 0
debug: sock_getopt (fd = 16, sockopt addr = 0x7ffe798c5ba0) is not implemented and always returns 0
debug: sock_getopt (fd = 17, sockopt addr = 0x7ffe798c5ba0) is not implemented and always returns 0
[P58259:T1:main] debug: add fd 5 (handle 0xfb098610) to epoll handle 0xfb098550
[P58259:T1:main] debug: add fd 3 (handle 0xfb0983c0) to epoll handle 0xfb098550
Listening on 172.17.0.1:8805
debug: sock_getopt (fd = 15, sockopt addr = 0x7ffe798c5ba0) is not implemented and always returns 0
[P58259:T1:main] debug: add fd 7 (handle 0xfb098860) to epoll handle 0xfb098550
post-read, readLen->%d, buf->%s: 19 [116 101 115 116 32 111 117 116 32 116 104 101 32 115 101 114 118 101 114 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0]
[P58259:T1:main] debug: delete fd 7 (handle 0xfb098860) from epoll handle 0xfb098550

//Client -> Using a terminal:
echo -n "test out the server" | nc -v 172.17.0.1 8805
Connection to 172.17.0.1 8805 port [tcp/*] succeeded!
Message received.

[sudharkrish]

@dimakuv , you seem to be right that when I tried a sample TCP server Go program, it worked fine in Graphene. So, it appears like issue is seen when having UDP server in Go running in Graphene. Also, for the actual Go application server that I am using-> https://github.com/omec-project/upf-epc/tree/master/pfcpiface, the team that supports this application, have informed that they need to use UDP server for their use-case.

[dimakuv]

@sudharkrish Have you tried gramineproject/graphene#2679? This solves a problem with partial writes, both in TCP and in UDP modes. Not sure if this can help.

Anyway, can you run your failing app with loader.log_level = "all"? And attach the resulting log. May show some interesting info.

[sudharkrish]

@dimakuv , I applied PR gramineproject/graphene#2679 but it did not make a difference. I am attaching logs with loader.log_level = "all"-> (
udp_graphene_logs1.zip)

[sudharkrish]

@dimakuv did some debugging and here is some info.
When Go Program calls net.ListenUDP, this api invokes 2 syscalls, 1. to create socket, 2. bind
When Go server program calls net.ListenUDP these 2 syscalls are successful.
I can break in LibOS at shim_do_socket and shim_do_bind

Later, when Go server code, does a read, I can break at
shim_do_read, and below is the trace:

shim_do_read-> (hdl type is socket)-> shim_socket.c::do_recvmsg-> DkStreamRead(which returns -13 Permission Denied)...

DkStreamRead calls _DkStreamRead...and here
handler for udp’s socket_read function is NOT set, it is NULL,
in _DkStreamRead, -PAL_ERROR_NOTSUPPORT is returned,
which gets converted to EACCES
error thro pal_to_unix_errno in do_recvmsg function.

Looked into PAL code, when socket fd gets created for UDP stream, the handler-functions in db_socket, such as udp_receive
is NOT getting set for udp-server object.

From the code, it looks like Graphene’s PAL does NOT support a UDP server object that
can support udp_receive(which does NOT take the addr parameter)

So when udp-server’s code in Go (sConn.Read(buf) ), invokes a read(without address parameter), Graphene’s PAL layer errors out saying -PAL_ERROR_NOTSUPPORT,
Which then shows up as permission denied error for read.

struct handle_ops g_udpsrv_ops = {
.getname = &socket_getname,
.open = &udp_open,
.readbyaddr = &udp_receivebyaddr,
.writebyaddr = &udp_sendbyaddr,
.delete = &socket_delete,
.close = &socket_close,
.attrquerybyhdl = &socket_attrquerybyhdl,
.attrsetbyhdl = &socket_attrsetbyhdl,
};

I added an experimental change to add entry for .read = &udp_receive in the above struct for g_udpsrv_ops

And with this change in Pal, I am seeing a different error,
read: transport endpoint is not connected

[sudharkrish]

Here is some GDB trace output:

Thread 1 "loader" hit Breakpoint 5, shim_do_read (fd=3, buf=0xc8184aa8, count=1024) at ../LibOS/shim/src/sys/shim_open.c:45
45          if (!is_user_memory_writable(buf, count))
(gdb) c
Continuing.
[P1:T2:main] trace: ---- shim_futex(0x5a6978, FUTEX_PRIVATE|FUTEX_WAIT, 0, 0xc80e9eb0, 0x0, 0) ...

Thread 1 "loader" hit Breakpoint 6, do_recvmsg (fd=3, bufs=0xfb0913e0, nbufs=1, flags=0, addr=0x0, addrlen=0x0) at ../LibOS/shim/src/sys/shim_socket.c:1303
1303        struct shim_handle* hdl = get_fd_handle(fd, NULL, NULL);
(gdb) c
Continuing.

Thread 1 "loader" hit Breakpoint 8, DkStreamRead (handle=0xfb6b3d30 <g_mem_pool+863920>, offset=0, count=0xfb091300, buffer=0xc8184aa8, source=0x0, size=0)
    at db_streams.c:240
240         if (!handle || !buffer) {
(gdb) n
244         int64_t ret = _DkStreamRead(handle, offset, *count, (void*)buffer, size ? (char*)source : NULL,
(gdb) s
_DkStreamRead (handle=0xfb6b3d30 <g_mem_pool+863920>, offset=0, count=1024, buf=0xc8184aa8, addr=0x0, addrlen=0) at db_streams.c:216
216         const struct handle_ops* ops = HANDLE_OPS(handle);
(gdb) n
218         if (!ops)
(gdb) n
223         if (addr) {
(gdb) n
229             if (!ops->read)
(gdb) print ops->read
$1 = (int64_t (*)(PAL_HANDLE, uint64_t, uint64_t, void *)) 0x0
(gdb) print ops
$2 = (const struct handle_ops *) 0xfb5e0840 <g_udpsrv_ops>
(gdb) print ops->read
$3 = (int64_t (*)(PAL_HANDLE, uint64_t, uint64_t, void *)) 0x0
(gdb) print ops->open
$4 = (int (*)(PAL_HANDLE *, const char *, const char *, int, int, int, int)) 0xfb38d490 <udp_open>
(gdb) print ops->write
$5 = (int64_t (*)(PAL_HANDLE, uint64_t, uint64_t, const void *)) 0x0
(gdb) print ops->readbyaddr 
$6 = (int64_t (*)(PAL_HANDLE, uint64_t, uint64_t, void *, char *, size_t)) 0xfb38c860 <udp_receivebyaddr>
(gdb) print ops->write
write        writebyaddr  
(gdb) print ops->writebyaddr 
$7 = (int64_t (*)(PAL_HANDLE, uint64_t, uint64_t, const void *, const char *, size_t)) 0xfb38db90 <udp_sendbyaddr>
(gdb) n
[P1:T2:main] trace: ---- return from shim_futex(...) = -110
[P1:T2:main] trace: ---- shim_clock_gettime(1, 0xc80e9ea0) = 0x0
[P1:T2:main] trace: ---- shim_clock_gettime(1, 0xc80e9f00) = 0x0
[P1:T2:main] trace: ---- shim_epoll_pwait(4, 0xc80e9920, 128, 0, 0x0, 0x0) ...
[P1:T2:main] trace: ---- return from shim_epoll_pwait(...) = 0x0
[P1:T2:main] trace: ---- shim_nanosleep([0,20000], 0x0) ...
[P1:T2:main] trace: ---- return from shim_nanosleep(...) = 0x0
[P1:T2:main] trace: ---- shim_clock_gettime(1, 0xc80e9f00) = 0x0
[P1:T2:main] trace: ---- shim_clock_gettime(1, 0xc80e9ea0) = 0x0
[P1:T2:main] trace: ---- shim_futex(0x5a6978, FUTEX_PRIVATE|FUTEX_WAIT, 0, 0xc80e9eb0, 0x0, 0) ...
230                 return -PAL_ERROR_NOTSUPPORT;
(gdb) bt
#0  _DkStreamRead (handle=0xfb6b3d30 <g_mem_pool+863920>, offset=0, count=1024, buf=0xc8184aa8, addr=0x0, addrlen=0) at db_streams.c:230
gramineproject/graphene#1  0x00000000fb382dcc in DkStreamRead (handle=0xfb6b3d30 <g_mem_pool+863920>, offset=0, count=0xfb091300, buffer=0xc8184aa8, source=0x0, size=0)
    at db_streams.c:244
gramineproject/graphene#2  0x00000000fb13d855 in do_recvmsg (fd=3, bufs=0xfb0913e0, nbufs=1, flags=0, addr=0x0, addrlen=0x0) at ../LibOS/shim/src/sys/shim_socket.c:1443
gramineproject/graphene#3  0x00000000fb13dcf4 in shim_do_recvfrom (sockfd=3, buf=0xc8184aa8, len=1024, flags=0, addr=0x0, addrlen=0x0) at ../LibOS/shim/src/sys/shim_socket.c:1571
gramineproject/graphene#4  0x00000000fb134d21 in shim_do_read (fd=3, buf=0xc8184aa8, count=1024) at ../LibOS/shim/src/sys/shim_open.c:55
gramineproject/graphene#5  0x00000000fb121349 in shim_emulate_syscall (context=0xfb091f38) at ../LibOS/shim/src/shim_syscalls.c:36
gramineproject/graphene#6  0x00000000fb142ada in syscalldb () at ../LibOS/shim/src/arch/x86_64/syscallas.S:173
gramineproject/graphene#7  0x00000000fb142add in __morestack () at ../LibOS/shim/src/arch/x86_64/syscallas.S:204
gramineproject/graphene#8  0x000000000047cd3b in syscall.Syscall () at /usr/local/go/src/syscall/asm_linux_amd64.s:25
gramineproject/graphene#9  0x000000000047b8ed in syscall.read (fd=0, p=...) at /usr/local/go/src/syscall/zsyscall_linux_amd64.go:687
gramineproject/graphene#10 0x000000000048a045 in syscall.Read (fd=3, n=<optimized out>, err=..., p=...) at /usr/local/go/src/syscall/syscall_unix.go:189
gramineproject/graphene#11 internal/poll.ignoringEINTRIO (fd=3, fn=<optimized out>, p=...) at /usr/local/go/src/internal/poll/fd_unix.go:582
gramineproject/graphene#12 internal/poll.(*FD).Read (fd=0xc819c000, p=..., ~r1=<optimized out>, ~r2=...) at /usr/local/go/src/internal/poll/fd_unix.go:163
gramineproject/graphene#13 0x00000000004aafa9 in net.(*netFD).Read (fd=0xc819c000, p=...) at /usr/local/go/src/net/fd_posix.go:56
gramineproject/graphene#14 0x00000000004b53e5 in net.(*conn).Read (c=0xc800e028, b=...) at /usr/local/go/src/net/net.go:183
gramineproject/graphene#15 0x00000000004c0505 in main.main () at /main/main.go:23
gramineproject/graphene#16 0x0000000000433da7 in runtime.main () at /usr/local/go/src/runtime/proc.go:255
gramineproject/graphene#17 0x000000000045f0a1 in runtime.goexit () at /usr/local/go/src/runtime/asm_amd64.s:1581
gramineproject/graphene#18 0x0000000000000000 in ?? ()

Since the handler for udp’s socket_read function is NOT set, it is NULL,
in _DkStreamRead, -PAL_ERROR_NOTSUPPORT is returned,
which gets converted to EACCES
error thro pal_to_unix_errno in do_recvmsg function.

int64_t _DkStreamRead(PAL_HANDLE handle, uint64_t offset, uint64_t count, void* buf, char* addr,
                      int addrlen) {
    const struct handle_ops* ops = HANDLE_OPS(handle);
 
    if (!ops)
        return -PAL_ERROR_BADHANDLE;
 
    int64_t ret;
 
    if (addr) {
        if (!ops->readbyaddr)
            return -PAL_ERROR_NOTSUPPORT;
 
        ret = ops->readbyaddr(handle, offset, count, buf, addr, addrlen);
    } else {
        if (!ops->read)
            return -PAL_ERROR_NOTSUPPORT;

[dimakuv]

@sudharkrish Thanks for debugging this. Indeed, you are correct. UDP support in the PAL code is... bad.

To explain again what Sudha already found out. (We have the same problem in Linux and Linux-SGX PALs, so below I describe on the example of the Linux PAL.)

We have two PAL handle types in Gramine: PAL_TYPE_UDP and PAL_TYPE_UDPSRV.

We have two implementations of the recv function for UDP: udp_receive() and udp_receivebyaddr(). See this:

gramine/Pal/src/host/Linux/db_sockets.c

Lines 715 to 785 in 74420be

	static int64_t udp_receive(PAL_HANDLE handle, uint64_t offset, size_t len, void* buf) {
	if (offset)
	return -PAL_ERROR_INVAL;
	if (HANDLE_HDR(handle)->type != PAL_TYPE_UDP)
	return -PAL_ERROR_NOTCONNECTION;
	if (handle->sock.fd == PAL_IDX_POISON)
	return -PAL_ERROR_BADHANDLE;
	struct msghdr hdr;
	struct iovec iov;
	iov.iov_base = buf;
	iov.iov_len = len;
	hdr.msg_name = NULL;
	hdr.msg_namelen = 0;
	hdr.msg_iov = &iov;
	hdr.msg_iovlen = 1;
	hdr.msg_control = NULL;
	hdr.msg_controllen = 0;
	hdr.msg_flags = 0;
	int64_t bytes = DO_SYSCALL(recvmsg, handle->sock.fd, &hdr, 0);
	if (bytes < 0)
	return unix_to_pal_error(bytes);
	return bytes;
	}
	static int64_t udp_receivebyaddr(PAL_HANDLE handle, uint64_t offset, size_t len, void* buf,
	char* addr, size_t addrlen) {
	if (offset)
	return -PAL_ERROR_INVAL;
	if (HANDLE_HDR(handle)->type != PAL_TYPE_UDPSRV)
	return -PAL_ERROR_NOTCONNECTION;
	if (handle->sock.fd == PAL_IDX_POISON)
	return -PAL_ERROR_BADHANDLE;
	struct sockaddr_storage conn_addr;
	struct msghdr hdr;
	struct iovec iov;
	iov.iov_base = buf;
	iov.iov_len = len;
	hdr.msg_name = &conn_addr;
	hdr.msg_namelen = sizeof(conn_addr);
	hdr.msg_iov = &iov;
	hdr.msg_iovlen = 1;
	hdr.msg_control = NULL;
	hdr.msg_controllen = 0;
	hdr.msg_flags = 0;
	int64_t bytes = DO_SYSCALL(recvmsg, handle->sock.fd, &hdr, 0);
	if (bytes < 0)
	return unix_to_pal_error(bytes);
	char* addr_uri = strcpy_static(addr, URI_PREFIX_UDP, addrlen);
	if (!addr_uri)
	return -PAL_ERROR_OVERFLOW;
	int ret = inet_create_uri(addr_uri, addr + addrlen - addr_uri, (struct sockaddr*)&conn_addr,
	hdr.msg_namelen, NULL);
	if (ret < 0)
	return ret;
	return bytes;
	}

So far so good. But then we have rather interesting mapping "PAL handle type -> supported operations" here:

gramine/Pal/src/host/Linux/db_sockets.c

Lines 1150 to 1170 in 74420be

	struct handle_ops g_udp_ops = {
	.getname = &socket_getname,
	.open = &udp_open,
	.read = &udp_receive,
	.write = &udp_send,
	.delete = &socket_delete,
	.close = &socket_close,
	.attrquerybyhdl = &socket_attrquerybyhdl,
	.attrsetbyhdl = &socket_attrsetbyhdl,
	};
	struct handle_ops g_udpsrv_ops = {
	.getname = &socket_getname,
	.open = &udp_open,
	.readbyaddr = &udp_receivebyaddr,
	.writebyaddr = &udp_sendbyaddr,
	.delete = &socket_delete,
	.close = &socket_close,
	.attrquerybyhdl = &socket_attrquerybyhdl,
	.attrsetbyhdl = &socket_attrsetbyhdl,
	};

In other words:

• PAL_TYPE_UDP (used by UDP client apps) maps only to udp_receive()
• PAL_TYPE_UDPSRV (used by UDP server apps) maps only to udp_receivebyaddr()

So the UDP client can never use udp_receivebyaddr(), and UDP server can never use udp_receive(). This difference in behavior for UDP client and server was always in Graphene/Gramine, why -- I have no idea. Note that there is no difference in TCP client and server behavior (of course, TCP is different from UDP, so maybe there is some logic in this).

Conclusion: Our current UDP code doesn't support the code in this issue. Someone needs to add this functionality...

[dimakuv]

I added an experimental change to add entry for .read = &udp_receive in the above struct for g_udpsrv_ops
And with this change in Pal, I am seeing a different error,
read: transport endpoint is not connected

You see this error because we have these checks:

• gramine/Pal/src/host/Linux/db_sockets.c

  Line 719 in 74420be

  if (HANDLE_HDR(handle)->type != PAL_TYPE_UDP)

• gramine/Pal/src/host/Linux/db_sockets.c

  Line 750 in 74420be

  if (HANDLE_HDR(handle)->type != PAL_TYPE_UDPSRV)

@sudharkrish What happens if you simply remove these checks? Then your Go program may start working.

[sudharkrish]

@dimakuv per your suggestion, after allowing udp-server objects to invoke udp_receive which does NOT take address parameter, read works.

    if ((!IS_HANDLE_TYPE(handle, udp)) && (!IS_HANDLE_TYPE(handle, udpsrv)))
        return -PAL_ERROR_NOTCONNECTION;

In general, UDP-server code in C, passes address parameter passed along with read or write on that UDP socket handle, so Graphene's UDP-server code is able to handle it.
In contrast with Golang UDP-server code, we can get call to udp.read(without address parameter).

Question is, how to formalize these changes. Besides this additional use-case, where udp-server instance can do a read(without passing address parameter), not sure if there are other cases. For example, I am not sure, if it is feasible to do a write on udp-server instance, that is call to udp_send, without a address parameter.

[dimakuv]

@sudharkrish Could you show your resulting modifications to Gramine (via git diff is enough)? It will be helpful to see the midifications to know if these modifications is all that is needed, or for this feature to be bullet-proof, we need to add additional checks/logic.

[sudharkrish]

@dimakuv , it is just 2 lines of change, here is the diff below:

git diff Pal/src/host/Linux-SGX/db_sockets.c
diff --git a/Pal/src/host/Linux-SGX/db_sockets.c b/Pal/src/host/Linux-SGX/db_sockets.c
index 1e5cbe20..45294779 100644
--- a/Pal/src/host/Linux-SGX/db_sockets.c
+++ b/Pal/src/host/Linux-SGX/db_sockets.c
@@ -542,7 +542,7 @@ static int64_t udp_receive(PAL_HANDLE handle, uint64_t offset, uint64_t len, voi
     if (offset)
         return -PAL_ERROR_INVAL;
 
-    if (!IS_HANDLE_TYPE(handle, udp))
+    if ((!IS_HANDLE_TYPE(handle, udp)) && (!IS_HANDLE_TYPE(handle, udpsrv)))
         return -PAL_ERROR_NOTCONNECTION;
 
     if (handle->sock.fd == PAL_IDX_POISON)
@@ -926,6 +926,7 @@ struct handle_ops g_udp_ops = {
 struct handle_ops g_udpsrv_ops = {
     .getname        = &socket_getname,
     .open           = &udp_open,
+    .read           = &udp_receive,
     .readbyaddr     = &udp_receivebyaddr,
     .writebyaddr    = &udp_sendbyaddr,
     .delete         = &socket_delete,

[sudharkrish]

@dimakuv did some reading on this UDP topic. As per info in this blog-> (https://dadrian.io/blog/posts/udp-in-go/), atleast in Golang, it is not possible to do a udp_write(without a address parameter) for a non-connected socket. But udp_read is possible on a non-connected socket. This is the use-case this PR tries to resolve(that is udp_read(from a UDP-server) for a non-connected socket, in which case, no need to provide peer's address).
Reading the info in that blog, in C code, it is feasible to convert a non-connected socket into a connected socket, and then issue udp_send(without address parameter). Some UDP sample code in C, where UDP-client does NOT pass peer's address, since it uses connected socket->https://www.geeksforgeeks.org/udp-client-server-using-connect-c-implementation/. For UDP-client case, not sure, if Graphene has code to support all use-cases. For UDP-server case, not sure, if it is possible to do a write in C with a non-connected socket. If yes, then we would also need to add udp_send in the handler(g_udp_ops ) for udp-server use-case.

[dimakuv]

Thanks, @sudharkrish, this is very useful. I assigned this issue labels, we'll need to add such UDP functionality to Gramine. Probably not just yet because it needs more reading and testing, and we are currently busy with other things.

[boryspoplawski]

@sudharkrish please verify that #579 fixes the issue. It does for me.

[skris14]

@boryspoplawski , due to system upgrade to newer ubuntu, and other setup issues, not able to quickly test this. Since, it works for you, you can close this PR.