RFC: A collection of abstractions for network programming

[deanberris]

Background

The goal of the project is to provide some abstractions to use for network programming at large. These include defining vocabulary concepts which will allow us to build higher level abstractions on top of. Some of these will sound familiar, but will require some elaboration.

Some guidelines for the discussion:

• We will attempt to operate only on the abstractions. If these are "concepts" we will provide type erasure facilities which can be used in ABI boundaries.

• We will not constrain the APIs to just async, blocking, or streaming. Consider when an operation on an abstraction is defined that it may be any one of the aforementioned styles.

• We may consider implementing the abstractions directly, or wrapping a dependency. For example, we might decide that we don't want to depend on Asio or implementations of the networking TS and instead implement these lower-level libraries ourselves.

As a conversation starter, here's a list of the abstractions I've been thinking about and opening up the list for comments on what might be the minimum required set of abstractions for us to build, say an HTTP server and client? Or if someone wanted to build/implement an RPC system with a custom protocol?

Proposed Abstractions

Here's a non-complete list of abstractions:

• Connection -- an abstraction which allows for performing write(Connection, ...) and read(Connection, ...) operations.
• Buffer -- a backing store of data which may be segmented in nature (i.e. a list of blocks) or associated with an external resource (can be associated with a Connection) which provides iterators (in the STL sense) for the data associated with a Buffer.
• Address -- a data type which represents a location of a resource. There may be many kinds of addresses, which depend on the underlying network implementation -- this should be able to represent things like domain sockets, named pipes, IPv4 and IPv6 addresses, etc.

The abstractions themselves don't mean much on their own, but the operations on these abstractions define much of the requirements. We elucidate some of these below.

Connection

First, establishing a connection will be dependent on the type of the Connection which can be as diverse as it needs to be. Consider the many ways to establish an HTTPS connection with TLS behind an HTTP proxy -- we may need to first establish a connection to the proxy, then set up a tunnel to a remote resource, and then "upgrade" the connection to be a TLS-encrypted connection. Some protocols enable multiple "streams" in a conceptual connection (for HTTP/2 for instance there can be multiple streams associated with a single connection, similar to IRC with different channels through the same session) which all compose.

We want to ensure that all Connection types support at least the following operations:

• write(Connection, ...)
• read(Connection, ...)
• query(Connection, ...)
• configure(Connection, ...)

The results of these operations will be specific to the Connection, but typically these free functions will be implemented as a forwarding template to public member functions of the Connection type.

All of these operations can fail, and failures are not exceptional conditions -- therefore we will refrain from using exceptions to report course-of-normal-operation failure states. We will reserve the use of exceptions to reporting failures in constructing types which have no way to represent a valid Connection object with invalid internal state (one example is if the construction fails to acquire a critical resource).

Buffer

We can represent a lot of data for higher level protocols with the concept of a Buffer -- these objects provide a means of staging data that is not immediately written to a Connection. The main operations for a buffer are:

• write(Buffer, ...)
• read(Buffer, ...)

We may have more refined buffer types which allow for seeking, and connection-backed buffers which also supports flushing contents of the buffer to the underlying connection.

Address

This abstraction covers the gamut of addressing systems which are supported/defined by the Internet Protocol addressing system, POSIX domain socket addressing, and even non-IP/POSIX addressing. The only requirement for an Address is that it support a query(Address, ...) operation which is dependent on the domain for which a particular address is defined.

Implementation Details

Sometimes we will need to expose underlying implementation details. We'll only require that the implementation details be domain-specific, non-ABI stable, and in its own namespace. While we want to provide the abstractions for ease of use, we recognise that some users (including us!) will need to reach into the implementation details at a higher level for platform/domain-specific configuration.

We may, given experience over time, consolidate more implementation details into abstraction-level functionality/support. This will require a consultation with the community on the extent/acceptability of the change.

[glynos]

@deanberris , I'm looking at the "address" abstraction. Bear in mind that much of this already exists. It conforms well with the WhatWG URL spec and the tests are pretty comprehensive, covering all sorts of types of address, as well as supporting different encodings.

[glynos]

I might start by hacking on cpp-netlib and extracting the current "connection" and "buffer" concepts" and removing everything else. Then see what I can do by building stuff back up again.

[deanberris]

@glynos I rather we work on this in isolation, to avoid having to port code from one repository to another. While a URL is definitely useful as a type on its own I'm looking more at the overall idea of a conceptual "address" -- in networking there are a number of potential addresses (IP addresses, MAC addresses, bluetooth device IDs, a channel in IRC, etc.). I just want to make sure that all the code/contributions that make it into the repository are covered by the CLA.

[glynos]

@deanberris Understood. I'd be happy and willing to port the URI code to this project if/when necessary, including updating the license.

[AngleNet]

Is this still in progress? @deanberris