Berlin Institute of Technology

Marchstr. 23 Berlin Germany philipp@inet.tu-berlin.de

Berlin Institute of Technology

Marchstr. 23 Berlin Germany theresa@inet.tu-berlin.de

Transport TAPS Working Group Internet-Draft This document describes a prototype implementation of Socket Intents for the BSD Socket API as an illustrative example how Socket Intents could be implemented. It described the experiences made with the prototype and lessons learned from trying to extend the BSD Socket API.

With the proliferation of devices that have multiple paths to the internet and an increasing number of transport protocols available, the number of transport options to serve a communication unit explodes. Implementing a heuristic or strategy for choosing from this overwhelming set of transport options by each application puts a huge burden on the application developer. Thus, the decisions regarding all transport options mentioned so far should be supported and, if requested by the application, automated within the transport layer. Socket Intents allow an application to express what it knows, assumes, expects or wants to prioritize regarding its own network communication. This information can than be used by the OS to perform destination selection, path selection and transport protocol stack instance selection. Our Socket Intents prototype for the BSD Socket API is a first attempt to automate transport option selection within the OS. It is primarily targeted at path and destination address selection and tries to be as close as possible to the semantics of the BSD Socket API. The prototype mostly excludes the problem of transport protocol stack instance selection, which is more closely discussed in . We implemented the prototype as a wrapper for the BSD Socket API that communicates to a central Multiple Access Manager that makes the actual decisions and can optimize across applications. The whole implementation was done in about 15k lines of C code. The code is available at Github under BSD License. This document describes our Socket Intents prototype for the BSD Socket API. It details important aspects of the implementation and the API variants we developed over time based on lessons learned. Finally, it summarizes these lessons and points out why the BSD Socket API is not particularly well suited to integrate automated transport protocol stack instance selection. Furthermore, it describes the limitations for destination address and path selection within the BSD Socket API.

The Socket Intents prototype consists of the following components, also shown in : The Socket Intents API, a BSD Socket API wrapper for applications to use, including a representation of the actual Socket Intents. The Socket Intents Library which implements the Socket Intents API. It sends requests to the Multiple Access Manager, e.g. before establishing a connection, and gets back a response regarding what interface to use. The Multiple Access Manager (MAM), a daemon which gets informed about all application requests and has knowledge of the available network interfaces. The Policy, a dynamically loaded library hosted by the MAM. It chooses which of the available interfaces to use based on the available knowledge about them and the Socket Intents. Data collectors that that reside inside the MAM and that provide information like bandwidth usage, smoothed RTT estimate and RSSI for wireless links to the policy.

| Manager | | Socket Intents | (MAM Response) | +---------------+ | | Library | <---------------- | | Policy | | +------------------------+ | +---------------+ | | BSD Sockets | | |Data Collectors| | +------------------------+ +-+---------------+-+ ]]>

The Multiple Access Manager (MAM) is the central transport option selection instance on a host. It is realized as a daemon that runs in userspace and receives requests from each application that uses the Socket Intents Library. The MAM hosts the Policy, which is the actual decision making component, e.g., deciding which source address and therefore which source interface to use. Upon events, such as an application requesting to resolve a name or to connect a socket (see for details), the Socket Intents Library issues a MAM request and the MAM invokes a callback to the policy – see for details - which can either communicate its decision right away or defer its decision, e.g., when it has to wait for the results of name resolution. The results and decisions are communicated back to the Socket Intents Library through the MAM response, where they are applied to the actual socket, see also . To support the policy, the MAM maintains a list of IP prefixes that are configured on the local interfaces and available for outgoing communications. As destination address selection and path selection are highly dependent on each other, the MAM integrates DNS resolution and maintains separate resolver configurations per prefix (see for further discussion on multiple PvDs and DNS resolution). Furthermore, the MAM includes data collectors which periodically gather statistics on the available paths, see for details.

In the Socket Intents prototype, the Policy to select among the available transport options is hosted by the MAM, see . We implement different interchangeable policies as dynamically loaded libraries. In our current implementation, only one policy can be active at a given time. When launching the MAM, the user has to choose a policy and supply a policy configuration, which can contain arbitrary data. Examples of policy configuration include: A list of IP prefixes configured on local interfaces to consider as source for the communication Name server(s) to use for each of the IP prefixes Preferences to instrument the policy The policy is initialized with this configuration and then waits for the callback of an incoming MAM request. Upon a callback, the policy can use information from the MAM request, such as Socket Intents, and information available within the MAM, such as recently measured path characteristics (see ), to make decisions. Policy decisions can include: The source address(es) used for name resolution How to order the results of name resolution (i.e., preferring certain IP addresses over others) Picking an IP protocol version Picking a transport protocol Setting socket options (e.g., disable TCP Nagle) Choosing a source address for the outgoing communication Reusing a socket from a given socket set (only for the API variant described in ) Note that in our current implementation, the policy is a piece of code which can in principle execute arbitrary instructions. We assume this is acceptable for an experimental platform but would prefer an abstract description like a domain-specific language for a production system.

The data collectors are implemented as a component of the MAM, within a callback that is executed periodically, e.g., every 100 ms. When this callback is invoked, the MAM passively gathers statistics about the current usage and properties of the available local interfaces and stores them in per-interface or per-network prefix data structures. Measured properties include: Minimum Smoothed Round Trip Time (SRTT) of current TCP connections using a network prefix, as an estimate for last-mile latency Transmitted and received bytes per second over an interface within the last callback period, as an estimate for current utilization Smoothed transmitted and received bytes per second over an interface, as an estimate for recent utilization Maximum transmitted and received bytes per second over an interface within the last 5 minutes, as an estimate for maximum available bandwidth On 802.11 interfaces, the Received Signal Strength Indicator (RSSI) of the last received frame on that interface, as an estimate for reception strength On 802.11 interfaces, the modulation rate of the last received and the last transmitted unicast data frame on that interface, as an estimate for the available data transmission rate on the first hop When a policy callback is invoked, the policy can use the latest measured properties to guide its decisions, see . Note that we do not perform active measurements from within the MAM to avoid overhead.

As described in , Socket Intents are pieces of information about upcoming traffic. An application can share the information that it has available through the Socket Intents API. In our implementation, Socket Intents are represented as socket options for get/setsockopt on its own socket option level (SOL_INTENTS). For some of the API variants, we had to introduce socket option lists, i.e., data structures that can hold multiple socket options and therefore multiple Socket Intents. Which of these variants is actually used depends on the API variant, see .

The Socket Intents API is a wrapper around the BSD Socket API. It sends requests to the Multiple Access Manager (MAM) at certain events, e.g., before a connection is established, and applies the suggestions that it gets from the MAM, e.g., to bind to a certain local interface or to set a certain socket option. There exist different variants of this API, see , that try to fit different concepts: The Classic API with muacc_context, see , was attempting to stick as close as possible to the call sequence of BSD Sockets. The second variant of the classic API does all transport option selection in getaddrinfo, see . This variant tries to simplify the implementation without deriving too much from the usage of BSD Sockets. It minimizes the changes to the BSD Socket API, but adds additional overhead to the application. The socketconnect API, see , tries to automate as much functionality as possible and adds support for automating connection caching. It replaces the usual sequence of BSD Socket API calls with a single call.

In the first variant, we add a parameter called muacc_context to the BSD Socket API calls and to getaddrinfo. This parameter holds properties provided by the socket calls and retains them across function calls to enable automation of the connection properties by our Socket Intents Prototype. The shadow data structures behind the muacc_context parameter are initialized by API wrapper at the time of the first call (which we assume to be muacc_getaddrinfo most of the time) with most of its fields empty. Then within each call to our modified Socket API, it is filled with data. Properties include: Socket file descriptor API calls that were already performed on this context domain, type, and protocol of the socket remote hostname remote address hints for resolving the remote address local address to bind to that the application requested local address to bind to that the MAM suggested current socket options that were set socket options suggested by MAM

This function resolves a host name or service to an addrinfo data structure, usually containing an IP address or port. Internally, the Socket Intents prototype sends a getaddrinfo request to the MAM, which should do the name resolution. It can, e.g., resolve the name over multiple available interfaces at the same time, and then order the results according to a policy decision, or only return results obtained over a specific interface. SIGNATURE: int muacc_getaddrinfo(muacc_context_t *ctx, const char *hostname, const char *servname, const struct addrinfo *hints, struct addrinfo **res) ARGUMENTS: Context that can contain properties of this socket/connection and retains them across function calls. This function is mostly called with an empty context, which is then filled within the function. Remote host name to be resolved Remote service to be resolved Hints for resolving the name Data structure for result of name resolution RETURN VALUE: Returns 0 on success, or an error code as provided by getaddrinfo().

This function creates a socket file descriptor just like the regular socket call. SIGNATURE: int muacc_socket(muacc_context_t *ctx, int domain, int type, int protocol) ARGUMENTS: Context that can contain properties of this socket/connection and retains them across function calls. This function is mostly called after muacc_getaddrinfo(), since domain, type, and protocol can depend on the type of resolved address. Domain of the socket Type of the socket Protocol of the socket RETURN VALUE: Returns a file descriptor of the new socket on success, or -1 on failure.

This call allows to set socket options (including Socket Intents). For Socket Intents, this function can be called on a valid muacc_context and an invalided file descriptor (-1) to provide assertional hints to muacc_getaddrinfo(). SIGNATURE: int muacc_setsockopt(muacc_context_t *ctx, int socket, int level, int option_name, const void *option_value, socklen_t option_len) ARGUMENTS: Context that can contain properties of this socket/connection and retains them across function calls. This function is mostly called to set Intents as socket options within the context. Socket file descriptor Level of the socket option to set Name of the socket option to set Value of the socket option to set Length of the socket option to set RETURN VALUE: Returns 0 on success, or -1 on failure.

Like the regular connect call, but also binds to the source address selected by the Socket Intents Policy and applies socket options suggested by the Socket Intents Policy. SIGNATURE: int muacc_connect(muacc_context_t *ctx, int socket, const struct sockaddr *address, socklen_t address_len) ARGUMENTS: Context that can contain properties of this socket/connection and retains them across function calls. This function is mostly called after all Socket Intents for this connection have been set via muacc_setsockopt(). Socket file descriptor Remote address to connect to Length of the remote address RETURN VALUE: Returns 0 on success, or -1 on failure.

Like regular close, but also cleans up state held in shadow structures behind muacc_context SIGNATURE: int muacc_close(muacc_context_t *ctx, int socket) ARGUMENTS: Context that can contain properties of this socket/connection and retains them across function calls. This function deinitializes and releases the context. Socket file descriptor RETURN VALUE: Returns 0 on success, or -1 on failure.

In this variant, Socket Intents are passed directly to getaddrinfo() as part of the hints parameter. The name resolution is done by the MAM, which makes all decisions and stores them in the “result” data structure as list of options ordered by preference. Subsequently, applications can use this information for calls to the unmodified BSD Socket API or other APIs. We provide helpers to apply all socket options from the “result” data structure. All relevant infos are stored in our addrinfo struct (see ) SIGNATURE: int muacc_ai_getaddrinfo(const char * hostname, const char * service, const struct muacc_addrinfo * hints, struct muacc_addrinfo ** result) ARGUMENTS: Remote host name to be resolved Remote service to be resolved Hints for resolving the name. Contents include family, socket type, protocol, socket options (including Socket Intents for this socket/connection), local address to bind to. Data structure for result of name resolution RETURN VALUE: Returns 0 on success, or an error code as provided by getaddrinfo().

shows an example usage of the classic API with most functionality in getaddrinfo.

In this API variant, we move the functionality of resolving a hostname and connecting to the resulting address into one function called socketconnect(). This API makes it possible to call socketconnect not only for each connection, but also to multiplex messages across multiple existing sockets. This function returns a file descriptor of a connected socket for the application to use. This socket can either be a newly created one or a socket that existed previously and is now being reused. Furthermore, a socket can belong to a socket set of sockets with common destination and service. These sockets may, e.g., be bound to different local addresses, but are treated as interchangeable by the API implementation. So if the application passes a socket file descriptor to this function, it may get back a different file descriptor to a socket from the same set, e.g., to use the connection over a different local interface for its following communication. SIGNATURE: int socketconnect(int *socket, const char *host, size_t hostlen, const char *serv, size_t servlen, struct socketopt *sockopts, int domain, int type, int proto) ARGUMENTS: Existing socket file descriptor as representant to a socket set, “-1” to create a new socket, or “0” to automatically try to find a suitable socket set Remote hostname to be resolved Length of remote hostname Remote service or port Length of remote service List of socket options, including Socket Intents Domain of the socket Type of the socket Protocol of the socket RETURN VALUE: Returns 0 on success if socket is from an existing socket set, 1 on success if socket was newly created, or -1 on fail. shows an example usage of the Socketconnect API.

While designing and implementing the different parts of the system as described in this document, we faced several challenges. In the Multiple Access Manager discovering the currently available paths and statistics about their performance turned out to be quite complex and had to be implemented in a partially platform-dependent way. However, the most challenging parts were the Socket Intents API and Library, on which we focus in the following sections.

Transport option selection is most useful if crucial information, such as Socket Intents or other socket options, is available as early as possible, i.e., for name resolution. The primary problem here is the order of the function calls that are involved in name resolution, destination selection, protocol, and path selection, and how they are linked. In the classic BSD Socket API, most functions either take a socket file descriptor as argument or return it, and thus link different function calls to the same flow. However, getaddrinfo() is not linked to a socket file descriptor, and it is typically called before the socket is created. At this point, it is not yet possible to set a socket option, because the socket does not exist yet. Consequently, across BSD Socket API calls, several choices are being made before it is possible to set a Socket Intent: A call to getaddrinfo() returns a linked list of addrinfo structs, where each entry contains an ai_family (IP version), the pair of ai_socktype and ai_protocol (transport protocol), and a sockaddr struct containing an IP address and port to connect to. Then a socket of the given family, type, and protocol is created. Only after this has been done, socket options can be set on the socket, but at this point destination, IP version, and transport protocol are already fixed. Before calling connect(), only the path to be used (i.e., the local address to bind to) can still be chosen, but the available paths and which one to prefer may be constrained by the choice of destination. The three variants described in work around this problem in different ways: The approach in places the whole automation of transport option selection into the getaddrinfo() function. The results are returned in an extended addrinfo struct and have to be applied manually by the application, including binding to a source address representing the selected path and applying all socket options provided in a list, for each connection attempt. The approach in adds a context to all socket- and name resolution-related API calls. The approach in puts all functionality into one call. All of these approaches add the missing link between name resolution and the other parts of the API, but add a lot of state keeping either to the API, which the application developer has to manage, or to the Socket Intents library.

When using BSD sockets, file descriptors are the abstraction for network flows. Depending on the transport protocol used, their semantics changes and these file handles represent streams (SOCK_STREAM), associations (SOCK_DRAM) or network interfaces (SOCK_RAW). This does not provide a unified API, but is merely an artifact of squeezing networking into the “Everything is a file” UNIX philosophy. File descriptors make no good abstraction for automated protocol stack instance selection as applications have to adopt to changed semantics, e.g., whether message boundaries are preserved, depending on the transport protocol chosen. File descriptors make no good abstraction for destination instance selection and path selection either. Once a socket has been created, its protocol stack instance is fixed, so selecting a path by binding to a local address and connecting to a destination instance is now only possible using this protocol stack instance. If such a connection attempt fails, it is possible to retry using another path and destination, but changing the protocol stack instance requires creating a new socket with a different file descriptor. For further discussion of other asynchronous I/O weirdness with file descriptors see end of .

Network I/O is asynchronous, but asynchronous I/O within the POSIX filesystem API is hard to use. There are at least three different asynchronous I/O APIs for each operating system. To implement asynchronous I/O for our Socket Intents prototype, we wrapped one of the asynchronous I/O APIs that is available on most platforms: select(). To make Socket Intents accessible to more applications and on more platforms, a production-grade system would need to wrap all asynchronous I/O APIs and implement most of the socket creation logic, path selection and connection logic within these wrappers. However, mixing asynchronous I/O and multithreading may lead to unintuitive behavior, e.g., calling our prototype’s select() from different threads could lead to anything from deadlocks to busy waiting. Another issue is that we use Unix domain sockets to communicate between our Multiple Access Manager and the Socket Intents API library called by the application, so we need to make sure that the application does not block on communication with the Multiple Access Manager. Also the problems with using file descriptors get even worse. If a Socket API call should return immediately, it needs to provide the application with a reference to a flow that has not yet been fully set up, i.e., a reference to a “future” socket. An implementation of such an asynchronous API has to return an unconnected socket file descriptor, on which the application then calls, e.g., select(), and starts using it once it becomes readable and writable. If the destination, path and transport protocol have not been chosen yet at this point, the file descriptor returned by the implementation might not yet have the final family and transport protocol. When the implementation later creates the final socket of the right type, it can re-bind it to the file-id of the originally returned file descriptor using dup2. This procedure can easily lead to time-of-check to time-of-use confusion. To make things even worse, the application can copy the “future” file descriptor using dup, which is rarely useful for sockets, but in combination with file descriptors used as “future” it leads to unexpected behavior.

The API variants described in and need to keep a lot of state in shadow structures that cannot be passed between the Socket API calls otherwise. This state needs to be cleaned up when the last copy of the file descriptor is closed or the last socket held for reuse has timed out. In addition, access to these shadow structures has to be thread-safe. Implementing both has turned out to be extremely error-prone and there is a high amount of unspecified behavior and platform-dependent extensions in the system library. These issues guarantee that an implementation of transport option selection that nicely integrates with BSD Sockets will come with lots of limitations and will not be portable across POSIX-compliant operating systems.

Adding transport option selection to BSD Sockets is hard, as the API calls are not designed to defer making and applying choices to a moment where all information needed for transport option selection is available. After all, if limiting transport option selection to the granularity BSD Sockets typically provide today (TCP connections and UDP associations), the API variant described in seems to be a good compromise, even if it forces the application to try all candidates itself (either in a sequential or partial parallel fashion). This option is easily deployable, but does not include automation of techniques like connection caching or HTTP pipelining. The most versatile API variant described in implements connection caching on the transport layer. This comes at the cost of heavily modifying existing applications. If feasible, given the unnecessary complexity of the file I/O integration of BSD sockets, it seems easier to move to a totally different system like .

Thanks to Tobias Kaiser mail@tb-kaiser.de for drafting and implementing the API variant described in as part of his BA thesis.

In many standards track documents several words are used to signify the requirements in the specification. These words are often capitalized. This document defines these words as they should be interpreted in IETF documents. This document specifies an Internet Best Current Practices for the Internet Community, and requests discussion and suggestions for improvements. TCP/IP communication is currently restricted to a single path per connection, yet multiple paths often exist between peers. The simultaneous use of these multiple paths for a TCP/IP session would improve resource usage within the network and, thus, improve user experience through higher throughput and improved resilience to network failure.Multipath TCP provides the ability to simultaneously use multiple paths between peers. This document presents a set of extensions to traditional TCP to support multipath operation. The protocol offers the same type of service to applications as TCP (i.e., reliable bytestream), and it provides the components necessary to establish and use multiple TCP flows across potentially disjoint paths. This document defines an Experimental Protocol for the Internet community. This document describes an experimental TCP mechanism called TCP Fast Open (TFO). TFO allows data to be carried in the SYN and SYN-ACK packets and consumed by the receiving end during the initial connection handshake, and saves up to one full round-trip time (RTT) compared to the standard TCP, which requires a three-way handshake (3WHS) to complete before data can be exchanged. However, TFO deviates from the standard TCP semantics, since the data in the SYN could be replayed to an application in some rare circumstances. Applications should not use TFO unless they can tolerate this issue, as detailed in the Applicability section. This document is a product of the work of the Multiple Interfaces Architecture Design team. It outlines a solution framework for some of the issues experienced by nodes that can be attached to multiple networks simultaneously. The framework defines the concept of a Provisioning Domain (PvD), which is a consistent set of network configuration information. PvD-aware nodes learn PvD-specific information from the networks they are attached to and/or other sources. PvDs are used to enable separation and configuration consistency in the presence of multiple concurrent connections. This document outlines an API-independent concept that allows applications to share their knowledge about upcoming communication and express their performance preferences in a portable and abstract way: Socket Intents. Socket Intents express what an application knows, assumes, expects or wants to prioritize regarding its own network communication. The information provided by Socket Intents should be taken into account by the network stack in a best-effort way. Socket Intent can be used to stem against the complexity and make use of multiple provisioning domains as well as new transport protocols and features available to a larger user base by expressing the applications intents in an abstract and portable way. This document provides an abstract framework to reason about the composition of multi-path aware systems in a protocol-independent fashion. It discusses basic mechanisms that are used in multi-path systems and their applicability to different granularities of communication units. This document is targeted as consideration basis for automation of destination, path and transport protocol selection within the transport layer. This document describes Post Sockets, an asynchronous abstract programming interface for the atomic transmission of messages in an inherently multipath environment. Post replaces connections with long-lived associations between endpoints, with the possibility to cache cryptographic state in order to reduce amortized connection latency. We present this abstract interface as an illustration of what is possible with present developments in transport protocols when freed from the strictures of the current sockets API.

In this example, a client application sets up a connection to a remote host and sends data to it. It specifies two Socket Intents on this connection: The Category of Bulk Transfer and the File Size of 1 MB.

ai_family, result->ai_socktype, result->ai_protocol); // Connect the socket to the remote endpoint as determined by // getaddrinfo. This involves another request to MAM, which may at this // point, e.g., choose to bind the socket to a local IP address before // connecting it. muacc_connect(&ctx, socket, result->ai_addr, result->ai_addrlen); // Send data to the remote host over the socket. write(socket, &buf, LENGTH_OF_DATA); // Close the socket. This de-initializes any data that was stored within // the muacc_context. muacc_close(&ctx, socket); ]]>

As in , the application sets the Intents “Category” and “File Size”.

ai_family, result->ai_socktype, result->ai_protocol); muacc_ai_simple_connect(fd, result); // Send data to the remote host over the socket, then close it. write(fd, &buf, LENGTH_OF_DATA); close(fd); muacc_ai_freeaddrinfo(result); ]]>

As in , the application sets the Intents “Category” and “File Size”. As we provide -1 as socket, no we do not reuse existing connections.