Optimal Use Of The TCP/IP Stack in User-space Storage Applications with ADQ feature in NIC

Optimal Use Of The TCP/IP Stack in User-space Storage Applications with ADQ feature in NIC

연구 분야: Networking

학회: 2020 IEEE 26th International Conference on Parallel and Distributed Systems (ICPADS)

For storage applications based on TCP/IP, performance of the TCP/IP stack is often the dominant driver of the application's overall performance. In this paper, we introduce Tbooster, which is a library designed for TCP/IP based storage applications that optimally leverages the Linux kernel's TCP/IP stack and the socket interface to improve performance. This library allows for efficient grouping of connections onto threads and asynchronous, poll-mode operation, scaling to a massive number of TCP connections on each thread. Tbooster is designed to avoid making expensive system calls by batching and merging operations into a single operation per connection for each poll loop. Further, Tbooster leverages the Application Device Queues (ADQ) feature available in some Network Interface Cards to steer incoming data to the correct NIC receive queue. This avoids expensive coordination and message passing within the kernel when handling incoming data and especially improves outlier tail latencies of requests. Compared with a more standard usage of the Linux kernel TCP/IP stack, Tbooster improves storage I/O per second significantly (e.g., 9% to 22.7% IOPS improvement for an iSCSI target at 4KiB I/O size, 36.3% to 59.4% IOPS improvement for NVMe-oF TCP target on 8KiB I/O size). Moreover, when using the ADQ feature from Intel's 100GbE NIC, it demonstrates 30% average time reduction of 99.99% long tail latency under heavy workloads for NVMe-oF TCP.