Extract basic data structures to NIOCore. (#1895)

Motivation:

Currently the "core" NIO module (currently called NIO) contains two
fairly distinct kinds of things. The first is the core abstractions and
data structures: the things that all NIO programs must necessarily deal
with. This includes our base abstraction protocols (e.g. `EventLoop`,
`Channel`, and `ChannelHandler`), as well as our base data types (e.g.
`ByteBuffer`, `SocketAddress`, `EventLoopFuture`).

The second thing the core NIO module contains is the default POSIX
sockets event-loop and channels. These are the mainline production loops
for NIO programs, and are supported in the various BSD-like OSes to
varying extents. These can be thought of as a baseline implementation of
the core abstraction protocols that form the first part of NIO.

These two fairly distinct use-cases have no particular reason to be
glued together, and in fact in gluing them together we have done a bit
of a disservice to our adopters. The biggest issue is that it has become
impossible to write a NIO program or library that does not bring along
the POSIX layer of NIO. This has made porting NIO to other platforms
difficult (Windows has been painful, and Webassembly is coming down the
road as well), and has also bloated the size of iOS/macOS applications
that use NIO via NIOTransportServices, as all of these applications must
bring along an event loop and several Channels they will not use.

To that end, we plan to begin a scope of work to split the core NIO
module in two. Specifically, we intend to extract the baseline protocols
and data types into a new module, called `NIOCore`, which will be able
to be the baseline NIO module.

The end goal is that libraries that implement protocols on top of
SwiftNIO should be able to depend solely on `NIOCore`, not `NIO`. This
will allow NIO applications to bring along only the event loops they
want or need, without needing to pay for the POSIX sockets
implementation. This should also make porting easier, as there will be
no pressure to bring the existing `NIO` module to new platforms if it
does not fit well.

Modifications:

This is the first in a series of patches, which extracts some of the
low-hanging fruit. In particular, it extracts:

- `ByteBuffer` (and associated types)
- `RecvByteBufferAllocator`
- `CircularBuffer`

This also adds in the backwards compatibility shim: the `NIO` package
performs an exported import of `NIOCore`. This will allow this change to
avoid breaking API.

However, to make a clean break we intend not to release a new NIO minor
version until we have completed our extraction. If we fail to do this it
won't be the end of the world, we can release subsequent minor versions
that continue to extract new types. However, as much as possible should
go at once.

This patch also duplicates a few methods and files. At the end of the
patch series we need to reconcile this duplication. In many cases the
duplication will no longer be necessary, but in some cases we may have
necessary duplicates.

Result:

The first step along the road to better platform citizenship is taken.

Package.swift

@@ -16,11 +16,14 @@
 import PackageDescription
 
 var targets: [PackageDescription.Target] = [
+    .target(name: "NIOCore",
+            dependencies: ["NIOConcurrencyHelpers"]),
     .target(name: "NIO",
             dependencies: ["CNIOLinux",
                            "CNIODarwin",
                            "CNIOWindows",
-                           "NIOConcurrencyHelpers"]),
+                           "NIOConcurrencyHelpers",
+                           "NIOCore"]),
     .target(name: "_NIOConcurrency",
             dependencies: ["NIO"]),
     .target(name: "NIOFoundationCompat", dependencies: ["NIO"]),
@@ -86,6 +89,7 @@ var targets: [PackageDescription.Target] = [
 let package = Package(
     name: "swift-nio",
     products: [
+        .library(name: "NIOCore", targets: ["NIOCore"]),
         .library(name: "NIO", targets: ["NIO"]),
         .library(name: "_NIOConcurrency", targets: ["_NIOConcurrency"]),
         .library(name: "NIOTLS", targets: ["NIOTLS"]),

Sources/NIO/ExportCore.swift

@@ -0,0 +1,14 @@
+//===----------------------------------------------------------------------===//
+//
+// This source file is part of the SwiftNIO open source project
+//
+// Copyright (c) 2021 Apple Inc. and the SwiftNIO project authors
+// Licensed under Apache License v2.0
+//
+// See LICENSE.txt for license information
+// See CONTRIBUTORS.txt for the list of SwiftNIO project authors
+//
+// SPDX-License-Identifier: Apache-2.0
+//
+//===----------------------------------------------------------------------===//
+@_exported import NIOCore

Sources/NIOCore/ByteBuffer-core.swift

@@ -14,6 +14,10 @@
 
 #if os(Windows)
 import ucrt
+#elseif os(macOS) || os(iOS) || os(tvOS) || os(watchOS)
+import Darwin
+#else
+import Glibc
 #endif
 
 let sysMalloc: @convention(c) (size_t) -> UnsafeMutableRawPointer? = malloc

Sources/NIOCore/IntegerBitPacking.swift

@@ -0,0 +1,103 @@
+//===----------------------------------------------------------------------===//
+//
+// This source file is part of the SwiftNIO open source project
+//
+// Copyright (c) 2021 Apple Inc. and the SwiftNIO project authors
+// Licensed under Apache License v2.0
+//
+// See LICENSE.txt for license information
+// See CONTRIBUTORS.txt for the list of SwiftNIO project authors
+//
+// SPDX-License-Identifier: Apache-2.0
+//
+//===----------------------------------------------------------------------===//
+
+// FIXME: Duplicated in NIO.
+
+@usableFromInline
+enum _IntegerBitPacking {}
+
+extension _IntegerBitPacking {
+    @inlinable
+    static func packUU<Left: FixedWidthInteger & UnsignedInteger,
+                       Right: FixedWidthInteger & UnsignedInteger,
+                       Result: FixedWidthInteger & UnsignedInteger>(_ left: Left,
+                                                                    _ right: Right,
+                                                                    type: Result.Type = Result.self) -> Result {
+        assert(MemoryLayout<Left>.size + MemoryLayout<Right>.size <= MemoryLayout<Result>.size)
+
+        let resultLeft = Result(left)
+        let resultRight = Result(right)
+        let result = (resultLeft << Right.bitWidth) | resultRight
+        assert(result.nonzeroBitCount == left.nonzeroBitCount + right.nonzeroBitCount)
+        return result
+    }
+
+    @inlinable
+    static func unpackUU<Input: FixedWidthInteger & UnsignedInteger,
+                         Left: FixedWidthInteger & UnsignedInteger,
+                         Right: FixedWidthInteger & UnsignedInteger>(_ input: Input,
+                                                                     leftType: Left.Type = Left.self,
+                                                                     rightType: Right.Type = Right.self) -> (Left, Right) {
+        assert(MemoryLayout<Left>.size + MemoryLayout<Right>.size <= MemoryLayout<Input>.size)
+
+        let leftMask = Input(Left.max)
+        let rightMask = Input(Right.max)
+        let right = input & rightMask
+        let left = (input >> Right.bitWidth) & leftMask
+
+        assert(input.nonzeroBitCount == left.nonzeroBitCount + right.nonzeroBitCount)
+        return (Left(left), Right(right))
+    }
+}
+
+@usableFromInline
+enum IntegerBitPacking {}
+
+extension IntegerBitPacking {
+    @inlinable
+    static func packUInt32UInt16UInt8(_ left: UInt32, _ middle: UInt16, _ right: UInt8) -> UInt64 {
+        return _IntegerBitPacking.packUU(
+            _IntegerBitPacking.packUU(right, middle, type: UInt32.self),
+            left
+        )
+    }
+
+    @inlinable
+    static func unpackUInt32UInt16UInt8(_ value: UInt64) -> (UInt32, UInt16, UInt8) {
+        let leftRight = _IntegerBitPacking.unpackUU(value, leftType: UInt32.self, rightType: UInt32.self)
+        let left = _IntegerBitPacking.unpackUU(leftRight.0, leftType: UInt8.self, rightType: UInt16.self)
+        return (leftRight.1, left.1, left.0)
+    }
+
+    @inlinable
+    static func packUInt8UInt8(_ left: UInt8, _ right: UInt8) -> UInt16 {
+        return _IntegerBitPacking.packUU(left, right)
+    }
+
+    @inlinable
+    static func unpackUInt8UInt8(_ value: UInt16) -> (UInt8, UInt8) {
+        return _IntegerBitPacking.unpackUU(value)
+    }
+
+    @inlinable
+    static func packUInt16UInt8(_ left: UInt16, _ right: UInt8) -> UInt32 {
+        return _IntegerBitPacking.packUU(left, right)
+    }
+
+    @inlinable
+    static func unpackUInt16UInt8(_ value: UInt32) -> (UInt16, UInt8) {
+        return _IntegerBitPacking.unpackUU(value)
+    }
+
+    @inlinable
+    static func packUInt32CInt(_ left: UInt32, _ right: CInt) -> UInt64 {
+        return _IntegerBitPacking.packUU(left, UInt32(truncatingIfNeeded: right))
+    }
+
+    @inlinable
+    static func unpackUInt32CInt(_ value: UInt64) -> (UInt32, CInt) {
+        let unpacked = _IntegerBitPacking.unpackUU(value, leftType: UInt32.self, rightType: UInt32.self)
+        return (unpacked.0, CInt(truncatingIfNeeded: unpacked.1))
+    }
+}

Sources/NIOCore/IntegerTypes.swift

@@ -0,0 +1,116 @@
+//===----------------------------------------------------------------------===//
+//
+// This source file is part of the SwiftNIO open source project
+//
+// Copyright (c) 2017-2018 Apple Inc. and the SwiftNIO project authors
+// Licensed under Apache License v2.0
+//
+// See LICENSE.txt for license information
+// See CONTRIBUTORS.txt for the list of SwiftNIO project authors
+//
+// SPDX-License-Identifier: Apache-2.0
+//
+//===----------------------------------------------------------------------===//
+
+// MARK: _UInt24
+
+// FIXME: Duplicated in NIO.
+
+/// A 24-bit unsigned integer value type.
+@usableFromInline
+struct _UInt24 {
+    @usableFromInline var _backing: (UInt16, UInt8)
+
+    @inlinable
+    init(_ value: UInt32) {
+        assert(value & 0xff_00_00_00 == 0, "value \(value) too large for _UInt24")
+        self._backing = IntegerBitPacking.unpackUInt16UInt8(value)
+    }
+
+    static let bitWidth: Int = 24
+
+    @usableFromInline
+    static let max: _UInt24 = .init((UInt32(1) << 24) - 1)
+
+    @usableFromInline
+    static let min: _UInt24 = .init(0)
+}
+
+extension UInt32 {
+    @inlinable
+    init(_ value: _UInt24) {
+        self = IntegerBitPacking.packUInt16UInt8(value._backing.0, value._backing.1)
+    }
+}
+
+extension Int {
+    @inlinable
+    init(_ value: _UInt24) {
+        self = Int(UInt32(value))
+    }
+}
+
+
+extension _UInt24: Equatable {
+    @inlinable
+    public static func ==(lhs: _UInt24, rhs: _UInt24) -> Bool {
+        return lhs._backing == rhs._backing
+    }
+}
+
+extension _UInt24: CustomStringConvertible {
+    @usableFromInline
+    var description: String {
+        return UInt32(self).description
+    }
+}
+
+// MARK: _UInt56
+
+/// A 56-bit unsigned integer value type.
+struct _UInt56 {
+    @usableFromInline var _backing: (UInt32, UInt16, UInt8)
+
+    @inlinable init(_ value: UInt64) {
+        self._backing = IntegerBitPacking.unpackUInt32UInt16UInt8(value)
+    }
+
+    static let bitWidth: Int = 56
+
+    private static let initializeUInt64 : UInt64 = (1 << 56) - 1
+    static let max: _UInt56 = .init(initializeUInt64)
+    static let min: _UInt56 = .init(0)
+}
+
+extension _UInt56 {
+    init(_ value: Int) {
+        self.init(UInt64(value))
+    }
+}
+
+extension UInt64 {
+    init(_ value: _UInt56) {
+        self = IntegerBitPacking.packUInt32UInt16UInt8(value._backing.0,
+                                                       value._backing.1,
+                                                       value._backing.2)
+    }
+}
+
+extension Int {
+    init(_ value: _UInt56) {
+        self = Int(UInt64(value))
+    }
+}
+
+extension _UInt56: Equatable {
+    @inlinable
+    public static func ==(lhs: _UInt56, rhs: _UInt56) -> Bool {
+        return lhs._backing == rhs._backing
+    }
+}
+
+extension _UInt56: CustomStringConvertible {
+    var description: String {
+        return UInt64(self).description
+    }
+}

Sources/NIOCore/PointerHelpers.swift

@@ -0,0 +1,43 @@
+//===----------------------------------------------------------------------===//
+//
+// This source file is part of the SwiftNIO open source project
+//
+// Copyright (c) 2017-2018 Apple Inc. and the SwiftNIO project authors
+// Licensed under Apache License v2.0
+//
+// See LICENSE.txt for license information
+// See CONTRIBUTORS.txt for the list of SwiftNIO project authors
+//
+// SPDX-License-Identifier: Apache-2.0
+//
+//===----------------------------------------------------------------------===//
+
+// MARK: Rebasing shims
+
+// FIXME: Duplicated in NIO.
+
+// These methods are shimmed in to NIO until https://github.com/apple/swift/pull/34879 is resolved.
+// They address the fact that the current rebasing initializers are surprisingly expensive and do excessive
+// checked arithmetic. This expense forces them to often be outlined, reducing the ability to optimise out
+// further preconditions and branches.
+extension UnsafeRawBufferPointer {
+    @inlinable
+    init(fastRebase slice: Slice<UnsafeRawBufferPointer>) {
+        let base = slice.base.baseAddress?.advanced(by: slice.startIndex)
+        self.init(start: base, count: slice.endIndex &- slice.startIndex)
+    }
+
+    @inlinable
+    init(fastRebase slice: Slice<UnsafeMutableRawBufferPointer>) {
+        let base = slice.base.baseAddress?.advanced(by: slice.startIndex)
+        self.init(start: base, count: slice.endIndex &- slice.startIndex)
+    }
+}
+
+extension UnsafeMutableRawBufferPointer {
+    @inlinable
+    init(fastRebase slice: Slice<UnsafeMutableRawBufferPointer>) {
+        let base = slice.base.baseAddress?.advanced(by: slice.startIndex)
+        self.init(start: base, count: slice.endIndex &- slice.startIndex)
+    }
+}

Sources/NIOCore/Utilities.swift

@@ -0,0 +1,24 @@
+//===----------------------------------------------------------------------===//
+//
+// This source file is part of the SwiftNIO open source project
+//
+// Copyright (c) 2021 Apple Inc. and the SwiftNIO project authors
+// Licensed under Apache License v2.0
+//
+// See LICENSE.txt for license information
+// See CONTRIBUTORS.txt for the list of SwiftNIO project authors
+//
+// SPDX-License-Identifier: Apache-2.0
+//
+//===----------------------------------------------------------------------===//
+
+/// A utility function that runs the body code only in debug builds, without
+/// emitting compiler warnings.
+///
+/// This is currently the only way to do this in Swift: see
+/// https://forums.swift.org/t/support-debug-only-code/11037 for a discussion.
+@inlinable
+internal func debugOnly(_ body: () -> Void) {
+    // FIXME: duplicated with NIO.
+    assert({ body(); return true }())
+}

Tests/NIOTests/ByteBufferTest.swift

@@ -14,7 +14,7 @@
 
 import struct Foundation.Data
 import XCTest
-@testable import NIO
+@testable import NIOCore
 import NIOFoundationCompat
 
 class ByteBufferTest: XCTestCase {

Tests/NIOTests/ChannelTests.swift

@@ -13,6 +13,7 @@
 //===----------------------------------------------------------------------===//
 
 import XCTest
+@testable import NIOCore
 @testable import NIO
 import NIOConcurrencyHelpers
 import NIOTestUtils

Tests/NIOTests/CircularBufferTests.swift

@@ -13,7 +13,7 @@
 //===----------------------------------------------------------------------===//
 
 import XCTest
-@testable import NIO
+@testable import NIOCore
 
 class CircularBufferTests: XCTestCase {
     func testTrivial() {

Tests/NIOTests/CodecTest.swift

@@ -12,8 +12,8 @@
 //
 //===----------------------------------------------------------------------===//
 
-//
 import XCTest
+@testable import NIOCore
 @testable import NIO
 
 private var testDecoderIsNotQuadratic_mallocs = 0

Tests/NIOTests/IntegerTypesTest.swift

@@ -13,7 +13,7 @@
 //===----------------------------------------------------------------------===//
 
 import XCTest
-@testable import NIO
+@testable import NIOCore
 
 public final class IntegerTypesTest: XCTestCase {
     func testNextPowerOfOfTwoZero() {

Tests/NIOTests/PendingDatagramWritesManagerTests.swift

@@ -12,6 +12,7 @@
 //
 //===----------------------------------------------------------------------===//
 
+@testable import NIOCore
 @testable import NIO
 import XCTest