Motivation:
Channels can read `ChannelOptions.maxMessagesPerRead` times from a
socket in each read cycle. They typically re-use the same buffer for
each read and rely on it CoWing if necessary. If we read more than once
in a cycle then we may CoW the buffer. Instead of reusing one buffer we
can reuse a pool of buffers limited by `maxMessagesPerRead` and cycle
through each, reducing the chance of CoWing the buffers.
Modifications:
- Extend `RecvByteBufferAllocator` to provide the size of the next
buffer with a default implementation returning `nil`.
- Add an recv buffer pool which lazily grows up to a fixed size and
attempts to reuse buffers where possible if doing so avoids CoWing.
Results:
Fewer allocations
Motivation:
UUIDs are often sent over the wire but writing and reading their bytes
to/from a buffer is a bit of a pain.
Modifications:
- Add utilties to 'NIOFoundationCompat' for reading/writing and
getting/setting a UUID on a `ByteBuffer`.
Result:
Easier to write/read UUIDs to/from a buffer.
Co-authored-by: Cory Benfield <lukasa@apple.com>
Motivation:
As 5.7 has shipped we no longer need to keep these APIs in _NIOBeta.
We're going to do a two-stage removal: first we're going to move the
APIs to NIOCore and keep them in _NIOBeta with deprecations on them. In
a later release, we'll remove the APIs from _NIOBeta entirely.
Modifications:
- Move the TimeAmount + Duration APIs to NIOCore
- Deprecate the APIs in _NIOBeta.
Result:
We're on a path to remove _NIOBeta
Motivation:
SwiftNIO periodically drops support for older Swift versions. Now that
5.7 has been released, 5.4 will be dropped.
Modifications:
- Remove 5.4 specific Package.swift and docker-compose
- Update the 5.7 docker-compose to use the released 5.7 and move from
focal (2004) to jammy (2204)
- Remove unused swiftformat from Dockerfile
- Update tools version in syscall wrapper tests to 5.5
- Update docs
Results:
Minimum Swift version is 5.5
Motivation
HTTP headers are prevented from containing certain characters that can
potentially affect parsing or interpretation. Inadequately policing this
can lead to vulnerabilities in web applications, most notably HTTP
Response Splitting.
NIO was insufficiently policing the correctness of the header fields we
emit in HTTP/1.1. We've therefore added a new handler that is
automatically added to channel pipelines that will police the validity
of header fields.
For projects that are already running the validation themselves, this
can be easily disabled. Note that by default NIO does not validate
content length is correctly calculated, so applications can have their
framing fall out of sync unless they appropriately calculate this
themselves or use chunked transfer encoding.
Modifications
- Add thorough unit testing to confirm we will not emit invalid header
fields.
- Error if a user attempts to send an invalid header field.
Result
NIO applications are no longer vulnerable to response splitting by CRLF
injection by default.
Motivation
Testing versions of NIO code that involve interfacing with Swift
Concurrency is currently a difficult business. In particular,
EmbeddedChannel is not available in Swift concurrency, making it
difficult to write tests where you fully control the I/O.
To that end, we should provide a variation of EmbeddedChannel that makes
testing these things possible.
Modifications
Provide an implementation of NIOAsyncTestingChannel.
Results
Users can write tests confidently with async/await.
* Enhance and rename AsyncEmbeddedEventLoop
Motivation
AsyncEmbeddedEventLoop is an important part of our ongoing testing story
for NIO. However, it suffers from two problems.
The first is a usability one. As we discovered during the original
implementation, following EmbeddedEventLoop's pattern of not having any
EventLoop "Tasks" execute until run() meant that simple constructs like
`EventLoopFuture.wait` and `EventLoopFuture.get` didn't work at all,
forcing us to add an annoying `awaitFuture` method.
When playing with implementing EmbeddedChannel, this got worse, as I/O
methods also don't run immediately if called from the testing thread. We
couldn't easily work around this issue, and it meant that common
patterns (like calling `channel.writeAndFlush`) would deadlock!
This is unacceptable, so a change had to be made.
While we're here, we received feedback that the name is unclear to
users. Given that this particular event loop is in no sense "embedded",
we no longer need the name, so we can take this opportunity to use a
better one.
Modifications
Changed `func execute` to immediately execute its task body, and to
dequeue all pending tasks at this time. Essentially, it's the equivalent
to run(). This is a major change in its behaviour.
Renamed the loop to `NIOAsyncTestingEventLoop`.
Result
Better names, easier to use.
* Make soundness happy
* Remove awaitFuture
* Provide conversion APIs between TimeAmount and Swift.Duration
Signed-off-by: Si Beaumont <beaumont@apple.com>
* fixup: Remove unnecessary overflow check
Signed-off-by: Si Beaumont <beaumont@apple.com>
* fixup: Move into new _NIOBeta57 module
Signed-off-by: Si Beaumont <beaumont@apple.com>
* fixup: Rename _NIOBeta57 module to _NIOBeta and move tests to own target
Signed-off-by: Si Beaumont <beaumont@apple.com>
* soundness: copyright years for some reason
Signed-off-by: Si Beaumont <beaumont@apple.com>
Motivation
The rise of Swift concurrency has meant that a number of our APIs need
to be recontextualised as async/await capable. While generally this is a
straightforward task, any time those APIs were tested using
EmbeddedChannel we have a testing issue. Swift Concurrency requires the
use of its own cooperative thread pool, which is completely incapable of
safely interoperating with EmbeddedChannel and EmbeddedEventLoop. This
is becuase those two types "embed" into the current thread and are not
thread-safe, but our concurrency-focused APIs want to enable users to
use them from any Task.
To that end we need to develop new types that serve the needs of
EmbeddedChannel and EmbeddedEventLoop (control over I/O and task
scheduling) while remaining fully thread-safe. This is the first of a
series of patches that adds this functionality, starting with the
AsyncEmbeddedEventLoop.
Modifications
- Define AsyncEmbeddedEventLoop
Result
A required building block for AsyncEmbeddedChannel exists.
Co-authored-by: Franz Busch <privat@franz-busch.de>
### Motivation:
Sometimes, it is nice to check `AddressedEnvelope`s for equality or hash them. Especially, in tests this comes in handy.
### Modifications:
This PR, adds conditional conformances to `AddressedEnvelope` for `Equatable` and `Hashable` depending on its generic `DataType` .
I also added a new module `NIOCoreTests` which was an open todoleft from the creation of the `NIOCore` module. To not add more tests that we have to migrate in the future, I started to create the new tests in the new module right away. I also created issue https://github.com/apple/swift-nio/issues/2016, to keep track of our open task to move the other tests over.
### Result:
`AddressedEnvelope` is conditionally `Equatable` and `Hashable`
Motivation:
The remaining NIO code really conceptually belongs in a module called
NIOPosix, and NIOCore should really be called NIO. We can't really do
that last step, but we can prepare by pushing the bulk of the remaining
code into a module called NIOPosix.
Modifications:
- Move NIO to NIOPosix
- Make NIO an umbrella module.
Result:
NIOPosix exists.
Motivation:
EmbeddedChannel is an important testing tool, and we want to use it
without needing to bring along the POSIX layer. They are not tightly
coupled. However, it also doesn't belong naturally in NIOCore, so we
should probably put it in its own place.
Modifications:
- Moved EmbeddedChannel and EmbeddedEventLoop to NIOEmbedded.
- Moved the tests to NIOEmbeddedTests
- Duplicated some test helpers
Result:
Easy to use EmbeddedChannel without the POSIX layer.
Motivation:
I'd like to move EmbeddedChannel and friends out of the main NIO
repository and into their own module. Unfortunately, EmbeddedChannel
shares the PriorityQueue implementation we wrote with the various POSIX
channels. To avoid duplicating the code, we should pull it out to its
own module.
However, we've never wanted to commit to the API of this data structure,
and the same is true now. To that end, I'm pulling it into an
underscored module that is not a product of the package.
We could have used the `@_spi` annotation here but honestly I'm a bit
nervous about doing that at the low-level of NIO itself, as if the Swift
team does change the spelling of it at any point in the future we'll be
in real trouble. This way works almost as well, and makes our intent a
lot clearer.
Modifications:
- Extracted Heap and PriorityQueue to a new module.
- Made everything @inlinable to do our best to make performance
acceptable.
Result:
We can re-use PriorityQueue.
Motivation:
Issue for #1891 to add JSONSerialization in NIOFoundationCompat.
Modifications:
Adds:
JSONSerialization+ByteBuffer.swift
JSONSerialization+ByteBufferTest.swift
JSONSerialization+ByteBufferTest+XCTest.swift
Result:
Support for transforming byteBuffer data to a Foundation object.
Co-authored-by: Cory Benfield <lukasa@apple.com>
A convenience method to write a message and prepend its length in front of it.
### Motivation:
Many protocols use a length field to split a byte stream into multiple separate messages. `swift-nio-extras` supports these kind of protocols through `LengthFieldPrepender` and friends. However, sometimes a channel handler is not the right tool, especially if the a message itself contains multiple dynamic length fields. One example is [RSocket Routing Metadata Extensions](https://github.com/rsocket/rsocket/blob/master/Extensions/Routing.md).
A `ByteBuffer` is probably a better tool to create this message. This can be tricky to get right if strings are part of the message. A naive solution like the following has two issues:
```swift
func add(tag: String, to buffer: inout ByteBuffer) {
let length = tag.count
buffer.writeInteger(UInt8(length))
buffer.writeString(tag)
}
```
The `count` property in Swift is different than in most other languages. It does **not** return the number of UTF-8 code points and can result in incorrect behavior if multi-byte characters are used (e.g. Emojis). Furthermore, the size of the string could be larger than `UInt8` can represent.
### Modifications:
This PR introduces a new method on `ByteBuffer` called `writeLengthPrefix(endianness:as:writeMessage:)`. It solves both problems mentioned above. It first reserves space for the length prefix and delegates the actually writing of the message to `writeMessage`. Afterwards, it counts how many bytes were written during the call to `writeMessage` and then sets it on the previously reserved space in front of the actually message.
### Result:
We can now write arbitrary content to a buffer and prefix it with the number of bytes written.
```swift
func add(tag: String, to buffer: inout ByteBuffer) {
buffer.writeLengthPrefix(as: UInt8.self) { buffer in
buffer.writeString(tag)
}
}
```
* issue-1036: Fix for dates on Linux test generation script.
Motivation:
Fix for issue-1036 to add a date or date range for the header.
These dates are derived from the git log on the file.
Modifications:
Parsing method to extract the date or date range from the git log
of the file. This was added to generate_linux_tests.
Result:
Dynamic date or date range added to the file header.
* issue-1036: Added revised headers from generation script
* Regenerating dates on XCTests.
Motivation:
Addressing feedback from PR.
Modifications:
All of the XCTests now have -2021 as a header date attached to them.
Result:
Dates modified on the XCTests.
Co-authored-by: Cory Benfield <lukasa@apple.com>
* add randomRequestKey method
* fix some typos
* fix compilation on 5.0
* run generate_linux_tests.rb
* add test with default random number generator
* @inlineable
* base64Encoding @inlineable
* add static method to generate a random mask
* use SystemRandomNumberGenerator by default and add documentation
* use WebSocketMaskingKey instead of Self to support Swift 5.0
* add tests for random masking key
* add return keyword to support Swift 5.0
* run scripts/generate_linux_tests.rb
* rename T to Generator
* test SystemRandomNumberGenerator
* Revert "test SystemRandomNumberGenerator"
This reverts commit d9bdbe57ac.
* work around thread sanitizer bug on Swift 5.3
* implement and test WebSocketFrameAggregator
* add documentation
* fix review comments
* run scripts/generate_linux_tests.rb
* fix Swift 5.0
* move redundant channel creation into init
* create channel in setUp
* wrap all trys in XCTAssertNoThrow
* cache accumulatedFrameSize
* accumulate buffered frames into the first frames data buffer
* Revert "accumulate buffered frames into the first frames data buffer"
This reverts commit 0f83823f95.
* use channel allocator
Motivation:
Right now all tasks scheduled onto an event loop from a different loop
wake up the underlying selector by writing to an eventfd. This is a
fairly inefficient way to handle things: it incurs a lot of syscall
traffic unnecessarily.
Given that we currently protect the pending tasks queue with a big dumb
lock, we can safely keep track of whether we're going to dequeue this
task. If we are, we don't need to wake the selector.
Modifications:
- Keep track of whether we're de-queueing tasks or not.
- Arrange to wake the selector only once.
Result:
Cheaper task enqueueing on hot loops.
Motivation:
With the registration ID refactoring (#1801) we'll now have the third
reimplementation of integer bit packing. Three is one too many, let's
make it generic.
Modifications:
Introduce generic integer bit packing and use it for _UInt56/24.
Result:
- Less bit packing code.
- Will simplify #1801
Motivation:
Be able to run and test swift-nio on Android.
Modifications:
- Remove the custom ifaddrs and use the one from the Android NDK instead.
- Enable a bunch of conditionally-compiled code for Android.
- Add a handful of constants and other Android declarations.
- Cast some types because of mismatches specific to Android.
Result:
Most tests pass on Android AArch64 and ARMv7.
Co-authored-by: Cory Benfield <lukasa@apple.com>
* HTTPServerObjectAggregator for requests
* Apply suggestions from code review
Co-authored-by: Cory Benfield <lukasa@apple.com>
* most of code review comments addressed
* tidy up state machine close
* bad line breaks
* more verbose error reporting
* removes expect:continue functionality due to #1422
* public type naming and error handling tweaks
* wrong expectation in test
* do not quietly swallow unexpected errors in channelRead
Co-authored-by: Cory Benfield <lukasa@apple.com>
Motivation:
SwiftNIO lacks a convenience API for performing blocking IO / tasks. As
this is a fairly common task it then requires the clients to make ad hoc
implementations that address this requirement.
Modifications:
Extension to DispatchQueue with the following method to schedule a work
item to the `DispatchQueue` and return and `EventLoopFuture` for the
result returned:
- `asyncWithFuture<NewValue>(eventLoop: EventLoop, _ callbackMayBlock: @escaping () throws -> NewValue) -> EventLoopFuture<NewValue>`
Added new unit tests for this function both when the promise succeeds
and fails.
Extention to EventLoopFuture with the following public functions:
- `flatMapBlocking<NewValue)(onto queue DispatchQueue, _ callbackMayBlock: @escpaing (Value) throws -> NewValue) -> EventLoopFuture<NewValue>`
- `whenSuccessBlocking(onto queue DispatchQueue, _ callbackMayBlock: @escaping (Value) -> Void) -> EventLoopFuture<NewValue>`
- `whenFailureBlocking()onto queue DispatchQueue, _ callbackMayBlock: @escaping (Error) -> Void) -> EventLoopFuture<NewValue>`
- `whenCompleteBlocking(onto queue DispatchQueue, _ callbackMayBlock: @escaping (Result<Value, Error>) -> Void) -> EventLoopFuture<NewValue>`
These functions may all be called safely with callbacks that perform blocking IO / Tasks.
Added new unit tests to EventLoopFutureTest.swift for each new function.
Result:
New public API for `EventLoopFuture` that allows scheduling of blocking IO / Tasks.
Motivation:
All tests should get run.
Modifications:
The script to locate linux tests only scans files with names ending with Test/Tests.
Change the filename to match this.
Result:
All tests are run now on linux.
Motivation:
Per SR-10219, Data does not implement withContiguousStorageWithAvailable
and the maintainers do not believe it can. This forces code that writes
Data into ByteBuffers into slow-paths. Given that Data conforms to
ContiguousBytes and DataProtocol, we can arrange to serve a number of
use-cases by providing fast paths for all conforming types. This makes
it easier to use types vended by other Foundation-using libraries, such
as Swift Crypto.
Modifications:
- Implement `writeContiguousBytes` and `setContiguousBytes`.
- Implement `writeData` and `setData` for `DataProtocol` implementations
Result:
Better support for writing `Data` and friends into `ByteBuffer`s.
Motivation:
cmsghdrs can be used to send an receive extra data on UDP packets.
For example ECN data.
Modifications:
Map in Linux and Darwin versions of cmsghdr macros as functions.
Create strctures for holding a received collection; parsing ecn
data from a received collection and building a collection suitable
for sending.
Result:
Functions to manipulate cmsghdr and data exist.
Motivation:
TimeAmount wasn't `Hashable` and there's no clear reason it shouldn't
be.
Modifications:
Add synthesized `Hashable` conformance to `TimeAmount`
Result:
`TimeAmount` is hashable.
Motivation:
The current System.coreCount implementation relies on _SC_NPROCESSORS_ONLN so isn't cgroups aware which might have a bad impact for apps runnings in containers (e.g. Docker, Kubernetes, Amazon ECS...).
Modifications:
- Changed System.coreCount on Linux only to make it read from CFS quotas and cpusets when present.
- Removed incorrect precondition in Sources/NIO/LinuxCPUSet.swift
Result:
System.coreCount returns correct values when apps run in containers.
Co-authored-by: Johannes Weiss <johannesweiss@apple.com>
Motivation:
Previously, NIOThreadPool would use super long thread names. Linux
doesn't allow more than 15 bytes (+ \0 char) so in the end they would
never get set at all.
Modifications:
- Shorten NIOThreadPool's thread names
- Only take the first 15 characters of thread names in general on Linux
Result:
We don't lose thread names on Linux.
resolves#1355 shaving off some bytes from IOError struct
Motivation:
It's important for frequently used errors to fit in existential containers because otherwise throwing them leads to allocations.
Modifications:
- Don't store `StaticString`s anymore which are 2 words + 1 bytes wide
Result:
Fewer allocations when IOErrors are thrown.
## Motivation:
In SwiftNIO's codebase, most things were always testable just as well as user code is. With one big missing piece: We could never make the `Selector` do arbitrary things to the `Channel` implementations which lead to super expensive to (create, read, understand) integration tests like [`testWriteAndFlushFromReentrantFlushNowTriggeredOutOfWritabilityWhereOuterSaysAllWrittenAndInnerDoesNot `](5dc9083c6a/Tests/NIOTests/StreamChannelsTest.swift (L468-L760)).
## Modification
Add SAL, a 'syscall abstraction layer' which makes it possible to assert all syscalls that come through a `Socket` or the `Selector` by mocking them. It lets the NIO programmer play the kernel in these "SAL tests" which makes it possible to assert the exact syscalls (with their parameters) and decide the syscalls' returns.
## Result
Finally, we're able to write proper unit tests for scenarios where the `Channel` drove the `Selector` incorrectly.
* Add NIOSingleStepByteToMessageDecoder and NIOSingleStepByteToMessageProcessor
for processing byte streams outside of the channel pipeline.
Motivation:
ByteToMessageHandler works well when you can embed it in your pipeline, but if
you have a wrapped protocol (e.g. sending a protocol over HTTP), you can't use
it without losing the context of the outer protocol (e.g. HTTP headers/trailers).
Modifications:
Added NIOSingleStepByteToMessageDecoder and NIOSingleStepByteToMessageProcessor.
Result:
New functionality above.
Motivation:
It's important to also test deprecated functionliaty. One way of
achieving this without warnings is to also deprecate the tests that test
this deprecated functionality. Unfortunately, on Linux we need to
generate lists of tests which would then reference deprecated tests
(which gives us a warning).
Modifications:
Deprecate test suites and the main test runner all the way to the top so
never get warnings.
Result:
Possible to test deprecated functionlity without warnings.
Motivation:
So far, it has been harder than necessary to use Codable & ByteBuffer.
These new APIs should simplify that and allow future optimisations.
Modifications:
Add new API to use `JSONEncoder` and `JSONDecoder` directly with
`ByteBuffer`.
Result:
Easier Codable + ByteBuffer usage.
Motivation:
There are use-cases for SwiftNIO where there are no actual sockets but
rather two pipe file descriptors - one for input, one for output.
There's no real reason why SwiftNIO shouldn't work for those.
Modifications:
Add a PipeChannel.
Result:
More use-cases for SwiftNIO.
* NIOHTTP1TestServer implementation
* Test simple request and two concurrent requests
* Accepting multiple connections but keeping only one active at a time
* Linux tests
* Addressed some PR comments
Motivation:
I ran across a case in a test where `HTTPResponseStatus(statusCode: 418)`
unexpectedly returned a `.custom` response status.
Modifications:
- Add a missing `case` for creating a status from `418`
- Shuffled some of cases around so that they are correctly listed in the
correct 2xx, 3xx, 4xx etc. sections
Result:
- `HTTPResponseStatus(statusCode: 418)` now returns `.imATeapot`
Motivation:
There is a client protocol upgrader but, unlike the server protocol upgrader, it does not have a WebSocket protocol as part of the project.
Modifications:
Made the magic WebSocket GUID public to the WebSockets project.
Added a NIOWebSocketClientUpgrader.
Added tests for the upgrader.
Updated the Linux test script files.
Result:
The project now has a WebSocket client upgrader to match the WebSocket server upgrader.
* Reorder ‘channel active’ calls to the same order as `_ChannelInboundHandler` and their likely chronological order.
Motivation:
When first viewing the example classes, coming to the ‘read’ method first, leaves the subject unclear as to what the method is ‘reading’.
It is preferable to view something being sent first, and then to view the reading of the response.
It also matches the call order against the protocol making it a little easier for those unfamiliar with the protocol to see which methods have been implemented.
Modifications:
Moved channel active calls to be top of the class.
Despite the diff there are no actual code modifications.
UDP Client changed to indent using spaces to match rest of project. Incidental change.
Result:
The examples are slighter clearer to read, particularly for newcomers to swift-no as the calls are in a logical chronological order.
* Fix an error in the upgrader and its tests which checked the connection header incorrectly.
Motivation:
To correct the logic in the upgrader. Previously it checked if the upgraders required headers include the connection header. Now it checks that the connection header value was present as a separate header in the incoming request.
To prepare the class naming for the presence of a client upgrader.
Modifications:
Slight change in server upgrader handler logic with accompanying tests.
Renamed upgrader and test files to include the word ‘server’.
Ran the linux test script.
Result:
The server upgrader now checks for the presence of the connection header value as a header of its own.
* Rename WebSocketServer upgrader to make the naming clearer and make way for a client version.
Motivation:
To make the web socket upgrader naming clearer, particularly once we add a client version.
Modifications:
Rename WebSocketUpgrader too WebSocketServerUpgrader.
Result:
Improved clarity of naming on the web socket upgrader.
* Adds correct naming conventions to NIOWebSocketServerUpgrader. Sets back an incorrect fix (non-fix).
* Move deprecation for NIOWebSocketServerUpgrader typealias out of API shims file.
* Makes HTTPEncoder removable.
Adds a client upgrade handler with tests.
Adds the ability to use a client upgrader to the client setup.
Initial linux file update.
* Allow assertContains to be accessed publicly, so that it can be used in the client side tests.
* Update tests to remove server upgrader from client tests.
Change client tests to use Embedded channel.
Update HTTPUpgradeConfiguration class name to be NIOHTTPServerUpgradeConfiguration
Few other small stylistic changes.
* Removed awaiting upgrader state.
Removed future from handle upgrade call as is synchronous.
Removed protocol iterator from handle upgrade call as is not required by the client.
* Ensure that the client handler calls for the HTTPPipeline are backwards compatible.
Ensure that incoming promises to the handler write call are completed.
Neaten the upgrade header formation to remove looping.
Improve the correctness of the upgrade error throwing.
* Update scripts to match new unit tests.
* Change the documentation for HTTPServerPipeline to remove nil options which have now been removed.
* Restore an incorrectly added server pipeline comment change and make it to the client pipeline instead.
* Raise the allocation limits in the docker images to allow the tests to pass.
Motivation:
To make the web socket server end to end tests class more descriptive.
This will allow ‘WebSocketClientEndToEndTests’ or other end to end tests to be added at a later date.
Modifications:
Renames the class and filename for ‘EndToEndTests’ to ‘WebSocketServerEndToEndTests’
Result:
The nomenclature now allow for other end to end tests to be added.
Motivation:
In many unit tests, it's important to count the events. So far we have
many many ad-hoc handlers to do so but having one that counts all events
would be handy.
Modifications:
Add `EventCounterHandler` to count all events travelling the
`ChannelPipeline`.
Result:
Easier unit testing
Motivation:
When writing B2MDs, there are a couple of scenarios that always need to
be tested: firehose feeding, drip feeding, many messages, ...
It's tedious writing those tests over and over again for every B2MD.
Modifications:
- Add a simple B2MD verifier that users can use in unit tests.
- Add a new, public `NIOTestUtils` module which contains utilities
mostly useful for testing. Crucially however, it does not depend on
`XCTest` so it can be used it all targets.
Result:
Hopefully fewer bugs in B2MDs.
Motivation:
To make the web socket upgrader naming clearer, particularly once we add a client version.
Modifications:
Rename WebSocketUpgrader too WebSocketServerUpgrader.
Result:
Improved clarity of naming on the web socket upgrader.
Johannes Weiss