diff --git a/Documentation/Proposals/0000-proposal-template.md b/Documentation/Proposals/0000-proposal-template.md deleted file mode 100644 index 342c04000..000000000 --- a/Documentation/Proposals/0000-proposal-template.md +++ /dev/null @@ -1,184 +0,0 @@ -# Feature name - -* Proposal: [SWT-NNNN](NNNN-filename.md) -* Authors: [Author 1](https://github.com/author1), [Author 2](https://github.com/author2) -* Status: **Awaiting implementation** or **Awaiting review** -* Bug: _if applicable_ [swiftlang/swift-testing#NNNNN](https://github.com/swiftlang/swift-testing/issues/NNNNN) -* Implementation: [swiftlang/swift-testing#NNNNN](https://github.com/swiftlang/swift-testing/pull/NNNNN) -* Previous Proposal: _if applicable_ [SWT-XXXX](XXXX-filename.md) -* Previous Revision: _if applicable_ [1](https://github.com/swiftlang/swift-testing/blob/...commit-ID.../Documentation/Proposals/NNNN-filename.md) -* Review: ([pitch](https://forums.swift.org/...)) - -When filling out this template, you should delete or replace all of the text -except for the section headers and the header fields above. For example, you -should delete everything from this paragraph down to the Introduction section -below. - -As a proposal author, you should fill out all of the header fields. Delete any -header fields marked _if applicable_ that are not applicable to your proposal. - -When sharing a link to the proposal while it is still a PR, be sure to share a -live link to the proposal, not an exact commit, so that readers will always see -the latest version when you make changes. On GitHub, you can find this link by -browsing the PR branch: from the PR page, click the "username wants to merge ... -from username:my-branch-name" link and find the proposal file in that branch. - -`Status` should reflect the current implementation status while the proposal is -still a PR. The proposal cannot be reviewed until an implementation is available, -but early readers should see the correct status. - -`Bug` should be used when this proposal is fixing a bug with significant -discussion in the bug report. It is not necessary to link bugs that do not -contain significant discussion or that merely duplicate discussion linked -somewhere else. Do not link bugs from private bug trackers. - -`Implementation` should link to the PR(s) implementing the feature. If the -proposal has not been implemented yet, or if it simply codifies existing -behavior, just say that. If the implementation has already been committed to the -main branch (as an experimental feature or SPI), mention that. If the -implementation is spread across multiple PRs, just link to the most important -ones. - -`Previous Proposal` should be used when there is a specific line of succession -between this proposal and another proposal. For example, this proposal might -have been removed from a previous proposal so that it can be reviewed separately, -or this proposal might supersede a previous proposal in some way that was felt -to exceed the scope of a "revision". Include text briefly explaining the -relationship, such as "Supersedes SWT-1234" or "Extracted from SWT-01234". If -possible, link to a post explaining the relationship, such as a review decision -that asked for part of the proposal to be split off. Otherwise, you can just -link to the previous proposal. - -`Previous Revision` should be added after a major substantive revision of a -proposal that has undergone review. It links to the previously reviewed revision. -It is not necessary to add or update this field after minor editorial changes. - -`Review` is a history of all discussion threads about this proposal, in -chronological order. Use these standardized link names: `pitch` `review` -`revision` `acceptance` `rejection`. If there are multiple such threads, spell -the ordinal out: `first pitch` `second review` etc. - -## Introduction - -A short description of what the feature is. Try to keep it to a single-paragraph -"elevator pitch" so the reader understands what problem this proposal is -addressing. - -## Motivation - -Describe the problems that this proposal seeks to address. If the problem is -that some common pattern is currently hard to express, show how one can -currently get a similar effect and describe its drawbacks. If it's completely -new functionality that cannot be emulated, motivate why this new functionality -would help Swift developers test their code more effectively. - -## Proposed solution - -Describe your solution to the problem. Provide examples and describe how they -work. Show how your solution is better than current workarounds: is it cleaner, -safer, or more efficient? - -This section doesn't have to be comprehensive. Focus on the most important parts -of the proposal and make arguments about why the proposal is better than the -status quo. - -## Detailed design - -Describe the design of the solution in detail. If it includes new API, show the -full API and its documentation comments detailing what it does. If it involves -new macro logic, describe the behavior changes and include a succinct example of -the additions or modifications to the macro expansion code. The detail in this -section should be sufficient for someone who is *not* one of the authors to be -able to reasonably implement the feature. - -## Source compatibility - -Describe the impact of this proposal on source compatibility. As a general rule, -all else being equal, test code that worked in previous releases of the testing -library should work in new releases. That means both that it should continue to -build and that it should continue to behave dynamically the same as it did -before. - -This is not an absolute guarantee, and the testing library administrators will -consider intentional compatibility breaks if their negative impact can be shown -to be small and the current behavior is causing substantial problems in practice. - -For proposals that affect testing library API, consider the impact on existing -clients. If clients provide a similar API, will type-checking find the right one? -If the feature overloads an existing API, is it problematic that existing users -of that API might start resolving to the new API? - -## Integration with supporting tools - -In this section, describe how this proposal affects tools which integrate with -the testing library. Some features depend on supporting tools gaining awareness -of the new feature for users to realize new benefits. Other features do not -strictly require integration but bring improvement opportunities which are worth -considering. Use this section to discuss any impact on tools. - -This section does need not to include details of how this proposal may be -integrated with _specific_ tools, but it should consider the general ways that -tools might support this feature and note any accompanying SPI intended for -tools which are included in the implementation. Note that tools may evolve -independently and have differing release schedules than the testing library, so -special care should be taken to ensure compatibility across versions according -to the needs of each tool. - -## Future directions - -Describe any interesting proposals that could build on this proposal in the -future. This is especially important when these future directions inform the -design of the proposal, for example by making sure an interface meant for tools -integration can be extended to include additional information. - -The rest of the proposal should generally not talk about future directions -except by referring to this section. It is important not to confuse reviewers -about what is covered by this specific proposal. If there's a larger vision that -needs to be explained in order to understand this proposal, consider starting a -discussion thread on the forums to capture your broader thoughts. - -Avoid making affirmative statements in this section, such as "we will" or even -"we should". Describe the proposals neutrally as possibilities to be considered -in the future. - -Consider whether any of these future directions should really just be part of -the current proposal. It's important to make focused, self-contained proposals -that can be incrementally implemented and reviewed, but it's also good when -proposals feel "complete" rather than leaving significant gaps in their design. -An an example from the Swift project, when -[SE-0193](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0193-cross-module-inlining-and-specialization.md) -introduced the `@inlinable` attribute, it also included the `@usableFromInline` -attribute so that declarations used in inlinable functions didn't have to be -`public`. This was a relatively small addition to the proposal which avoided -creating a serious usability problem for many adopters of `@inlinable`. - -## Alternatives considered - -Describe alternative approaches to addressing the same problem. This is an -important part of most proposal documents. Reviewers are often familiar with -other approaches prior to review and may have reasons to prefer them. This -section is your first opportunity to try to convince them that your approach is -the right one, and even if you don't fully succeed, you can help set the terms -of the conversation and make the review a much more productive exchange of ideas. - -You should be fair about other proposals, but you do not have to be neutral; -after all, you are specifically proposing something else. Describe any -advantages these alternatives might have, but also be sure to explain the -disadvantages that led you to prefer the approach in this proposal. - -You should update this section during the pitch phase to discuss any -particularly interesting alternatives raised by the community. You do not need -to list every idea raised during the pitch, just the ones you think raise points -that are worth discussing. Of course, if you decide the alternative is more -compelling than what's in the current proposal, you should change the main -proposal; be sure to then discuss your previous proposal in this section and -explain why the new idea is better. - -## Acknowledgments - -If significant changes or improvements suggested by members of the community -were incorporated into the proposal as it developed, take a moment here to thank -them for their contributions. This is a collaborative process, and everyone's -input should receive recognition! - -Generally, you should not acknowledge anyone who is listed as a co-author. diff --git a/Documentation/Proposals/0001-refactor-bug-inits.md b/Documentation/Proposals/0001-refactor-bug-inits.md index 0a4f00566..18927db22 100644 --- a/Documentation/Proposals/0001-refactor-bug-inits.md +++ b/Documentation/Proposals/0001-refactor-bug-inits.md @@ -1,168 +1,10 @@ # Dedicated `.bug()` functions for URLs and IDs -* Proposal: [SWT-0001](0001-refactor-bug-inits.md) -* Authors: [Jonathan Grynspan](https://github.com/grynspan) -* Status: **Implemented (Swift 6.0)** -* Implementation: [swiftlang/swift-testing#401](https://github.com/swiftlang/swift-testing/pull/401) -* Review: ([pitch](https://forums.swift.org/t/pitch-dedicated-bug-functions-for-urls-and-ids/71842)), ([acceptance](https://forums.swift.org/t/swt-0001-dedicated-bug-functions-for-urls-and-ids/71842/2)) - -## Introduction - -One of the features of swift-testing is a test traits system that allows -associating metadata with a test suite or test function. One trait in -particular, `.bug()`, has the potential for integration with development tools -but needs some refinement before integration would be practical. - -## Motivation - -A test author can associate a bug (AKA issue, problem, ticket, etc.) with a test -using the `.bug()` trait, to which they pass an "identifier" for the bug. The -swift-testing team's intent here was that a test author would pass the unique -identifier of the bug in the test author's preferred bug-tracking system (e.g. -GitHub Issues, Bugzilla, etc.) and that any tooling built around this trait -would be able to infer where the bug was located and how to view it. - -It became clear immediately that a generic system for looking up bugs by unique -identifier in an arbitrary and unspecified database wouldn't be a workable -solution. So we modified the description of `.bug()` to explain that, if the -identifier passed to it was a valid URL, then it would be "interpreted" as a URL -and that tools could be designed to open that URL as needed. - -This design change then placed the burden of parsing each `.bug()` trait and -potentially mapping it to a URL on tools. swift-testing itself avoids linking to -or using Foundation API such as `URL`, so checking for a valid URL inside the -testing library was not feasible either. - -## Proposed solution - -To solve the underlying problem and allow test authors to specify a URL when -available, or just an opaque identifier otherwise, we propose splitting the -`.bug()` function up into two overloads: - -- The first overload takes a URL string and additional optional metadata; -- The second overload takes a bug identifier as an opaque string or integer and, - optionally, a URL string. - -Test authors are then free to specify any combination of URL and opaque -identifier depending on the information they have available and their specific -needs. Tools authors are free to consume either or both of these properties and -present them where appropriate. - -## Detailed design - -The `Bug` trait type and `.bug()` trait factory function shall be refactored -thusly: - -```swift -/// A type representing a bug report tracked by a test. -/// -/// To add this trait to a test, use one of the following functions: -/// -/// - ``Trait/bug(_:_:)`` -/// - ``Trait/bug(_:id:_:)-10yf5`` -/// - ``Trait/bug(_:id:_:)-3vtpl`` -public struct Bug: TestTrait, SuiteTrait, Equatable, Hashable, Codable { - /// A URL linking to more information about the bug, if available. - /// - /// The value of this property represents a URL conforming to - /// [RFC 3986](https://www.ietf.org/rfc/rfc3986.txt). - public var url: String? - - /// A unique identifier in this bug's associated bug-tracking system, if - /// available. - /// - /// For more information on how the testing library interprets bug - /// identifiers, see . - public var id: String? - - /// The human-readable title of the bug, if specified by the test author. - public var title: Comment? -} - -extension Trait where Self == Bug { - /// Construct a bug to track with a test. - /// - /// - Parameters: - /// - url: A URL referring to this bug in the associated bug-tracking - /// system. - /// - title: Optionally, the human-readable title of the bug. - /// - /// - Returns: An instance of ``Bug`` representing the specified bug. - public static func bug(_ url: _const String, _ title: Comment? = nil) -> Self - - /// Construct a bug to track with a test. - /// - /// - Parameters: - /// - url: A URL referring to this bug in the associated bug-tracking - /// system. - /// - id: The unique identifier of this bug in its associated bug-tracking - /// system. - /// - title: Optionally, the human-readable title of the bug. - /// - /// - Returns: An instance of ``Bug`` representing the specified bug. - public static func bug(_ url: _const String? = nil, id: some Numeric, _ title: Comment? = nil) -> Self - - /// Construct a bug to track with a test. - /// - /// - Parameters: - /// - url: A URL referring to this bug in the associated bug-tracking - /// system. - /// - id: The unique identifier of this bug in its associated bug-tracking - /// system. - /// - title: Optionally, the human-readable title of the bug. - /// - /// - Returns: An instance of ``Bug`` representing the specified bug. - public static func bug(_ url: _const String? = nil, id: _const String, _ title: Comment? = nil) -> Self -} -``` - -The `@Test` and `@Suite` macros have already been modified so that they perform -basic validation of a URL string passed as input and emit a diagnostic if the -URL string appears malformed. - -## Source compatibility - -This change is expected to be source-breaking for test authors who have already -adopted the existing `.bug()` functions. This change is source-breaking for code -that directly refers to these functions by their signatures. This change is -source-breaking for code that uses the `identifier` property of the `Bug` type -or expects it to contain a URL. - -## Integration with supporting tools - -Tools that integrate with swift-testing and provide lists of tests or record -results after tests have run can use the `Bug` trait on tests to present -relevant identifiers and/or URLs to users. - -Tools that use the experimental event stream output feature of the testing -library will need a JSON schema for bug traits on tests. This work is tracked in -a separate upcoming proposal. - -## Alternatives considered - -- Inferring whether or not a bug identifier was a URL by parsing it at runtime - in tools. As discussed above, this option would require every tool that - integrates with swift-testing to provide its own URL-parsing logic. - -- Using different argument labels (e.g. the label `url` for the URL argument - and/or no label for the `id` argument.) We felt that URLs, which are - recognizable by their general structure, did not need labels. At least one - argument must have a label to avoid ambiguous resolution of the `.bug()` - function at compile time. - -- Inferring whether or not a bug identifier was a URL by parsing it at compile- - time or at runtime using `Foundation.URL` or libcurl. swift-testing actively - avoids linking to Foundation if at all possible, and libcurl would be a - platform-specific solution (Windows doesn't ship with libcurl, but does have - `InternetCrackUrlW()` whose parsing engine differs.) We also run the risk of - inappropriately interpreting some arbitrary bug identifier as a URL when it is - not meant to be parsed that way. - -- Removing the `.bug()` trait. We see this particular trait as having strong - potential for integration with tools and for use by test authors; removing it - because we can't reliably parse URLs would be unfortunate. - -## Acknowledgments - -Thanks to the swift-testing team and managers for their contributions! Thanks to -our community for the initial feedback around this feature. +> [!NOTE] +> This proposal was accepted before Swift Testing began using the Swift +> evolution review process. Its original identifier was SWT-0001 but its prefix +> has been changed to "ST" and it has been relocated to the +> [swift-evolution](https://github.com/swiftlang/swift-evolution) repository. + +To view this proposal, see +[ST-0001: Dedicated `.bug()` functions for URLs and IDs](https://github.com/swiftlang/swift-evolution/blob/main/proposals/testing/0001-refactor-bug-inits.md). diff --git a/Documentation/Proposals/0002-json-abi.md b/Documentation/Proposals/0002-json-abi.md index 0af939972..8fd1e3242 100644 --- a/Documentation/Proposals/0002-json-abi.md +++ b/Documentation/Proposals/0002-json-abi.md @@ -1,423 +1,10 @@ # A stable JSON-based ABI for tools integration -* Proposal: [SWT-0002](0002-json-abi.md) -* Authors: [Jonathan Grynspan](https://github.com/grynspan) -* Status: **Implemented (Swift 6.0)** -* Implementation: [swiftlang/swift-testing#383](https://github.com/swiftlang/swift-testing/pull/383), - [swiftlang/swift-testing#402](https://github.com/swiftlang/swift-testing/pull/402) -* Review: ([pitch](https://forums.swift.org/t/pitch-a-stable-json-based-abi-for-tools-integration/72627)), ([acceptance](https://forums.swift.org/t/pitch-a-stable-json-based-abi-for-tools-integration/72627/4)) - -## Introduction - -One of the core components of Swift Testing is its ability to interoperate with -Xcode 16, VS Code, and other tools. Swift Testing has been fully open-sourced -across all platforms supported by Swift, and can be added as a package -dependency (or—eventually—linked from the Swift toolchain.) - -## Motivation - -Because Swift Testing may be used in various forms, and because integration with -various tools is critical to its success, we need it to have a stable interface -that can be used regardless of how it's been added to a package. There are a few -patterns in particular we know we need to support: - -- An IDE (e.g. Xcode 16) that builds and links its own copy of Swift Testing: - the copy used by the IDE might be the same as the copy that tests use, in - which case interoperation is trivial, but it may also be distinct if the tests - use Swift Testing as a package dependency. - - In the case of Xcode 16, Swift Testing is built as a framework much like - XCTest and is automatically linked by test targets in an Xcode project or - Swift package, but if the test target specifies a package dependency on Swift - Testing, that dependency will take priority when the test code is compiled. - -- An IDE (e.g. VS Code) that does _not_ link directly to Swift Testing (and - perhaps, as with VS Code, cannot because it is not natively compiled): such an - IDE needs a way to configure and invoke test code and then to read events back - as they occur, but cannot touch the Swift symbols used by the tests. - - In the case of VS Code, because it is implemented using TypeScript, it is not - able to directly link to Swift Testing or other Swift libraries. In order for - it to interpret events from a test run like "test started" or "issue - recorded", it needs to receive those events in a format it can understand. - -Tools integration is important to the success of Swift Testing. The more tools -provide integrations for it, the more likely developers are to adopt it. The -more developers adopt, the more tests are written. And the more tests are -written, the better our lives as software engineers will be. - -## Proposed solution - -We propose defining and implementing a stable ABI for using Swift Testing that -can be reliably adopted by various IDEs and other tools. There are two aspects -of this ABI we need to implement: - -- A stable entry point function that can be resolved dynamically at runtime (on - platforms with dynamic loaders such as Darwin, Linux, and Windows.) This - function needs a signature that will not change over time and which will take - input and pass back asynchronous output in a format that a wide variety of - tools will be able to interpret (whether they are written in Swift or not.) - - This function should be implemented in Swift as it is expected to be used by - code that can call into Swift, but which cannot rely on the specific binary - minutiae of a given copy of Swift Testing. - -- A stable format for input that can be passed to the entry point function and - which can also be passed at the command line; and a stable format for output - that can be consumed by tools to interpret test results. - - Some tools cannot directly link to Swift code and must instead rely on - command-line invocations of `swift test`. These tools will be able to pass - their test configuration and options as an argument in the stable format and - will be able to receive event information in the same stable format via a - dedicated channel such as a file or named pipe. - -> [!NOTE] -> This document proposes defining a stable format for input and output, but only -> actually defines the JSON schema for _output_. We intend to define the schema -> for input in a subsequent proposal. -> -> In the interim, early adopters can encode an instance of Swift Testing's -> `__CommandLineArguments_v0` type using `JSONEncoder`. - -## Detailed design - -We propose defining the stable input and output format using JSON as it is -widely supported across platforms and languages. The proposed JSON schema for -output is defined [here](../ABI/JSON.md). - -### Example output - -The proposed schema is a sequence of JSON objects written to an event handler or -file stream. When a test run starts, Swift Testing first emits a sequence of -JSON objects representing each test that is part of the planned run. For -example, this is the JSON representation of Swift Testing's own `canGetStdout()` -test function: - -```json -{ - "kind": "test", - "payload": { - "displayName": "Can get stdout", - "id": "TestingTests.FileHandleTests/canGetStdout()/FileHandleTests.swift:33:4", - "isParameterized": false, - "kind": "function", - "name": "canGetStdout()", - "sourceLocation": { - "column": 4, - "fileID": "TestingTests/FileHandleTests.swift", - "line": 33 - } - }, - "version": 0 -} -``` - -A tool that is observing this data stream can build a map or dictionary of test -IDs to comprehensive test details if needed. Once all tests in the planned run -have been written out, testing begins. Swift Testing writes a sequence of JSON -objects representing various events such as "test started" or "issue recorded". -For example, here is an abridged sequence of events generated for a test that -records a failed expectation: - -```json -{ - "kind": "event", - "payload": { - "instant": { - "absolute": 266418.545786299, - "since1970": 1718302639.76747 - }, - "kind": "testStarted", - "messages": [ - { - "symbol": "default", - "text": "Test \"Can get stdout\" started." - } - ], - "testID": "TestingTests.FileHandleTests/canGetStdout()/FileHandleTests.swift:33:4" - }, - "version": 0 -} - -{ - "kind": "event", - "payload": { - "instant": { - "absolute": 266636.524236724, - "since1970": 1718302857.74857 - }, - "issue": { - "isKnown": false, - "sourceLocation": { - "column": 7, - "fileID": "TestingTests/FileHandleTests.swift", - "line": 29 - } - }, - "kind": "issueRecorded", - "messages": [ - { - "symbol": "fail", - "text": "Expectation failed: (EOF → -1) == (feof(fileHandle) → 0)" - } - ], - "testID": "TestingTests.FileHandleTests/canGetStdout()/FileHandleTests.swift:33:4" - }, - "version": 0 -} - -{ - "kind": "event", - "payload": { - "instant": { - "absolute": 266636.524741106, - "since1970": 1718302857.74908 - }, - "kind": "testEnded", - "messages": [ - { - "symbol": "fail", - "text": "Test \"Can get stdout\" failed after 0.001 seconds with 1 issue." - } - ], - "testID": "TestingTests.FileHandleTests/canGetStdout()/FileHandleTests.swift:33:4" - }, - "version": 0 -} -``` - -Each event includes zero or more "messages" that Swift Testing intends to -present to the user. These messages contain human-readable text as well as -abstractly-specified symbols that correspond to the output written to the -standard error stream of the test process. Tools can opt to present these -messages in whatever ways are appropriate for their interfaces. - -### Invoking from the command line - -When invoking `swift test`, we propose adding three new arguments to Swift -Package Manager: - -| Argument | Value Type | Description | -|---|:-:|---| -| `--configuration-path` | File system path | Specifies a path to a file, named pipe, etc. containing test configuration/options. | -| `--event-stream-output-path` | File system path | Specifies a path to a file, named pipe, etc. to which output should be written. | -| `--event-stream-version` | Integer | Specifies the version of the stable JSON schema to use for output. | - -The process for adding arguments to Swift Package Manager is separate from the -process for Swift Testing API changes, so the names of these arguments are -speculative and are subject to change as part of the Swift Package Manager -review process. - -If `--configuration-path` is specified, Swift Testing will open it for reading -and attempt to decode its contents as JSON. If `--event-stream-output-path` is -specified, Swift Testing will open it for writing and will write a sequence of -[JSON Lines](https://jsonlines.org) to it representing the data and events -produced by the test run. `--event-stream-version` determines the stable schema -used for output; pass `0` to match the schema proposed in this document. - > [!NOTE] -> If `--event-stream-output-path` is specified but `--event-stream-version` is -> not, the format _currently_ used is based on direct JSON encodings of the -> internal Swift structures used by Swift Testing. This format is necessary to -> support Xcode 16 Beta 1. In the future, the default value of this argument -> will be assumed to equal the newest available JSON schema version (`0` as of -> this document's acceptance, i.e. the JSON schema will match what we are -> proposing here until a new schema supersedes it.) -> -> Tools authors that rely on the JSON schema are strongly advised to specify a -> version rather than relying on this behavior to avoid breaking changes in the -> future. - -On platforms that support them, callers can use a named pipe with -`--event-stream-output-path` to get live results back from the test run rather -than needing to wait until the file is closed by the test process. Named pipes -can be created on Darwin or Linux with the POSIX [`mkfifo()`](https://developer.apple.com/library/archive/documentation/System/Conceptual/ManPages_iPhoneOS/man2/mkfifo.2.html) -function or on Windows with the [`CreateNamedPipe()`](https://learn.microsoft.com/en-us/windows/win32/api/namedpipeapi/nf-namedpipeapi-createnamedpipew) -function. - -If `--configuration-path` is specified in addition to explicit command-line -options like `--no-parallel`, the explicit command-line options take priority. - -### Invoking from Swift - -Tools that can link to and call Swift directly have the option of instantiating -the tools-only SPI type `Runner`, however this is only possible if the tools and -the test target link to the exact same copy of Swift Testing. To support tools -that may link to a different copy (intentionally or otherwise), we propose -adding an exported symbol to the Swift Testing library with the following Swift -signature: - -```swift -@_spi(ForToolsIntegrationOnly) -public enum ABIv0 { - /* ... */ - - /// The type of the entry point to the testing library used by tools that want - /// to remain version-agnostic regarding the testing library. - /// - /// - Parameters: - /// - configurationJSON: A buffer to memory representing the test - /// configuration and options. If `nil`, a new instance is synthesized - /// from the command-line arguments to the current process. - /// - recordHandler: A JSON record handler to which is passed a buffer to - /// memory representing each record as described in `ABI/JSON.md`. - /// - /// - Returns: Whether or not the test run finished successfully. - /// - /// - Throws: Any error that occurred prior to running tests. Errors that are - /// thrown while tests are running are handled by the testing library. - public typealias EntryPoint = @convention(thin) @Sendable ( - _ configurationJSON: UnsafeRawBufferPointer?, - _ recordHandler: @escaping @Sendable (_ recordJSON: UnsafeRawBufferPointer) -> Void - ) async throws -> Bool - - /// The entry point to the testing library used by tools that want to remain - /// version-agnostic regarding the testing library. - /// - /// The value of this property is a Swift function that can be used by tools - /// that do not link directly to the testing library and wish to invoke tests - /// in a binary that has been loaded into the current process. The value of - /// this property is accessible from C and C++ as a function with name - /// `"swt_abiv0_getEntryPoint"` and can be dynamically looked up at runtime - /// using `dlsym()` or a platform equivalent. - /// - /// The value of this property can be thought of as equivalent to - /// `swift test --event-stream-output-path` except that, instead of streaming - /// JSON records to a named pipe or file, it streams them to an in-process - /// callback. - public static var entryPoint: EntryPoint { get } -} -``` - -The inputs and outputs to this function are typed as `UnsafeRawBufferPointer` -rather than `Data` because the latter is part of Foundation, and adding a public -dependency on a Foundation type would make it very difficult for Foundation to -adopt Swift Testing. It is a goal of the Swift Testing team to keep our Swift -dependency list as small as possible. - -### Invoking from C or C++ - -We expect most tools that need to make use of this entry point will not be able -to directly link to the exported Swift symbol and will instead need to look it -up at runtime using a platform-specific interface such as [`dlsym()`](https://developer.apple.com/library/archive/documentation/System/Conceptual/ManPages_iPhoneOS/man3/dlsym.3.html) -or [`GetProcAddress()`](https://learn.microsoft.com/en-us/windows/win32/api/libloaderapi/nf-libloaderapi-getprocaddress). -The `ABIv0.entryPoint` property's getter will be exported to C and C++ as: - -```c++ -extern "C" const void *_Nonnull swt_abiv0_getEntryPoint(void); -``` - -The value returned from this C function is a direct representation of the value -of `ABIv0.entryPoint` and can be cast back to its Swift function type using -[`unsafeBitCast(_:to:)`](https://developer.apple.com/documentation/swift/unsafebitcast%28_%3Ato%3A%29). - -On platforms where data-pointer-to-function-pointer conversion is disallowed per -the C standard, this operation is unsupported. See §6.3.2.3 and §J.5.7 of -[the C standard](https://www.open-std.org/jtc1/sc22/wg14/www/docs/n1256.pdf). - -> [!NOTE] -> Swift Testing is statically linked into the main executable when it is -> included as a package dependency. On Linux and other platforms that use the -> ELF executable format, symbol information for the main executable may not be -> available at runtime unless the `--export-dynamic` flag is passed to the -> linker. - -## Source compatibility - -The changes proposed in this document are additive. - -## Integration with supporting tools - -Tools are able to use the proposed additions as described above. - -## Future directions - -- Extending the JSON schema to cover _input_ as well as _output_. As discussed, - we will do so in a subsequent proposal. - -- Extending the JSON schema to include richer information about events such as - specific mismatched values in `#expect()` calls. This information is complex - and we need to take care to model it efficiently and clearly. - -- Adding Markdown or other formats to event messages. Rich text can be used by - tools to emphasize values, switch to code voice, provide improved - accessibility, etc. - -- Adding additional entry points for different access patterns. We anticipate - that a Swift function and a command-line interface are sufficient to cover - most real-world use cases, but it may be the case that tools could use other - mechanisms for starting test runs such as: - - Pure C or Objective-C interfaces; - - A WebAssembly and/or JavaScript [`async`-compatible](https://github.com/WebAssembly/component-model/blob/2f447274b5028f54c549cb4e28ceb493a471dd4b/design/mvp/Async.md) - interface; - - Platform-specific interfaces; or - - Direct bindings to other languages like Rust, Go, C#, etc. - -## Alternatives considered - -- Doing nothing. If we made no changes, we would be effectively requiring - developers to use Xcode for all Swift Testing development and would be - requiring third-party tools to parse human-readable command-line output. This - approach would run counter to several of the Swift project's high-level goals - and would not represent a true cross-platform solution. - -- Using direct JSON encodings of Swift Testing's internal types to represent - output. We initially attempted this and you can see the results in the Swift - Testing repository if you look for "snapshot" types. A major downside became - apparent quickly: these data types don't make for particularly usable JSON - unless you're using `JSONDecoder` to convert back to them, and the default - JSON encodings produced with `JSONEncoder` are not stable if we e.g. add - enumeration cases with associated values or add non-optional fields to types. - -- Using a format other than JSON. We considered using XML, YAML, Apple property - lists, and a few other formats. JSON won out pretty quickly though: it is - widely supported across platforms and languages and it is trivial to create - Swift structures that encode to a well-designed JSON schema using - `JSONEncoder`. Property lists would be just as easy to create, but it is a - proprietary format and would not be trivially decodable on non-Apple platforms - or using non-Apple tools. - -- Exposing the C interface as a function that returns heap-allocated memory - containing a Swift function reference. This allows us to emit a "thick" Swift - function but requires callers to manually manage the resulting memory, and it - may be difficult to reason about code that requires an extra level of pointer - indirection. By having the C entry point function return a thin Swift function - instead, the caller need only bitcast it and can call it directly, and the - equivalent Swift interface can simply be a property getter rather than a - function call. - -- Exposing the C interface as a function that takes a callback and a completion - handler as might traditionally used by Objective-C callers, of the form: - - ```c++ - extern "C" void swt_abiv0_entryPoint( - __attribute__((__noescape__)) const void *_Nullable configurationJSON, - size_t configurationJSONLength, - void *_Null_unspecified context, - void (*_Nonnull recordHandler)( - __attribute__((__noescape__)) const void *recordJSON, - size_t recordJSONLength, - void *_Null_unspecified context - ), - void (*_Nonnull completionHandler)( - _Bool success, - void *_Null_unspecified context - ) - ); - ``` - - The known clients of the native entry point function are all able to call - Swift code and do not need this sort of interface. If there are other clients - that would need the entry point to use a signature like this one, it would be - straightforward to implement it in a future amendment to this proposal. - -## Acknowledgments - -Thanks much to [Dennis Weissmann](https://github.com/dennisweissmann) for his -tireless work in this area and to [Paul LeMarquand](https://github.com/plemarquand) -for putting up with my incessant revisions and nitpicking while he worked on -VS Code's Swift Testing support. +> This proposal was accepted before Swift Testing began using the Swift +> evolution review process. Its original identifier was SWT-0002 but its prefix +> has been changed to "ST" and it has been relocated to the +> [swift-evolution](https://github.com/swiftlang/swift-evolution) repository. -Thanks to the rest of the Swift Testing team for reviewing this proposal and the -JSON schema and to the community for embracing Swift Testing! +To view this proposal, see +[ST-0002: A stable JSON-based ABI for tools integration](https://github.com/swiftlang/swift-evolution/blob/main/proposals/testing/0002-json-abi.md). diff --git a/Documentation/Proposals/0003-make-serialized-trait-api.md b/Documentation/Proposals/0003-make-serialized-trait-api.md index dc2a98c28..4d96ece70 100644 --- a/Documentation/Proposals/0003-make-serialized-trait-api.md +++ b/Documentation/Proposals/0003-make-serialized-trait-api.md @@ -1,151 +1,10 @@ # Make .serialized trait API -* Proposal: [SWT-0003](0003-make-serialized-trait-api.md) -* Authors: [Dennis Weissmann](https://github.com/dennisweissmann) -* Status: **Implemented (Swift 6.0)** -* Implementation: -[swiftlang/swift-testing#535](https://github.com/swiftlang/swift-testing/pull/535) -* Review: -([pitch](https://forums.swift.org/t/pitch-make-serialized-trait-public-api/73147)), -([acceptance](https://forums.swift.org/t/pitch-make-serialized-trait-public-api/73147/5)) - -## Introduction - -We propose promoting the existing `.serialized` trait to public API. This trait -enables developers to designate tests or test suites to run serially, ensuring -sequential execution where necessary. - -## Motivation - -The Swift Testing library defaults to parallel execution of tests, promoting -efficiency and isolation. However, certain test scenarios demand strict -sequential execution due to shared state or complex dependencies between tests. -The `.serialized` trait provides a solution by allowing developers to enforce -serial execution for specific tests or suites. - -While global actors ensure that only one task associated with that actor runs -at any given time, thus preventing concurrent access to actor state, tasks can -yield and allow other tasks to proceed, potentially interleaving execution. -That means global actors do not ensure that a specific test runs entirely to -completion before another begins. A testing library requires a construct that -guarantees that each annotated test runs independently and completely (in its -suite), one after another, without interleaving. - -## Proposed Solution - -We propose exposing the `.serialized` trait as a public API. This attribute can -be applied to individual test functions or entire test suites, modifying the -test execution behavior to enforce sequential execution where specified. - -Annotating just a single test in a suite does not enforce any serialization -behavior - the testing library encourages parallelization and the bar to -degrade overall performance of test execution should be high. -Additionally, traits apply inwards - it would be unexpected to impact the exact -conditions of a another test in a suite without applying a trait to the suite -itself. -Thus, this trait should only be applied to suites (to enforce serial execution -of all tests inside it) or parameterized tests. If applied to just a test this -trait does not have any effect. - -## Detailed Design - -The `.serialized` trait functions as an attribute that alters the execution -scheduling of tests. When applied, it ensures that tests or suites annotated -with `.serialized` run serially. - -```swift -/// A type that affects whether or not a test or suite is parallelized. -/// -/// When added to a parameterized test function, this trait causes that test to -/// run its cases serially instead of in parallel. When applied to a -/// non-parameterized test function, this trait has no effect. When applied to a -/// test suite, this trait causes that suite to run its contained test functions -/// and sub-suites serially instead of in parallel. -/// -/// This trait is recursively applied: if it is applied to a suite, any -/// parameterized tests or test suites contained in that suite are also -/// serialized (as are any tests contained in those suites, and so on.) -/// -/// This trait does not affect the execution of a test relative to its peers or -/// to unrelated tests. This trait has no effect if test parallelization is -/// globally disabled (by, for example, passing `--no-parallel` to the -/// `swift test` command.) -/// -/// To add this trait to a test, use ``Trait/serialized``. -public struct ParallelizationTrait: TestTrait, SuiteTrait {} - -extension Trait where Self == ParallelizationTrait { - /// A trait that serializes the test to which it is applied. - /// - /// ## See Also - /// - /// - ``ParallelizationTrait`` - public static var serialized: Self { get } -} -``` - -The call site looks like this: - -```swift -@Test(.serialized, arguments: Food.allCases) func prepare(food: Food) { - // This function will be invoked serially, once per food, because it has the - // .serialized trait. -} - -@Suite(.serialized) struct FoodTruckTests { - @Test(arguments: Condiment.allCases) func refill(condiment: Condiment) { - // This function will be invoked serially, once per condiment, because the - // containing suite has the .serialized trait. - } - - @Test func startEngine() async throws { - // This function will not run while refill(condiment:) is running. One test - // must end before the other will start. - } -} - -@Suite struct FoodTruckTests { - @Test(.serialized) func startEngine() async throws { - // This function will not run serially - it's not a parameterized test and - // the suite is not annotated with the `.serialized` trait. - } - - @Test func prepareFood() async throws { - // It doesn't matter if this test is `.serialized` or not, traits applied - // to other tests won't affect this test don't impact other tests. - } -} -``` - -## Source Compatibility - -Introducing `.serialized` as a public API does not have any impact on existing -code. Tests will continue to run in parallel by default unless explicitly -marked with `.serialized`. - -## Integration with Supporting Tools - -N/A. - -## Future Directions - -There might be asks for more advanced and complex ways to affect parallelization -which include ways to specify dependencies between tests ie. "Require `foo()` to -run before `bar()`". - -## Alternatives Considered - -Alternative approaches, such as relying solely on global actors for test -isolation, were considered. However, global actors do not provide the -deterministic, sequential execution required for certain testing scenarios. The -`.serialized` trait offers a more explicit and flexible mechanism, ensuring -that each designated test or suite runs to completion without interruption. - -Various more complex parallelization and serialization options were discussed -and considered but ultimately disregarded in favor of this simple yet powerful -implementation. - -## Acknowledgments - -Thanks to the swift-testing team and managers for their contributions! Thanks -to our community for the initial feedback around this feature. +> [!NOTE] +> This proposal was accepted before Swift Testing began using the Swift +> evolution review process. Its original identifier was SWT-0003 but its prefix +> has been changed to "ST" and it has been relocated to the +> [swift-evolution](https://github.com/swiftlang/swift-evolution) repository. + +To view this proposal, see +[ST-0003: Make .serialized trait API](https://github.com/swiftlang/swift-evolution/blob/main/proposals/testing/0003-make-serialized-trait-api.md). diff --git a/Documentation/Proposals/0004-constrain-the-granularity-of-test-time-limit-durations.md b/Documentation/Proposals/0004-constrain-the-granularity-of-test-time-limit-durations.md index 8818dba71..8bee19894 100644 --- a/Documentation/Proposals/0004-constrain-the-granularity-of-test-time-limit-durations.md +++ b/Documentation/Proposals/0004-constrain-the-granularity-of-test-time-limit-durations.md @@ -1,203 +1,10 @@ # Constrain the granularity of test time limit durations -* Proposal: -[SWT-0004](0004-constrain-the-granularity-of-test-time-limit-durations.md) -* Authors: [Dennis Weissmann](https://github.com/dennisweissmann) -* Status: **Implemented (Swift 6.0)** -* Implementation: -[swiftlang/swift-testing#534](https://github.com/swiftlang/swift-testing/pull/534) -* Review: -([pitch](https://forums.swift.org/t/pitch-constrain-the-granularity-of-test-time-limit-durations/73146)), -([acceptance](https://forums.swift.org/t/pitch-constrain-the-granularity-of-test-time-limit-durations/73146/3)) - -## Introduction - -Sometimes tests might get into a state (either due the test code itself or due -to the code they're testing) where they don't make forward progress and hang. -Swift Testing provides a way to handle these issues using the TimeLimit trait: - -```swift -@Test(.timeLimit(.minutes(60)) -func testFunction() { ... } -``` - -Currently there exist multiple overloads for the `.timeLimit` trait: one that -takes a `Swift.Duration` which allows for arbitrary `Duration` values to be -passed, and one that takes a `TimeLimitTrait.Duration` which constrains the -minimum time limit as well as the increment to 1 minute. - -## Motivation - -Small time limit values in particular cause more harm than good due to tests -running in environments with drastically differing performance characteristics. -Particularly when running in CI systems or on virtualized hardware tests can -run much slower than at desk. -Swift Testing should help developers use a reasonable time limit value in its -API without developers having to refer to the documentation. - -It is crucial to emphasize that unit tests failing due to exceeding their -timeout should be exceptionally rare. At the same time, a spurious unit test -failure caused by a short timeout can be surprisingly costly, potentially -leading to an entire CI pipeline being rerun. Determining an appropriate -timeout for a specific test can be a challenging task. - -Additionally, when the system intentionally runs multiple tests simultaneously -to optimize resource utilization, the scheduler becomes the arbiter of test -execution. Consequently, the test may take significantly longer than -anticipated, potentially due to external factors beyond the control of the code -under test. - -A unit test should be capable of failing due to hanging, but it should not fail -due to being slow, unless the developer has explicitly indicated that it -should, effectively transforming it into a performance test. - -The time limit feature is *not* intended to be used to apply small timeouts to -tests to ensure test runtime doesn't regress by small amounts. This feature is -intended to be used to guard against hangs and pathologically long running -tests. - -## Proposed Solution - -We propose changing the `.timeLimit` API to accept values of a new `Duration` -type defined in `TimeLimitTrait` which only allows for `.minute` values to be -passed. -This type already exists as SPI and this proposal is seeking to making it API. - -## Detailed Design - -The `TimeLimitTrait.Duration` struct only has one factory method: -```swift -public static func minutes(_ minutes: some BinaryInteger) -> Self -``` - -That ensures 2 things: -1. It's impossible to create short time limits (under a minute). -2. It's impossible to create high-precision increments of time. - -Both of these features are important to ensure the API is self documenting and -conveying the intended purpose. - -For parameterized tests these time limits apply to each individual test case. - -The `TimeLimitTrait.Duration` struct is declared as follows: - -```swift -/// A type that defines a time limit to apply to a test. -/// -/// To add this trait to a test, use one of the following functions: -/// -/// - ``Trait/timeLimit(_:)`` -@available(macOS 13.0, iOS 16.0, tvOS 16.0, watchOS 9.0, *) -public struct TimeLimitTrait: TestTrait, SuiteTrait { - /// A type representing the duration of a time limit applied to a test. - /// - /// This type is intended for use specifically for specifying test timeouts - /// with ``TimeLimitTrait``. It is used instead of Swift's built-in `Duration` - /// type because test timeouts do not support high-precision, arbitrarily - /// short durations. The smallest allowed unit of time is minutes. - public struct Duration: Sendable { - - /// Construct a time limit duration given a number of minutes. - /// - /// - Parameters: - /// - minutes: The number of minutes the resulting duration should - /// represent. - /// - /// - Returns: A duration representing the specified number of minutes. - public static func minutes(_ minutes: some BinaryInteger) -> Self - } - - /// The maximum amount of time a test may run for before timing out. - public var timeLimit: Swift.Duration { get set } -} -``` - -The extension on `Trait` that allows for `.timeLimit(...)` to work is defined -like this: - -```swift -/// Construct a time limit trait that causes a test to time out if it runs for -/// too long. -/// -/// - Parameters: -/// - timeLimit: The maximum amount of time the test may run for. -/// -/// - Returns: An instance of ``TimeLimitTrait``. -/// -/// Test timeouts do not support high-precision, arbitrarily short durations -/// due to variability in testing environments. The time limit must be at -/// least one minute, and can only be expressed in increments of one minute. -/// -/// When this trait is associated with a test, that test must complete within -/// a time limit of, at most, `timeLimit`. If the test runs longer, an issue -/// of kind ``Issue/Kind/timeLimitExceeded(timeLimitComponents:)`` is -/// recorded. This timeout is treated as a test failure. -/// -/// The time limit amount specified by `timeLimit` may be reduced if the -/// testing library is configured to enforce a maximum per-test limit. When -/// such a maximum is set, the effective time limit of the test this trait is -/// applied to will be the lesser of `timeLimit` and that maximum. This is a -/// policy which may be configured on a global basis by the tool responsible -/// for launching the test process. Refer to that tool's documentation for -/// more details. -/// -/// If a test is parameterized, this time limit is applied to each of its -/// test cases individually. If a test has more than one time limit associated -/// with it, the shortest one is used. A test run may also be configured with -/// a maximum time limit per test case. -public static func timeLimit(_ timeLimit: Self.Duration) -> Self -``` - -And finally, the call site of the API looks like this: - -```swift -@Test(.timeLimit(.minutes(60)) -func serve100CustomersInOneHour() async { - for _ in 0 ..< 100 { - let customer = await Customer.next() - await customer.order() - ... - } -} -``` - -The `TimeLimitTrait.Duration` struct has various `unavailable` overloads that -are included for diagnostic purposes only. They are all documented and -annotated like this: - -```swift -/// Construct a time limit duration given a number of . -/// -/// This function is unavailable and is provided for diagnostic purposes only. -@available(*, unavailable, message: "Time limit must be specified in minutes") -``` - -## Source Compatibility - -This impacts clients that have adopted the `.timeLimit` trait and use overloads -of the trait that accept an arbitrary `Swift.Duration` except if they used the -`minutes` overload. - -## Integration with Supporting Tools - -N/A - -## Future Directions - -We could allow more finegrained time limits in the future that scale with the -performance of the test host device. -Or take a more manual approach where we detect the type of environment -(like CI vs local) and provide a way to use different timeouts depending on the -environment. - -## Alternatives Considered - -We have considered using `Swift.Duration` as the currency type for this API but -decided against it to avoid common pitfalls and misuses of this feature such as -providing very small time limits that lead to flaky tests in different -environments. - -## Acknowledgments - -The authors acknowledge valuable contributions and feedback from the Swift -Testing community during the development of this proposal. +> [!NOTE] +> This proposal was accepted before Swift Testing began using the Swift +> evolution review process. Its original identifier was SWT-0004 but its prefix +> has been changed to "ST" and it has been relocated to the +> [swift-evolution](https://github.com/swiftlang/swift-evolution) repository. + +To view this proposal, see +[ST-0004: Constrain the granularity of test time limit durations](https://github.com/swiftlang/swift-evolution/blob/main/proposals/testing/0004-constrain-the-granularity-of-test-time-limit-durations.md). diff --git a/Documentation/Proposals/0005-ranged-confirmations.md b/Documentation/Proposals/0005-ranged-confirmations.md index df1db331b..8777d6009 100644 --- a/Documentation/Proposals/0005-ranged-confirmations.md +++ b/Documentation/Proposals/0005-ranged-confirmations.md @@ -1,186 +1,10 @@ # Range-based confirmations -* Proposal: [SWT-0005](0005-ranged-confirmations.md) -* Authors: [Jonathan Grynspan](https://github.com/grynspan) -* Status: **Implemented (Swift 6.1)** -* Bug: rdar://138499457 -* Implementation: [swiftlang/swift-testing#598](https://github.com/swiftlang/swift-testing/pull/598), [swiftlang/swift-testing#689](https://github.com/swiftlang/swift-testing/pull689) -* Review: ([pitch](https://forums.swift.org/t/pitch-range-based-confirmations/74589)), - ([acceptance](https://forums.swift.org/t/pitch-range-based-confirmations/74589/7)) - -## Introduction - -Swift Testing includes [an interface](https://swiftpackageindex.com/swiftlang/swift-testing/main/documentation/testing/confirmation(_:expectedcount:isolation:sourcelocation:_:)) -for checking that some asynchronous event occurs a given number of times -(typically exactly once or never at all.) This proposal enhances that interface -to allow arbitrary ranges of event counts so that a test can be written against -code that may not always fire said event the exact same number of times. - -## Motivation - -Some tests rely on fixtures or external state that is not perfectly -deterministic. For example, consider a test that checks that clicking the mouse -button will generate a `.mouseClicked` event. Such a test might use the -`confirmation()` interface: - -```swift -await confirmation(expectedCount: 1) { mouseClicked in - var eventLoop = EventLoop() - eventLoop.eventHandler = { event in - if event == .mouseClicked { - mouseClicked() - } - } - await eventLoop.simulate(.mouseClicked) -} -``` - -But what happens if the user _actually_ clicks a mouse button while this test is -running? That might trigger a _second_ `.mouseClicked` event, and then the test -will fail spuriously. - -## Proposed solution - -If the test author could instead indicate to Swift Testing that their test will -generate _one or more_ events, they could avoid spurious failures: - -```swift -await confirmation(expectedCount: 1...) { mouseClicked in - ... -} -``` - -With this proposal, we add an overload of `confirmation()` that takes any range -expression instead of a single integer value (which is still accepted via the -existing overload.) - -## Detailed design - -A new overload of `confirmation()` is added: - -```swift -/// Confirm that some event occurs during the invocation of a function. -/// -/// - Parameters: -/// - comment: An optional comment to apply to any issues generated by this -/// function. -/// - expectedCount: A range of integers indicating the number of times the -/// expected event should occur when `body` is invoked. -/// - isolation: The actor to which `body` is isolated, if any. -/// - sourceLocation: The source location to which any recorded issues should -/// be attributed. -/// - body: The function to invoke. -/// -/// - Returns: Whatever is returned by `body`. -/// -/// - Throws: Whatever is thrown by `body`. -/// -/// Use confirmations to check that an event occurs while a test is running in -/// complex scenarios where `#expect()` and `#require()` are insufficient. For -/// example, a confirmation may be useful when an expected event occurs: -/// -/// - In a context that cannot be awaited by the calling function such as an -/// event handler or delegate callback; -/// - More than once, or never; or -/// - As a callback that is invoked as part of a larger operation. -/// -/// To use a confirmation, pass a closure containing the work to be performed. -/// The testing library will then pass an instance of ``Confirmation`` to the -/// closure. Every time the event in question occurs, the closure should call -/// the confirmation: -/// -/// ```swift -/// let minBuns = 5 -/// let maxBuns = 10 -/// await confirmation( -/// "Baked between \(minBuns) and \(maxBuns) buns", -/// expectedCount: minBuns ... maxBuns -/// ) { bunBaked in -/// foodTruck.eventHandler = { event in -/// if event == .baked(.cinnamonBun) { -/// bunBaked() -/// } -/// } -/// await foodTruck.bakeTray(of: .cinnamonBun) -/// } -/// ``` -/// -/// When the closure returns, the testing library checks if the confirmation's -/// preconditions have been met, and records an issue if they have not. -/// -/// If an exact count is expected, use -/// ``confirmation(_:expectedCount:isolation:sourceLocation:_:)`` instead. -public func confirmation( - _ comment: Comment? = nil, - expectedCount: some RangeExpression & Sequence Sendable, - isolation: isolated (any Actor)? = #isolation, - sourceLocation: SourceLocation = #_sourceLocation, - _ body: (Confirmation) async throws -> sending R -) async rethrows -> R -``` - -### Ranges without lower bounds - -Certain types of range, specifically [`PartialRangeUpTo`](https://developer.apple.com/documentation/swift/partialrangeupto) -and [`PartialRangeThrough`](https://developer.apple.com/documentation/swift/partialrangethrough), -may have surprising behavior when used with this new interface because they -implicitly include `0`. If a test author writes `...10`, do they mean "zero to -ten" or "one to ten"? The programmatic meaning is the former, but some test -authors might mean the latter. If an event does not occur, a test using -`confirmation()` and this `expectedCount` value would pass when the test author -meant for it to fail. - -The unbounded range (`...`) type `UnboundedRange` is effectively useless when -used with this interface and any use of it here is almost certainly a programmer -error. - -`PartialRangeUpTo` and `PartialRangeThrough` conform to `RangeExpression`, but -not to `Sequence`, so they will be rejected at compile time. `UnboundedRange` is -a non-nominal type and will not match either. We will provide unavailable -overloads of `confirmation()` for these types with messages that explain why -they are unavailable, e.g.: - -```swift -@available(*, unavailable, message: "Unbounded range '...' has no effect when used with a confirmation.") -public func confirmation( - _ comment: Comment? = nil, - expectedCount: UnboundedRange, - isolation: isolated (any Actor)? = #isolation, - sourceLocation: SourceLocation = #_sourceLocation, - _ body: (Confirmation) async throws -> R -) async rethrows -> R -``` - -## Source compatibility - -This change is additive. Existing tests are unaffected. - -Code that refers to `confirmation(_:expectedCount:isolation:sourceLocation:_:)` -by symbol name may need to add a contextual type to disambiguate the two -overloads at compile time. - -## Integration with supporting tools - -The type of the associated value `expected` for the `Issue.Kind` case -`confirmationMiscounted(actual:expected:)` will change from `Int` to -`any RangeExpression & Sendable`[^1]. Tools that implement event handlers and -distinguish between `Issue.Kind` cases are advised not to assume the type of -this value is `Int`. - -## Alternatives considered - -- Doing nothing. We have identified real-world use cases for this interface - including in Swift Testing’s own test target. -- Allowing the use of any value as the `expectedCount` argument so long as it - conforms to a protocol `ExpectedCount` (we'd have range types and `Int` - conform by default.) It was unclear what this sort of flexibility would let - us do, and posed challenges for encoding and decoding events and issues when - using the JSON event stream interface. - -## Acknowledgments - -Thanks to the testing team for their help preparing this pitch! - -[^1]: In the future, this type will change to - `any RangeExpression & Sendable`. Compiler support is required - ([96960993](rdar://96960993)). +> [!NOTE] +> This proposal was accepted before Swift Testing began using the Swift +> evolution review process. Its original identifier was SWT-0005 but its prefix +> has been changed to "ST" and it has been relocated to the +> [swift-evolution](https://github.com/swiftlang/swift-evolution) repository. + +To view this proposal, see +[ST-0005: Range-based confirmations](https://github.com/swiftlang/swift-evolution/blob/main/proposals/testing/0005-ranged-confirmations.md). diff --git a/Documentation/Proposals/0006-return-errors-from-expect-throws.md b/Documentation/Proposals/0006-return-errors-from-expect-throws.md index 502a18e17..7fe1f6b5c 100644 --- a/Documentation/Proposals/0006-return-errors-from-expect-throws.md +++ b/Documentation/Proposals/0006-return-errors-from-expect-throws.md @@ -1,267 +1,10 @@ # Return errors from `#expect(throws:)` -* Proposal: [SWT-0006](0006-return-errors-from-expect-throws.md) -* Authors: [Jonathan Grynspan](https://github.com/grynspan) -* Status: **Implemented (Swift 6.1)** -* Bug: rdar://138235250 -* Implementation: [swiftlang/swift-testing#780](https://github.com/swiftlang/swift-testing/pull/780) -* Review: ([pitch](https://forums.swift.org/t/pitch-returning-errors-from-expect-throws/75567)), ([acceptance](https://forums.swift.org/t/pitch-returning-errors-from-expect-throws/75567/5)) - -## Introduction - -Swift Testing includes overloads of `#expect()` and `#require()` that can be -used to assert that some code throws an error. They are useful when validating -that your code's failure cases are correctly detected and handled. However, for -more complex validation cases, they aren't particularly ergonomic. This proposal -seeks to resolve that issue by having these overloads return thrown errors for -further inspection. - -## Motivation - -We offer three variants of `#expect(throws:)`: - -- One that takes an error type, and matches any error of the same type; -- One that takes an error _instance_ (conforming to `Equatable`) and matches any - error that compares equal to it; and -- One that takes a trailing closure and allows test authors to write arbitrary - validation logic. - -The third overload has proven to be somewhat problematic. First, it yields the -error to its closure as an instance of `any Error`, which typically forces the -developer to cast it before doing any useful comparisons. Second, the test -author must return `true` to indicate the error matched and `false` to indicate -it didn't, which can be both logically confusing and difficult to express -concisely: - -```swift -try #require { - let potato = try Sack.randomPotato() - try potato.turnIntoFrenchFries() -} throws: { error in - guard let error = error as PotatoError else { - return false - } - guard case .potatoNotPeeled = error else { - return false - } - return error.variety != .russet -} -``` - -The first impulse many test authors have here is to use `#expect()` in the -second closure, but it doesn't return the necessary boolean value _and_ it can -result in multiple issues being recorded in a test when there's really only one. - -## Proposed solution - -I propose deprecating [`#expect(_:sourceLocation:performing:throws:)`](https://developer.apple.com/documentation/testing/expect(_:sourcelocation:performing:throws:)) -and [`#require(_:sourceLocation:performing:throws:)`](https://developer.apple.com/documentation/testing/require(_:sourcelocation:performing:throws:)) -and modifying the other overloads so that, on success, they return the errors -that were thrown. - -## Detailed design - -All overloads of `#expect(throws:)` and `#require(throws:)` will be updated to -return an instance of the error type specified by their arguments, with the -problematic overloads returning `any Error` since more precise type information -is not statically available. The problematic overloads will also be deprecated: - -```diff ---- a/Sources/Testing/Expectations/Expectation+Macro.swift -+++ b/Sources/Testing/Expectations/Expectation+Macro.swift -+@discardableResult - @freestanding(expression) public macro expect( - throws errorType: E.Type, - _ comment: @autoclosure () -> Comment? = nil, - sourceLocation: SourceLocation = #_sourceLocation, - performing expression: () async throws -> R --) -+) -> E? where E: Error - -+@discardableResult - @freestanding(expression) public macro require( - throws errorType: E.Type, - _ comment: @autoclosure () -> Comment? = nil, - sourceLocation: SourceLocation = #_sourceLocation, - performing expression: () async throws -> R --) where E: Error -+) -> E where E: Error - -+@discardableResult - @freestanding(expression) public macro expect( - throws error: E, - _ comment: @autoclosure () -> Comment? = nil, - sourceLocation: SourceLocation = #_sourceLocation, - performing expression: () async throws -> R --) where E: Error & Equatable -+) -> E? where E: Error & Equatable - -+@discardableResult - @freestanding(expression) public macro require( - throws error: E, - _ comment: @autoclosure () -> Comment? = nil, - sourceLocation: SourceLocation = #_sourceLocation, - performing expression: () async throws -> R --) where E: Error & Equatable -+) -> E where E: Error & Equatable - -+@available(swift, deprecated: 100000.0, message: "Examine the result of '#expect(throws:)' instead.") -+@discardableResult - @freestanding(expression) public macro expect( - _ comment: @autoclosure () -> Comment? = nil, - sourceLocation: SourceLocation = #_sourceLocation, - performing expression: () async throws -> R, - throws errorMatcher: (any Error) async throws -> Bool --) -+) -> (any Error)? - -+@available(swift, deprecated: 100000.0, message: "Examine the result of '#require(throws:)' instead.") -+@discardableResult - @freestanding(expression) public macro require( - _ comment: @autoclosure () -> Comment? = nil, - sourceLocation: SourceLocation = #_sourceLocation, - performing expression: () async throws -> R, - throws errorMatcher: (any Error) async throws -> Bool --) -+) -> any Error -``` - -(More detailed information about the deprecations will be provided via DocC.) - -The `#expect(throws:)` overloads return an optional value that is `nil` if the -expectation failed, while the `#require(throws:)` overloads return non-optional -values and throw instances of `ExpectationFailedError` on failure (as before.) - > [!NOTE] -> Instances of `ExpectationFailedError` thrown by `#require(throws:)` on failure -> are not returned as that would defeat the purpose of using `#require(throws:)` -> instead of `#expect(throws:)`. - -Test authors will be able to use the result of the above functions to verify -that the thrown error is correct: - -```swift -let error = try #require(throws: PotatoError.self) { - let potato = try Sack.randomPotato() - try potato.turnIntoFrenchFries() -} -#expect(error == .potatoNotPeeled) -#expect(error.variety != .russet) -``` - -The new code is more concise than the old code and avoids boilerplate casting -from `any Error`. - -## Source compatibility - -In most cases, this change does not affect source compatibility. Swift does not -allow forming references to macros at runtime, so we don't need to worry about -type mismatches assigning one to some local variable. - -We have identified two scenarios where a new warning will be emitted. - -### Inferred return type from macro invocation - -The return type of the macro may be used by the compiler to infer the return -type of an enclosing closure. If the return value is then discarded, the -compiler may emit a warning: - -```swift -func pokePotato(_ pPotato: UnsafePointer) throws { ... } - -let potato = Potato() -try await Task.sleep(for: .months(3)) -withUnsafePointer(to: potato) { pPotato in - // ^ ^ ^ ⚠️ Result of call to 'withUnsafePointer(to:_:)' is unused - #expect(throws: PotatoError.rotten) { - try pokePotato(pPotato) - } -} -``` - -This warning can be suppressed by assigning the result of the macro invocation -or the result of the function call to `_`: - -```swift -withUnsafePointer(to: potato) { pPotato in - _ = #expect(throws: PotatoError.rotten) { - try pokePotato(pPotato) - } -} -``` - -### Use of `#require(throws:)` in a generic context with `Never.self` - -If `#require(throws:)` (but not `#expect(throws:)`) is used in a generic context -where the type of thrown error is a generic parameter, and the type is resolved -to `Never`, there is no valid value for the invocation to return: - -```swift -func wrapper(throws type: E.Type, _ body: () throws -> Void) throws -> E { - return try #require(throws: type) { - try body() - } -} -let error = try #require(throws: Never.self) { ... } -``` - -We don't think this particular pattern is common (and outside of our own test -target, I'd be surprised if anybody's attempted it yet.) However, we do need to -handle it gracefully. If this pattern is encountered, Swift Testing will record -an "API Misused" issue for the current test and advise the test author to switch -to `#expect(throws:)` or to not pass `Never.self` here. - -## Integration with supporting tools - -N/A - -## Future directions - -- Adopting [typed throws](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0413-typed-throws.md) - to statically require that the error thrown from test code is of the correct - type. - - If we adopted typed throws in the signatures of these macros, it would force - adoption of typed throws in the code under test even when it may not be - appropriate. For example, if we adopted typed throws, the following code would - not compile: - - ```swift - func cook(_ food: consuming some Food) throws { ... } - - let error: PotatoError? = #expect(throws: PotatoError.self) { - var potato = Potato() - potato.fossilize() - try cook(potato) // 🛑 ERROR: Invalid conversion of thrown error type - // 'any Error' to 'PotatoError' - } - ``` - - We believe it may be possible to overload these macros or their expansions so - that the code sample above _does_ compile and behave as intended. We intend to - experiment further with this idea and potentially revisit typed throws support - in a future proposal. - -## Alternatives considered - -- Leaving the existing implementation and signatures in place. We've had - sufficient feedback about the ergonomics of this API that we want to address - the problem. - -- Having the return type of the macros be `any Error` and returning _any_ error - that was thrown even on mismatch. This would make the ergonomics of the - subsequent test code less optimal because the test author would need to cast - the error to the appropriate type before inspecting it. - - There's a philosophical argument to be made here that if a mismatched error is - thrown, then the test has already failed and is in an inconsistent state, so - we should allow the test to fail rather than return what amounts to "bad - output". - - If the test author wants to inspect any arbitrary thrown error, they can - specify `(any Error).self` instead of a concrete error type. - -## Acknowledgments +> This proposal was accepted before Swift Testing began using the Swift +> evolution review process. Its original identifier was SWT-0006 but its prefix +> has been changed to "ST" and it has been relocated to the +> [swift-evolution](https://github.com/swiftlang/swift-evolution) repository. -Thanks to the team and to [@jakepetroules](https://github.com/jakepetroules) for -starting the discussion that ultimately led to this proposal. +To view this proposal, see +[ST-0006: Return errors from `#expect(throws:)`](https://github.com/swiftlang/swift-evolution/blob/main/proposals/testing/0006-return-errors-from-expect-throws.md). diff --git a/Documentation/Proposals/0007-test-scoping-traits.md b/Documentation/Proposals/0007-test-scoping-traits.md index 9d01bbbc7..969f163fc 100644 --- a/Documentation/Proposals/0007-test-scoping-traits.md +++ b/Documentation/Proposals/0007-test-scoping-traits.md @@ -1,510 +1,10 @@ # Test Scoping Traits -* Proposal: [SWT-0007](0007-test-scoping-traits.md) -* Authors: [Stuart Montgomery](https://github.com/stmontgomery) -* Status: **Implemented (Swift 6.1)** -* Implementation: [swiftlang/swift-testing#733](https://github.com/swiftlang/swift-testing/pull/733), [swiftlang/swift-testing#86](https://github.com/swiftlang/swift-testing/pull/86) -* Review: ([pitch](https://forums.swift.org/t/pitch-custom-test-execution-traits/75055)), ([review](https://forums.swift.org/t/proposal-test-scoping-traits/76676)), ([acceptance](https://forums.swift.org/t/proposal-test-scoping-traits/76676/3)) - -### Revision history - -* **v1**: Initial pitch. -* **v2**: Dropped 'Custom' prefix from the proposed API names (although kept the - word in certain documentation passages where it clarified behavior). -* **v3**: Changed the `Trait` requirement from a property to a method which - accepts the test and/or test case, and modify its default implementations such - that custom behavior is either performed per-suite or per-test case by default. -* **v4**: Renamed the APIs to use "scope" as the base verb instead of "execute". - -## Introduction - -This introduces API which enables a `Trait`-conforming type to provide a custom -execution scope for test functions and suites, including running code before or -after them. - -## Motivation - -One of the primary motivations for the trait system in Swift Testing, as -[described in the vision document](https://github.com/swiftlang/swift-evolution/blob/main/visions/swift-testing.md#trait-extensibility), -is to provide a way to customize the behavior of tests which have things in -common. If all the tests in a given suite type need the same custom behavior, -`init` and/or `deinit` (if applicable) can be used today. But if only _some_ of -the tests in a suite need custom behavior, or tests across different levels of -the suite hierarchy need it, traits would be a good place to encapsulate common -logic since they can be applied granularly per-test or per-suite. This aspect of -the vision for traits hasn't been realized yet, though: the `Trait` protocol -does not offer a way for a trait to customize the execution of the tests or -suites it's applied to. - -Customizing a test's behavior typically means running code either before or -after it runs, or both. Consolidating common set-up and tear-down logic allows -each test function to be more succinct with less repetitive boilerplate so it -can focus on what makes it unique. - -## Proposed solution - -At a high level, this proposal entails adding API to the `Trait` protocol -allowing a conforming type to opt-in to providing a custom execution scope for a -test. We discuss how that capability should be exposed to trait types below. - -### Supporting scoped access - -There are different approaches one could take to expose hooks for a trait to -customize test behavior. To illustrate one of them, consider the following -example of a `@Test` function with a custom trait whose purpose is to set mock -API credentials for the duration of each test it's applied to: - -```swift -@Test(.mockAPICredentials) -func example() { - // ... -} - -struct MockAPICredentialsTrait: TestTrait { ... } - -extension Trait where Self == MockAPICredentialsTrait { - static var mockAPICredentials: Self { ... } -} -``` - -In this hypothetical example, the current API credentials are stored via a -static property on an `APICredentials` type which is part of the module being -tested: - -```swift -struct APICredentials { - var apiKey: String - - static var shared: Self? -} -``` - -One way that this custom trait could customize the API credentials during each -test is if the `Trait` protocol were to expose a pair of method requirements -which were then called before and after the test, respectively: - -```swift -public protocol Trait: Sendable { - // ... - func setUp() async throws - func tearDown() async throws -} - -extension Trait { - // ... - public func setUp() async throws { /* No-op */ } - public func tearDown() async throws { /* No-op */ } -} -``` - -The custom trait type could adopt these using code such as the following: - -```swift -extension MockAPICredentialsTrait { - func setUp() { - APICredentials.shared = .init(apiKey: "...") - } - - func tearDown() { - APICredentials.shared = nil - } -} -``` - -Many testing systems use this pattern, including XCTest. However, this approach -encourages the use of global mutable state such as the `APICredentials.shared` -variable, and this limits the testing library's ability to parallelize test -execution, which is -[another part of the Swift Testing vision](https://github.com/swiftlang/swift-evolution/blob/main/visions/swift-testing.md#parallelization-and-concurrency). - -The use of nonisolated static variables is generally discouraged now, and in -Swift 6 the above `APICredentials.shared` property produces an error. One way -to resolve that is to change it to a `@TaskLocal` variable, as this would be -concurrency-safe and still allow tests accessing this state to run in parallel: - -```swift -extension APICredentials { - @TaskLocal static var current: Self? -} -``` - -Binding task local values requires using the scoped access -[`TaskLocal.withValue()`](https://developer.apple.com/documentation/swift/tasklocal/withvalue(_:operation:isolation:file:line:)) -API though, and that would not be possible if `Trait` exposed separate methods -like `setUp()` and `tearDown()`. - -For these reasons, I believe it's important to expose this trait capability -using a single, scoped access-style API which accepts a closure. A simplified -version of that idea might look like this: - -```swift -public protocol Trait: Sendable { - // ... - - // Simplified example, not the actual proposal - func executeTest(_ body: @Sendable () async throws -> Void) async throws -} - -extension MockAPICredentialsTrait { - func executeTest(_ body: @Sendable () async throws -> Void) async throws { - let mockCredentials = APICredentials(apiKey: "...") - try await APICredentials.$current.withValue(mockCredentials) { - try await body() - } - } -} -``` - -### Avoiding unnecessarily lengthy backtraces - -A scoped access-style API has some potential downsides. To apply this approach -to a test function, the scoped call of a trait must wrap the invocation of that -test function, and every _other_ trait applied to that same test which offers -custom behavior _also_ must wrap the other traits' calls in a nesting fashion. -To visualize this, imagine a test function with multiple traits: - -```swift -@Test(.traitA, .traitB, .traitC) -func exampleTest() { - // ... -} -``` - -If all three of those traits provide a custom scope for tests, then each of them -needs to wrap the call to the next one, and the last trait needs to wrap the -invocation of the test, illustrated by the following: - -``` -TraitA.executeTest { - TraitB.executeTest { - TraitC.executeTest { - exampleTest() - } - } -} -``` - -Tests may have an arbitrary number of traits applied to them, including those -inherited from containing suite types. A naïve implementation in which _every_ -trait is given the opportunity to customize test behavior by calling its scoped -access API might cause unnecessarily lengthy backtraces that make debugging the -body of tests more difficult. Or worse: if the number of traits is great enough, -it could cause a stack overflow. - -In practice, most traits probably do _not_ need to provide a custom scope for -the tests they're applied to, so to mitigate these downsides it's important that -there be some way to distinguish traits which customize test behavior. That way, -the testing library can limit these scoped access calls to only traits which -need it. - -### Avoiding unnecessary (re-)execution - -Traits can be applied to either test functions or suites, and traits applied to -suites can optionally support inheritance by implementing the `isRecursive` -property of the `SuiteTrait` protocol. When a trait is directly applied to a -test function, if the trait customizes the behavior of tests it's applied to, it -should be given the opportunity to perform its custom behavior once for every -invocation of that test function. In particular, if the test function is -parameterized and runs multiple times, then the trait applied to it should -perform its custom behavior once for every invocation. This should not be -surprising to users, since it's consistent with the behavior of `init` and -`deinit` for an instance `@Test` method. - -It may be useful for certain kinds of traits to perform custom logic once for -_all_ the invocations of a parameterized test. Although this should be possible, -we believe it shouldn't be the default since it could lead to work being -repeated multiple times needlessly, or unintentional state sharing across tests, -unless the trait is implemented carefully to avoid those problems. - -When a trait conforms to `SuiteTrait` and is applied to a suite, the question of -when its custom scope (if any) should be applied is less obvious. Some suite -traits support inheritance and are recursively applied to all the test functions -they contain (including transitively, via sub-suites). Other suite traits don't -support inheritance, and only affect the specific suite they're applied to. -(It's also worth noting that a sub-suite _can_ have the same non-recursive suite -trait one of its ancestors has, as long as it's applied explicitly.) - -As a general rule of thumb, we believe most traits will either want to perform -custom logic once for _all_ children or once for _each_ child, not both. -Therefore, when it comes to suite traits, the default behavior should depend on -whether it supports inheritance: a recursive suite trait should by default -perform custom logic before each test, and a non-recursive one per-suite. But -the APIs should be flexible enough to support both, for advanced traits which -need it. - -## Detailed design - -I propose the following new APIs: - -- A new protocol `TestScoping` with a single required `provideScope(...)` method. - This will be called to provide scope for a test, and allows the conforming - type to perform custom logic before or after. -- A new method `scopeProvider(for:testCase:)` on the `Trait` protocol whose - result type is an `Optional` value of a type conforming to `TestScoping`. A - `nil` value returned by this method will skip calling the `provideScope(...)` - method. -- A default implementation of `Trait.scopeProvider(...)` which returns `nil`. -- A conditional implementation of `Trait.scopeProvider(...)` which returns `self` - in the common case where the trait type conforms to `TestScoping` itself. - -Since the `scopeProvider(...)` method's return type is optional and returns `nil` -by default, the testing library cannot invoke the `provideScope(...)` method -unless a trait customizes test behavior. This avoids the "unnecessarily lengthy -backtraces" problem above. - -Below are the proposed interfaces: - -```swift -/// A protocol that allows providing a custom execution scope for a test -/// function (and each of its cases) or a test suite by performing custom code -/// before or after it runs. -/// -/// Types conforming to this protocol may be used in conjunction with a -/// ``Trait``-conforming type by implementing the -/// ``Trait/scopeProvider(for:testCase:)-cjmg`` method, allowing custom traits -/// to provide custom scope for tests. Consolidating common set-up and tear-down -/// logic for tests which have similar needs allows each test function to be -/// more succinct with less repetitive boilerplate so it can focus on what makes -/// it unique. -public protocol TestScoping: Sendable { - /// Provide custom execution scope for a function call which is related to the - /// specified test and/or test case. - /// - /// - Parameters: - /// - test: The test under which `function` is being performed. - /// - testCase: The test case, if any, under which `function` is being - /// performed. When invoked on a suite, the value of this argument is - /// `nil`. - /// - function: The function to perform. If `test` represents a test suite, - /// this function encapsulates running all the tests in that suite. If - /// `test` represents a test function, this function is the body of that - /// test function (including all cases if it is parameterized.) - /// - /// - Throws: Whatever is thrown by `function`, or an error preventing this - /// type from providing a custom scope correctly. An error thrown from this - /// method is recorded as an issue associated with `test`. If an error is - /// thrown before `function` is called, the corresponding test will not run. - /// - /// When the testing library is preparing to run a test, it starts by finding - /// all traits applied to that test, including those inherited from containing - /// suites. It begins with inherited suite traits, sorting them - /// outermost-to-innermost, and if the test is a function, it then adds all - /// traits applied directly to that functions in the order they were applied - /// (left-to-right). It then asks each trait for its scope provider (if any) - /// by calling ``Trait/scopeProvider(for:testCase:)-cjmg``. Finally, it calls - /// this method on all non-`nil` scope providers, giving each an opportunity - /// to perform arbitrary work before or after invoking `function`. - /// - /// This method should either invoke `function` once before returning or throw - /// an error if it is unable to provide a custom scope. - /// - /// Issues recorded by this method are associated with `test`. - func provideScope(for test: Test, testCase: Test.Case?, performing function: @Sendable () async throws -> Void) async throws -} - -public protocol Trait: Sendable { - // ... - - /// The type of the test scope provider for this trait. - /// - /// The default type is `Never`, which cannot be instantiated. The - /// ``scopeProvider(for:testCase:)-cjmg`` method for any trait with this - /// default type must return `nil`, meaning that trait will not provide a - /// custom scope for the tests it's applied to. - associatedtype TestScopeProvider: TestScoping = Never - - /// Get this trait's scope provider for the specified test and/or test case, - /// if any. - /// - /// - Parameters: - /// - test: The test for which a scope provider is being requested. - /// - testCase: The test case for which a scope provider is being requested, - /// if any. When `test` represents a suite, the value of this argument is - /// `nil`. - /// - /// - Returns: A value conforming to ``Trait/TestScopeProvider`` which may be - /// used to provide custom scoping for `test` and/or `testCase`, or `nil` if - /// they should not have any custom scope. - /// - /// If this trait's type conforms to ``TestScoping``, the default value - /// returned by this method depends on `test` and/or `testCase`: - /// - /// - If `test` represents a suite, this trait must conform to ``SuiteTrait``. - /// If the value of this suite trait's ``SuiteTrait/isRecursive`` property - /// is `true`, then this method returns `nil`; otherwise, it returns `self`. - /// This means that by default, a suite trait will _either_ provide its - /// custom scope once for the entire suite, or once per-test function it - /// contains. - /// - Otherwise `test` represents a test function. If `testCase` is `nil`, - /// this method returns `nil`; otherwise, it returns `self`. This means that - /// by default, a trait which is applied to or inherited by a test function - /// will provide its custom scope once for each of that function's cases. - /// - /// A trait may explicitly implement this method to further customize the - /// default behaviors above. For example, if a trait should provide custom - /// test scope both once per-suite and once per-test function in that suite, - /// it may implement the method and return a non-`nil` scope provider under - /// those conditions. - /// - /// A trait may also implement this method and return `nil` if it determines - /// that it does not need to provide a custom scope for a particular test at - /// runtime, even if the test has the trait applied. This can improve - /// performance and make diagnostics clearer by avoiding an unnecessary call - /// to ``TestScoping/provideScope(for:testCase:performing:)``. - /// - /// If this trait's type does not conform to ``TestScoping`` and its - /// associated ``Trait/TestScopeProvider`` type is the default `Never`, then - /// this method returns `nil` by default. This means that instances of this - /// trait will not provide a custom scope for tests to which they're applied. - func scopeProvider(for test: Test, testCase: Test.Case?) -> TestScopeProvider? -} - -extension Trait where Self: TestScoping { - // Returns `nil` if `testCase` is `nil`, else `self`. - public func scopeProvider(for test: Test, testCase: Test.Case?) -> Self? -} - -extension SuiteTrait where Self: TestScoping { - // If `test` is a suite, returns `nil` if `isRecursive` is `true`, else `self`. - // Otherwise, `test` is a function and this returns `nil` if `testCase` is - // `nil`, else `self`. - public func scopeProvider(for test: Test, testCase: Test.Case?) -> Self? -} - -extension Trait where TestScopeProvider == Never { - // Returns `nil`. - public func scopeProvider(for test: Test, testCase: Test.Case?) -> Never? -} - -extension Never: TestScoping {} -``` - -Here is a complete example of the usage scenario described earlier, showcasing -the proposed APIs: - -```swift -@Test(.mockAPICredentials) -func example() { - // ...validate API usage, referencing `APICredentials.current`... -} - -struct MockAPICredentialsTrait: TestTrait, TestScoping { - func provideScope(for test: Test, testCase: Test.Case?, performing function: @Sendable () async throws -> Void) async throws { - let mockCredentials = APICredentials(apiKey: "...") - try await APICredentials.$current.withValue(mockCredentials) { - try await function() - } - } -} - -extension Trait where Self == MockAPICredentialsTrait { - static var mockAPICredentials: Self { - Self() - } -} -``` - -## Source compatibility - -The proposed APIs are purely additive. - -This proposal will replace the existing `CustomExecutionTrait` SPI, and after -further refactoring we anticipate it will obsolete the need for the -`SPIAwareTrait` SPI as well. - -## Integration with supporting tools - -Although some built-in traits are relevant to supporting tools (such as -SourceKit-LSP statically discovering `.tags` traits), custom test behaviors are -only relevant within the test executable process while tests are running. We -don't anticipate any particular need for this feature to integrate with -supporting tools. - -## Future directions - -### Access to suite type instances - -Some test authors have expressed interest in allowing custom traits to access -the instance of a suite type for `@Test` instance methods, so the trait could -inspect or mutate the instance. Currently, only instance-level members of a -suite type (including `init`, `deinit`, and the test function itself) can access -`self`, so this would grant traits applied to an instance test method access to -the instance as well. This is certainly interesting, but poses several technical -challenges that puts it out of scope of this proposal. - -### Convenience trait for setting task locals - -Some reviewers of this proposal pointed out that the hypothetical usage example -shown earlier involving setting a task local value while a test is executing -will likely become a common use of these APIs. To streamline that pattern, it -would be very natural to add a built-in trait type which facilitates this. I -have prototyped this idea and plan to add it once this new trait functionality -lands. - -## Alternatives considered - -### Separate set up & tear down methods on `Trait` - -This idea was discussed in [Supporting scoped access](#supporting-scoped-access) -above, and as mentioned there, the primary problem with this approach is that it -cannot be used with scoped access-style APIs, including (importantly) -`TaskLocal.withValue()`. For that reason, it prevents using that common Swift -concurrency technique and reduces the potential for test parallelization. - -### Add `provideScope(...)` directly to the `Trait` protocol - -The proposed `provideScope(...)` method could be added as a requirement of the -`Trait` protocol instead of being part of a separate `TestScoping` protocol, and -it could have a default implementation which directly invokes the passed-in -closure. But this approach would suffer from the lengthy backtrace problem -described above. - -### Extend the `Trait` protocol - -The original, experimental implementation of this feature included a protocol -named`CustomExecutionTrait` which extended `Trait` and had roughly the same -method requirement as the `TestScoping` protocol proposed above. This design -worked, provided scoped access, and avoided the lengthy backtrace problem. - -After evaluating the design and usage of this SPI though, it seemed unfortunate -to structure it as a sub-protocol of `Trait` because it means that the full -capabilities of the trait system are spread across multiple protocols. In the -proposed design, the ability to return a test scoping provider is exposed via -the main `Trait` protocol, and it relies on an associated type to conditionally -opt-in to custom test behavior. In other words, the proposed design expresses -custom test behavior as just a _capability_ that a trait may have, rather than a -distinct sub-type of trait. - -Also, the implementation of this approach within the testing library was not -ideal as it required a conditional `trait as? CustomExecutionTrait` downcast at -runtime, in contrast to the simpler and more performant Optional property of the -proposed API. - -### API names - -We first considered "execute" as the base verb for the proposed new concept, but -felt this wasn't appropriate since these trait types are not "the executor" of -tests, they merely customize behavior and provide scope(s) for tests to run -within. Also, the term "executor" has prior art in Swift Concurrency, and -although that word is used in other contexts too, it may be helpful to avoid -potential confusion with concurrency executors. - -We also considered "run" as the base verb for the proposed new concept instead -of "execute", which would imply the names `TestRunning`, `TestRunner`, -`runner(for:testCase)`, and `run(_:for:testCase:)`. The word "run" is used in -many other contexts related to testing though, such as the `Runner` SPI type and -more casually to refer to a run which occurred of a test, in the past tense, so -overloading this term again may cause confusion. - -## Acknowledgments - -Thanks to [Dennis Weissmann](https://github.com/dennisweissmann) for originally -implementing this as SPI, and for helping promote its usefulness. - -Thanks to [Jonathan Grynspan](https://github.com/grynspan) for exploring ideas -to refine the API, and considering alternatives to avoid unnecessarily long -backtraces. - -Thanks to [Brandon Williams](https://github.com/mbrandonw) for feedback on the -Forum pitch thread which ultimately led to the refinements described in the -"Avoiding unnecessary (re-)execution" section. +> [!NOTE] +> This proposal was accepted before Swift Testing began using the Swift +> evolution review process. Its original identifier was SWT-0007 but its prefix +> has been changed to "ST" and it has been relocated to the +> [swift-evolution](https://github.com/swiftlang/swift-evolution) repository. + +To view this proposal, see +[ST-0007: Test Scoping Traits](https://github.com/swiftlang/swift-evolution/blob/main/proposals/testing/0007-test-scoping-traits.md). diff --git a/Documentation/README.md b/Documentation/README.md index e41bc9568..3ef579986 100644 --- a/Documentation/README.md +++ b/Documentation/README.md @@ -29,8 +29,11 @@ The [Vision document](https://github.com/swiftlang/swift-evolution/blob/main/vis for Swift Testing offers a comprehensive discussion of the project's design principles and goals. -The [`Proposals`](Proposals/) directory contains API proposals that have been -accepted and merged into Swift Testing. +Feature and API proposals for Swift Testing are stored in the +[swift-evolution](https://github.com/swiftlang/swift-evolution) repository in +the `proposals/testing/` subdirectory, and new proposals should use the +[testing template](https://github.com/swiftlang/swift-evolution/blob/main/proposal-templates/0000-swift-testing-template.md) +there. ## Development and contribution