Add an ABI Stability vision document.

visions/abi-stability.md

@@ -0,0 +1,291 @@
+# A Vision for ABI Stability on non-Apple platforms
+
+## Introduction
+
+ABI or _Application Binary Interface_ refers to the details of how an
+application will interact with some other piece of code, including:
+
+* Calling conventions
+* Layout of data structures
+* Representations of scalar types
+* Availability of APIs
+
+If a piece of code has a _stable ABI_, that means that application
+programs can rely on there being no incompatible changes between the
+state of the ABI when they were being compiled and the ABI provided at
+runtime by the component they were linked with.
+
+End users of consumer operating systems, including both Apple and
+Microsoft platforms, tend to take ABI stability for granted; that is,
+they expect that they can install software onto their machine and that
+operating system updates will not arbitrarily break that software.  It
+would be easy to fool yourself into thinking, therefore, that ABI
+stability was the default, or that it was easy, whereas in practice
+considerable time and effort goes into making sure that things stay
+this way.
+
+A related concept is that of the _ABI boundary_, which is the level at
+which a given component provides a stable ABI.  For instance, on Apple
+and Microsoft operating systems, the stable ABI is provided by system
+libraries; while it is possible to dip below this level, for instance
+by doing direct system calls, no guarantee is made that the system
+call interface is ABI stable on those platforms---it is not part of
+the ABI boundary.  Contrast that with Linux, where the system call
+interface _is_ an ABI boundary, while some widely used libraries may
+not themselves be ABI stable _at all_.
+
+## What Does ABI Stability Buy Us?
+
+Put simply, it allows for shared binary components that get updated
+separately.  This is typically a requirement for consumer operating
+systems and software developed for them, so that software purchased by
+consumers can be guaranteed to operate across a wide range of
+operating system versions.
+
+These shared components might be system software or language runtimes
+on the one hand, but ABI stability also enables other patterns like
+binary plug-ins, which need a stable ABI from their host application.
+
+## Swift and ABI Stability
+
+At time of writing, Swift is officially ABI stable only on Apple
+platforms, where it forms part of the operating system, and indeed
+many operating system components are written in Swift.
+
+Apple's basic approach to ABI stability on its platforms relies on
+being able to tell the compiler the minimum system version on which
+the code it is building should run, coupled with the ability to
+annotate API with `@availability` annotations describing which system
+versions support which APIs.  The availability mechanism also allows
+the compiler to decide automatically whether to weak link the
+symbol(s) in question.
+
+That does not completely solve the problem, however, because the
+platform needs to be able to evolve the data types it exposes to
+applications, while allowing applications to subclass Swift or
+Objective-C types declared in the platform API.
+
+You may be familiar with the C++ "fragile base class" problem, which
+is essentially that a subclass of a C++ class needs to know the layout
+(or at least the size) of the base class, which implies that the
+latter cannot change.  In C++ the traditional solution to that is the
+_`pimpl` idiom_ (essentially, you make the base class have a single
+pointer member that points to its actual data).
+
+Both Swift and its predecessor language Objective-C solve this problem
+in a different way, namely for ABI stable structs and classes (all
+classes in Objective-C, but in Swift only `public` non-`frozen` types
+compiled with _library evolution_ enabled), the _runtime_ is in charge
+of data layout, and the compiler, when asked to access a specific data
+member, will make runtime calls as necessary to establish where it
+should read or write data.  This avoids the extra indirection of the
+C++ `pimpl` idiom and in practice it's possible to generate reasonably
+efficient code.
+
+## Platforms and ABI Stability
+
+### Darwin
+
+In addition to the `@availability`/minimum system version and
+non-fragile base class support, the Darwin linker has a number of
+features intended to allow code to be moved from one framework or
+dynamic library to another without an ABI break, including support for
+re-exporting symbols, as well as a variety of meta symbols that allow
+for manipulation of symbol visibility and indeed apparent location
+based on system versions.
+
+### Windows
+
+Like Apple platforms, Windows is ABI stable at the API layer; SPIs are
+not officially ABI stable (though in practice they may be), and
+neither is the system call interface itself.  Note, though, that there
+is a wrinkle here---the C/C++ runtime is _not_ part of the Windows
+API, and does not form part of the ABI boundary.
+
+Windows has a platform-wide exception handling mechanism (Structured
+Exception Handling or SEH), which means that the operating system
+specifies in detail how stack unwinding must take place.  Unlike other
+platforms we care about, Windows does not use DWARF for exception
+unwinding, but instead relies on knowledge of the standard function
+prologue and epilogue.  This creates problems for tail call
+optimization, which is also needed for Swift Concurrency.  While we
+have *a* solution in place, we need to be convinced that it's the
+*right* solution before declaring ABI stability on Windows.  Also
+unlike other systems, SEH unwinding can be triggered by *asynchronous*
+exceptions, which can occur during the prologue or epilogue, which
+means we can't simply declare that e.g. async functions don't need to
+follow the standard ABI here (since we aren't in charge of when these
+might happen).
+
+Windows DLLs also have some interesting design features that need to
+be borne in mind here.  In particular, symbols exported from DLLs can
+be exported by 16-bit _ordinal_ as well as by name; name exports are
+actually done by mapping the name to an ordinal, and then looking up
+the ordinal in the export table.  If symbols are exported by ordinal
+alone, the ordinals need to be stable in order to achieve ABI
+stability.  Further, because of the way linking works, people have in
+the past generated import libraries directly from DLL files, which
+tends to result in import-by-ordinal rather than import-by-name, which
+has actually forced Microsoft to stabilise ordinals when it otherwise
+would not have done so.
+
+Windows DLLs are also, like dylibs on Darwin, able to re-export
+symbols from other DLLs (Windows calls this feature _forwarding_); the
+Windows feature is also able to rename the symbol at the same time,
+and indeed the Win32 layer re-exports some NT SPIs directly as Win32
+APIs using this mechanism.
+
+As regards data structure ABI stability, Windows uses two different
+mechanisms here; the first is that data structures typically start
+with a header that contains a size field (which gets used,
+essentially, as a version).  The second is use of the Component Object
+Model (COM), which relies on the way the Microsoft compiler lays out
+C++ vtables.
+
+Windows doesn't have anything quite like the `availability` mechanism
+Apple platforms use, though there are C preprocessor macros like
+`WINVER` or `_WIN32_WINNT` that control exactly which versions of data
+structures and which functions are exported by the Windows SDK headers.
+
+Windows programs that need to cope with the possibility that a function
+might not be present at runtime need to use `GetProcAddress()` (the
+Win32 equivalent of `dlsym()`).
+
+### Linux
+
+Linux is a more complicated situation.  The Linux kernel's system call
+interface *is* ABI stable, though it isn't necessarily uniform across
+supported platforms (there are system calls where the exact set of and
+order of arguments is platform specific).  The Swift Static SDK for
+Linux relies on this feature in that it generates binaries that
+directly talk to the Linux kernel, albeit via a statically linked C
+library.
+
+Beyond that, though, the platform _as a whole_ is not ABI stable,
+though individual dynamic libraries might be (Glibc is a case in
+point), and particular distributions might guarantee some degree of
+ABI stability for each major release (e.g. a program built for Ubuntu
+18.04 will likely run on Ubuntu 18.10, but might not work on Ubuntu
+22.04).  Even if all the libraries you are using are themselves ABI
+stable, the versions of libraries installed by different distributions
+may differ---distribution X may have a newer version of Glibc, but an
+older version of (say) zlib, while distribution Y has an older Glibc
+but a newer zlib.  And some distributions make different choices about
+what "the system C library" or "the system C++ library" might be (this
+isn't limited to low-level libraries either---similar choices might
+happen for much higher layers of the system, such as which window
+system to use, whether to use MIT Kerberos or Heimdall, and so on).
+
+The upshot is that it is difficult to distribute binary programs for
+Linux, without doing what the Swift Static SDK for Linux does and
+simply not relying on external dynamic libraries.  This difficulty has
+also led to the creation of systems like Snaps and Flatpak, which let
+an application bundle all its dependencies in such a way that they
+won't interfere with other installed applications.  It is also a
+factor in the rise in popularity of OCI images, particularly for
+server applications.
+
+There are some system features intended to support ABI stability at
+the library level, including versioned shared object names and symbol
+versioning.  These get used by ABI stable libraries like Glibc such
+that it's generally possible to run a program that depends on such a
+library on the version it was built for _or any newer version_.  This
+does make it possible to ship binaries that will run on a wide variety
+of Linux systems by ensuring that your program is linked against the
+oldest version(s) of dynamic libraries that you care about, _provided
+those libraries are themselves ABI stable_.  Python's "manylinux"
+package builds take this approach, but have to restrict themselves as
+a result to a small subset of libraries that are known to be ABI
+stable.
+
+## Declaring ABI Stability
+
+Supported platforms for Swift must have a Platform Owner.  Because a
+declaration of ABI stability for a platform has implications for
+compiler and runtime engineers who do not develop on that platform, we
+intend to set a high bar for declaring ABI stability.  As such, to
+become officially ABI stable:
+
+1. The Platform Owner will raise a Swift Evolution proposal for
+   platform ABI stability.  The actual proposal could come from
+   someone other than the Platform Owner, but if so, the Platform
+   Owner will be asked to indicate their support for the proposal.
+
+2. The proposal should address, at a minimum:
+
+   - Calling conventions, including `async`.
+   - Layout of data structures.
+   - Representations of scalar types (e.g. size of `Int`/`UInt`).
+   - API availability mechanisms.
+   - Linkage issues (particularly regarding dynamic libraries).
+
+   It is acceptable, and indeed encouraged, to refer to other
+   documentation (e.g. the Itanium ABI, documents in the `doc`
+   directory in the Swift repo), as well as to the behaviour of
+   existing ABI stable platforms.
+
+3. The proposal should also address any unique platform features
+   that may be relevant (e.g. Windows DLL behaviour, exception
+   handling concerns and so on).
+
+4. The Platform Steering Group will review the proposal.  If the
+   PSG and the Platform Owner agree that all relevant concerns have
+   been met, the platform will be declared ABI Stable and all future
+   changes to the compiler, runtime or standard library will be
+   expected to maintain ABI stability for that platform.
+
+## Goals
+
+* Swift should support existing platform ABI stability mechanisms,
+  so that the language is a good citizen when writing native software.
+
+  - `@availability` should be well-defined for Windows; it should be
+    possible to annotate Windows APIs to say e.g. that they were
+    introduced in Windows 10 version 1709.  Swift should automatically
+    generate the code to weakly reference functions it imports where
+    necessary.
+
+  - `@availability` as a system-wide concept doesn't make as much
+    sense on non-Apple UNIX/UNIX-like platforms where it's possible to
+    install different versions of many standard packages.  This is
+    especially true on Linux where there are multiple "distributions",
+    so the major versions of packages in major versions of those
+    distributions are fixed, but one distribution's set of packages
+    may bear little relation to another's.  In this case,
+    `@availability` annotations could perhaps be used for the system C
+    library and also for the Swift runtime itself.
+
+  - It would be good to be able to support Linux-style symbol
+    versioning on ELF platforms. [^1]
+
+[^1]: See Ulrich Drepper's [_How to Write Shared
+    Libraries_](https://archive.org/download/dsohowto/dsohowto.pdf),
+    which goes into considerable detail on the Linux ABI stability
+    story, albeit mostly focusing on Glibc.
+
+* Swift should provide an ABI stable runtime and standard library
+  wherever it makes sense to do so.
+
+  - It is already ABI stable on Apple platforms.
+
+  - It makes a great deal of sense for it to be ABI stable on Windows.
+
+  - On other UNIX/UNIX-like platforms, we should aim for it to be ABI
+    stable when dynamically linked, even if underlying libraries (like
+    the C library) are not.  This would at least mean that programs
+    that just use the standard library would work as long as a
+    suitable copy of the Swift runtime was present on the system.
+
+* It should be possible to write ABI stable APIs _in_ Swift.
+
+## Non-Goals
+
+At this point, the following are explicitly not goals:
+
+* Providing `@availability`-like support for third party library code.
+  This is an interesting idea, but would require a method of
+  specifying the minimum version, which probably means being able to
+  specify version numbers when importing modules, e.g.
+  e.g. `@version(12.3) import Libfoo` or `import [email protected]` or
+  similar.  It would also complicate the compiler's availability
+  tracking code considerably.