Derive ElementaryFunctions conformances for structs.

[dan-zheng]

ElementaryFunctions derived conformances enable elementary math functions to work with product spaces formed from ElementaryFunctions-conforming types.

Enables efficient, elegant mathematical optimizers.
Example:

public class Adam<Model: Differentiable>
where
    Model.TangentVector: VectorProtocol & ElementaryFunctions,
    Model.TangentVector.VectorSpaceScalar: BinaryFloatingPoint & ElementaryFunctions
{
    public typealias Scalar = Model.TangentVector.VectorSpaceScalar
    public var learningRate: Scalar
    public var beta1: Scalar
    public var beta2: Scalar
    public var epsilon: Scalar
    public var decay: Scalar
    public var step: Int = 0
    public var firstMoments: Model.TangentVector = .zero
    public var secondMoments: Model.TangentVector = .zero

    public init(
        for model: __shared Model,
        learningRate: Scalar = 1e-3,
        beta1: Scalar = 0.9,
        beta2: Scalar = 0.999,
        epsilon: Scalar = 1e-8,
        decay: Scalar = 0
    ) {
        precondition(learningRate >= 0, "Learning rate must be non-negative")
        precondition(0 <= beta1 && beta1 <= 1, "Beta parameter must be between 0 and 1")
        precondition(0 <= beta2 && beta2 <= 1, "Beta parameter must be between 0 and 1")
        precondition(decay >= 0, "Learning rate decay must be non-negative")

        self.learningRate = learningRate
        self.beta1 = beta1
        self.beta2 = beta2
        self.epsilon = epsilon
        self.decay = decay
    }

    public func update(_ model: inout Model, along direction: Model.TangentVector) {
        step += 1
        let learningRate = self.learningRate / (1 + decay * Scalar(step))
        // Note: `stepSize` is split into two lines to avoid the "compiler is unable to type-check
        // this expression in reasonable time" error.
        var stepSize = learningRate * Scalar.sqrt(1 - Scalar.pow(beta2, step))
        stepSize = stepSize / (1 - Scalar.pow(beta1, step))
        firstMoments = firstMoments * beta1 + (1 - beta1) * direction
        secondMoments = secondMoments * beta2 + (1 - beta2) * Model.TangentVector.pow(direction, 2)
        model.move(along: -stepSize * firstMoments / (Model.TangentVector.sqrt(secondMoments) + epsilon))
    }
}

Resolves TF-578.
Enables revamping optimizers in tensorflow/swift-apis#218.

[dan-zheng]

This logic (conform synthesized Differentiable associated types to ElementaryFunctions if possible) is not super principled. Similar logic exists for conforming synthesized types to VectorProtocol if possible.

We could remove this logic by constraining TangentVector to VectorProtocol and/or ElementaryFunctions, but it's not clear to me whether that's desirable. Constraining TangentVector to VectorProtocol seems more reasonable.

[dan-zheng]

Note: adding a filter function here is necessary because, for the first time, derived conformances must handle two protocol requirements with the same name: pow(_ x: Self, _ y: Self) -> Self and pow(_ x: Self, _ n: Int) -> Self.

[dan-zheng]

Verifying whether tests pass.
@swift-ci Please test tensorflow

[dan-zheng]

@swift-ci Please test tensorflow

[dan-zheng]

https://ci-external.swift.org is down, locally verified that tests pass.
AdditiveArithmetic and ElementaryFunctions derived conformances will be pitched soon via Swift Evolution.