Processing Parameters Extension #276 #545 (#471)

* Jobs and services: Added `create_parameters` property in responses to define the parameters that can be submitted suring creation #276

* Add processing options to a separate endpoint #276

* Options should be optional

* Move to an extension

* Add UDP extension

* Apply suggestions from code review

Co-authored-by: Stefaan Lippens <[email protected]>

* Renamed according to PR review

* Clarify where UDPs can come from

* Apply suggestions from code review

* Add example

* Update extensions/processing-parameters/README.md

Co-authored-by: Stefaan Lippens <[email protected]>

---------

Co-authored-by: Stefaan Lippens <[email protected]>

CHANGELOG.md

@@ -10,6 +10,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 - **New extensions:**
   - [Remote Process Definition Extension](./extensions/remote-process-definition/README.md)
+  - [Processing Parameters Extension](./extensions/processing-parameters/README.md)
 - Added `version` property to `GET /processes` [#517](https://github.com/Open-EO/openeo-api/issues/517)
 - `POST /result`: Added response header "OpenEO-Identifier" to expose an identifier associated with a synchronous processing request.
 

README.md

@@ -33,6 +33,7 @@ See also the [changelog](CHANGELOG.md) and the [milestones](https://github.com/O
 | ------------------------------------------------------------------ | ------- | ------------ | ----------- |
 | [Commercial Data](extensions/commercial-data/)                     | 0.1.0   | experimental | Provides an interface for discovering, ordering and using commercial data. |
 | [Federation](extensions/federation/)                               | 0.1.0   | experimental | Covers federation aspects, i.e. where multiple back-ends are exposed as a single API. |
+| [Processing Parameters](extensions/processing-parameters/)         | 0.1.0   | experimental | Explore and handle additional processing parameters that a back-end can offer for the processing modes (sync. processing, batch jobs, web services). |
 | [Remote Process Definition](extensions/remote-process-definition/) | 0.1.0   | experimental | Load user-defined processes that are hosted externally through the process namespace into process graphs. |
 
 ## Repository

extensions/processing-parameters/.spectral.yml

@@ -0,0 +1,12 @@
+extends: "spectral:oas"
+rules:
+  contact-properties: true
+  tag-description: true
+  oas3-parameter-description: true
+  oas3-unused-component: true
+  operation-id-kebab-case:
+    given: "$..operationId"
+    then:
+      function: pattern
+      functionOptions:
+        match: ^[a-z][a-z0-9\-]*$

extensions/processing-parameters/README.md

@@ -0,0 +1,75 @@
+# Processing Parameters Extension
+
+The Processing Parameters Extension to the openEO API provides an interface to explore and handle additional processing parameters that a back-end can offer for the three processing modes (synchronous processing, batch jobs, secondary web services).
+
+- Version: **0.1.0**
+- Stability: **experimental**
+- [OpenAPI document](openapi.yaml)
+- Conformance class: `https://api.openeo.org/extensions/processing-parameters/0.1.0`
+
+**Note:** This document only documents the additions to the specification.
+Extensions can not change or break existing behavior of the openEO API.
+
+This extension adds a new endpoint (`GET /processing_parameters`, see [OpenAPI document](openapi.yaml))
+to discover the additional processing parameters that a back-end offers.
+
+Additionally, this extension allows to provide specific default values for user-defined processes (UDPs),
+which includes:
+
+- UDPs submitted directly for synchronous processing, as batch jobs, or as secondary web services
+- UDPs stored through the `/process_graphs` endpoints
+- UDPs stored external to the API and retrieved through the [Remote Process Definition Extension](../remote-process-definition/README.md)
+
+The parameters and its values are provided separately for each processing mode.
+
+## Embedding default processing options in UDPs
+
+UDPs can provide default values for specific processing parameters.
+
+The values for each parameter (so called 'options') are provided separately for each processing mode.
+The following properties are added to the top-level of a UDP (e.g. as sibling nodes to the "process_graph" property) for the respective processing modes:
+
+- `default_synchronous_options` for synchronous processing
+- `default_job_options` for batch jobs
+- `default_service_options` for secondary web services
+
+The schema for each of these properties is:
+
+```yaml
+type: object
+additionalProperties:
+  description: Any type
+```
+
+The keys of the object are the respective parameter names.
+The values of the object are the default values for the parameters.
+Schematic restrictions are not defined for the object, but the schemas for the parameters as defined in `GET /processing_parameters` apply to the given values.
+These values provide the defaults unless a user overrides them in the actual data processing request (e.g. `POST /jobs`, see below).
+
+Example UDP including defaults for the processing parameters `memory` and `block-sizes` of a batch job:
+
+```json
+{
+  "id": "my_evi",
+  "parameters": [...],
+  "process_graph": {...},
+  "default_job_options": {
+    "memory": "2GB",
+    "block-sizes": [128, 32]
+  }
+}
+```
+
+## Resolving parameters
+
+Due to the variety of places where processing parameters can be provided, the following
+list defines how the parameters must be resolved. The prioritization is as follows:
+
+1. If present, use the parameter specified in the processing request directly (e.g. in `POST /jobs` as a top-level property)
+2. If present, use the default parameter specified in the UDP
+3. Otherwise, use the default value for the parameter as specified in `GET /processing_parameters`
+
+"Present" means that the property is present in the JSON representation regardless of the value given, i.e.
+properties are present if an empty string, an empty array, an empty object, `false`, `0`, or `null` are provided.
+
+Unrecognized/unknown parameters that are provided through UDPs must be ignored by backends.

extensions/processing-parameters/openapi.yaml

@@ -0,0 +1,100 @@
+openapi: 3.0.2
+info:
+  title: openEO API - Processing Parameters Extension
+  version: 0.1.0
+  description: >-
+    The Processing Parameters Extension to the openEO API provides an interface to explore and handle additional processing parameters that a back-end can offer for the three processing modes (synchronous processing, batch jobs, and secondary web services).
+    The openEO API specification allows back-ends to accept "additional back-end specific properties" for each of the processing modes.
+    As the openEO API specification does not define the name and schemas of these properties, this extension provides a standardized way to define and document these properties.
+  contact:
+    name: openEO Consortium
+    url: 'https://openeo.org'
+    email: [email protected]
+  license:
+    name: Apache 2.0
+    url: 'http://www.apache.org/licenses/LICENSE-2.0.html'
+externalDocs:
+  url: https://github.com/Open-EO/openeo-api/blob/draft/extensions/workspaces/README.md
+tags:
+  - name: Capabilities
+    description: General information about the API implementation and other supported capabilities provided by the back-end.
+  - name: Data Processing
+    description: Organizes and manages data processing on the back-end, either as synchronous on-demand computation or batch jobs.
+  - name: Batch Jobs
+    description: Management of batch processing tasks (jobs) and their results.
+  - name: Secondary Services
+    description: On-demand access to data using other web service protocols.
+servers:
+  - url: 'https://openeo.example/api/{version}'
+    description: >-
+      The URL of the API MAY freely be chosen by the back-end providers. The
+      path, including API versioning, is a *recommendation* only. Nevertheless,
+      all servers MUST support HTTPS as the authentication methods are not
+      secure with HTTP only!
+    variables:
+      version:
+        default: v1
+        description: >-
+          API versioning is RECOMMENDED. As the openEO API is following
+          [SemVer](https://semver.org/) only the MAJOR part of the stable
+          version numbers (i.e. versions >= 1.0.0) SHOULD be used for API
+          versioning in the URL. The reason is that backward-incompatible
+          changes are usually introduced by major changes. Therefore, the
+          version number in the URL MUST not be used by the clients to detect
+          the version number of the API. Use the version number returned from
+          `GET /` instead.
+paths:
+  /processing_parameters:
+    get:
+      summary: Additional processing parameters
+      operationId: list-processing-parameters
+      description: |-
+        Lists additional custom processing parameters
+        that a back-end offers for the different processing modes (synchronous processing, batch jobs, secondary web services).
+        The parameters specified here can be added to the corresponding `POST` requests at the top-level of the object that is sent as the payload.
+        All parameters SHOULD explicitly be made optional with reasonable defaults as otherwise the interoperability between the implementations decreases.
+      tags:
+        - Capabilities
+        - Data Processing
+        - Batch Jobs
+        - Secondary Services
+      security:
+        - {}
+        - Bearer: []
+      responses:
+        '200':
+          description: >-
+            An object with a list of parameters per processing mode.
+          content:
+            application/json:
+              schema:
+                description: Processing parameters per processing mode.
+                type: object
+                properties:
+                  create_job_parameters:
+                    $ref: '#/components/schemas/processing_create_parameters'
+                  create_service_parameters:
+                    $ref: '#/components/schemas/processing_create_parameters'
+                  create_synchronous_parameters:
+                    $ref: '#/components/schemas/processing_create_parameters'
+components:
+  schemas:
+    processing_create_parameters:
+      title: Creation Parameters
+      description: |-
+        List of additional custom parameters that a back-end offers during the creation
+        of batch jobs (`POST /jobs`) and secondary web services (`POST /services`) respectively.
+      type: array
+      items:
+        $ref: '../../openapi.yaml#/components/schemas/parameter'
+      example:
+        - name: memory
+          description: Maximum amount of memory that will be allocated for processing, in gigabytes.
+          optional: true
+          default: 32
+          schema:
+            type: integer
+            minimum: 1
+  securitySchemes:
+    Bearer:
+      $ref: '../../openapi.yaml#/components/securitySchemes/Bearer'

extensions/processing-parameters/package.json

@@ -0,0 +1,24 @@
+{
+  "name": "@openeo/api-extension-processing-parameters",
+  "version": "0.1.0",
+  "author": "openEO Consortium",
+  "license": "Apache-2.0",
+  "description": "The openEO API specification.",
+  "homepage": "https://openeo.org",
+  "bugs": {
+    "url": "https://github.com/Open-EO/openeo-api/issues"
+  },
+  "repository": {
+    "type": "git",
+    "url": "git+https://github.com/Open-EO/openeo-api.git"
+  },
+  "devDependencies": {
+    "@stoplight/spectral-cli": "^6.6.0",
+    "redoc-cli": "^0.13.21"
+  },
+  "scripts": {
+    "start": "redoc-cli serve openapi.yaml --watch --options.expandResponses \"200,201,202,203,204\" --options.pathInMiddlePanel true",
+    "build": "redoc-cli bundle openapi.yaml -o redoc.html --title \"openEO API - Processing Parameters Extension\" --cdn --options.expandResponses \"200,201,202,203,204\" --options.pathInMiddlePanel true",
+    "test": "spectral lint openapi.yaml"
+  }
+}

extensions/remote-process-definition/README.md

@@ -13,9 +13,10 @@ This extension enables user to load user-defined processes that are hosted exter
 The openEO API defines the `namespace` property in a process node of a process graph as follows:
 
 > The following options are predefined by the openEO API, but additional namespaces may be introduced by back-ends or in a future version of the API.
-> * `null` [...]
-> * `backend` [...]
-> * `user` [...]
+>
+> - `null` [...]
+> - `backend` [...]
+> - `user` [...]
 
 This makes it possible for this extension to add additional allowed values to the `namespace` property.