Connexion-based mockup service implementing parts of the GA4GH Task Execution Service (TES) API schema. The service was developed to implement and test TEStribute, a task distribution logic package for TES instances. It features an extended TES specification to provide parameters required for the model underlying the task distribution logic.
Once deployed and started (see below), the service will be
available at:
http://localhost:9001/ga4gh/tes/v1/
You can explore the service via the Swagger UI:
firefox http://localhost:9001/ga4gh/tes/v1/ui/
The specifications, in JSON format, can be retrieved with:
wget http://localhost:9001/ga4gh/tes/v1/swagger.json
Note that host and port can be set manually in the config file. In that case, the values in the URLs above need to be replaced as well.
The client TES-cli can be used to send requests to the service.
mock-TES
can be deployed via containers (preferred) or after manual
installation of all dependencies.
In both cases, the repository first needs to be cloned with:
git clone [email protected]:elixir-europe/mock-TES.git
Afterwards traverse to the repository's root directory:
cd mock-TES
"Production-like" containerized deployment without HTTP server/load balancer etc.
- Git (tested with version 2.17.1)
- Docker (tested with version 18.09.6)
- docker-compose (tested with version 1.24.0)
# Build application image
# [NOTE] Image re-building is not always necessary. Inspect the `Dockerfile`
# to check which changes will need re-building.
docker-compose build
# Start service
docker-compose up -d
# Check logs
docker-compose logs
# Shut down service
docker-compose down
Deployment for local development without containers, HTTP server/load balancer etc.
- Git (tested with version 2.17.1)
- Python (tested with versions 2.7.15+ & 3.6.8)
- pip (tested with version 19.1.1)
- virtualenv (tested with version 15.1.0)
# Set up Python virtual environment
virtualenv -p `which python3` venv
source venv/bin/activate
# Install dependencies
pip install -r requirements.txt
# Install app
python setup.py develop
# Run service
python mock_tes/Server.py
Note that only those parts of the service are implemented that are required for developing, testing and debugging TEStribute. For all other endpoints only stubs are implemented that return the most basic valid response (typically an "empty" JSON object).
An endpoint /tasks/task-info
was added to commit d55bf88
of the Task
Execution Service OpenAPI specification. It provides parameters required for
TEStribute's task distribution logic.
The modified specifications as well as a snapshot of the unmodified specifications are available in this repository.
This section outlines the definitions of the endpoint and underlying request and response models.
The endpoint /tasks/task-info
returns estimations of queue time and incurred
costs if a task with a given set of resource requirements were to be computed
on this TES instance right now. Within the framework of TEStribute, these
parameters allow informed decisions with regards to which of a number of TES
instances a given task should be sent to. The endpoint is defined by:
/tasks/task-info:
post:
summary: |-
Provides estimates for the queue time and incurred costs for a task
with the given resource requirements.
operationId: GetTaskInfo
responses:
'200':
description: ''
schema:
$ref: '#/definitions/tesTaskInfo'
parameters:
- name: body
in: body
required: true
schema:
$ref: '#/definitions/tesResources'
tags:
- TaskService
x-swagger-router-controller: ga4gh.tes.server
A property execution_time_sec
was added to the model describing a task's
resource requirements. The entire model is now defined as follows:
tesResources:
type: object
properties:
execution_time_sec:
type: number
format: double
description: Requested execution in seconds (s)
cpu_cores:
type: integer
format: int64
description: Requested number of CPUs
preemptible:
type: boolean
format: boolean
description: Is the task allowed to run on preemptible compute instances (e.g. AWS Spot)?
ram_gb:
type: number
format: double
description: Requested RAM required in gigabytes (GB)
disk_gb:
type: number
format: double
description: Requested disk size in gigabytes (GB)
zones:
type: array
items:
type: string
description: Request that the task be run in these compute zones.
description: Resources describes the resources requested by a task.
A valid request to the endpoint results in a response defined in the following model:
type: object
properties:
estimated_compute_costs:
$ref: '#/definitions/tesCosts'
description: |-
Estimated total incurred compute costs for running a task with the
indicated resource requirements on this TES instance. Excludes data
storage and transfer costs.
estimated_storage_costs:
$ref: '#/definitions/tesCosts'
description: |-
Estimated total incurred data storage costs for running a task with
the indicated resource requirements on this TES instance. Excludes
compute and data transfer costs. Currently this is reported as a flat
fee for conceptually indefinite storage.
estimated_queue_time_sec:
type: number
format: double
description: |-
Given the current load on this TES instance, returns an estimate of
the time, in seconds (s), that a task with the indicated resource
requirements will spend in the task queue.
unit_costs_data_transfer:
$ref: '#/definitions/tesCosts'
description: |-
Unit costs for transferring 1 gigabyte (GB) of data (inputs, outputs)
across 1000 kilometers (km).
description: |-
Given a set of resource requirements, returns the estimated queue time
and total incurred costs. Allows informed decisions with regard to which
TES instance a given task should be sent to.
The response model further relies on model tesCosts
which specifies an
amount
and a currency
property:
tesCosts:
type: object
properties:
amount:
type: number
format: double
description: Numeric value specifying an amount of money.
currency:
type: string
enum:
- AUD
- BRL
- BTC
- CAD
- CHF
- CNH
- EUR
- GBP
- HKD
- INR
- KRW
- JPY
- MXN
- NOK
- NZD
- RUB
- SEK
- SGD
- TRY
- USD
- ZAR
description: Currency/unit of the costs.
description: Generic object specifying an amount of money.
The service can be configured with different unit costs for CPU and memory
usage, data transfer and storage. Default values for the corresponding
parameters are listed in task_info
section of the service's config and they
can be edited by the user before starting the service.
Alternatively (and preferably), these parameters can be modified in the running
service via the /update-config
endpoint, which is particularly useful for
setting up environments for various testing scenarios for TEStribute. The
endpoint is defined in the update config OpenAPI specifications:
/update-config:
post:
summary: Update task info config
operationId: UpdateTaskInfoConfig
responses:
'200':
description: ''
schema:
$ref: '#/definitions/tesTaskInfoConfig'
parameters:
- name: body
in: body
schema:
$ref: '#/definitions/tesTaskInfoConfig'
required: true
description: ''
tags:
- TaskService
x-swagger-router-controller: ga4gh.tes.server
It relies on the following models:
tesTaskInfoConfig:
type: object
properties:
currency:
type: string
enum:
- AUD
- BRL
- BTC
- CAD
- CHF
- CNH
- EUR
- GBP
- HKD
- INR
- KRW
- JPY
- MXN
- NOK
- NZD
- RUB
- SEK
- SGD
- TRY
- USD
- ZAR
description: Currency/unit of the costs.
unit_costs:
$ref: '#/definitions/tesTaskInfoCosts'
tesTaskInfoCosts:
type: object
properties:
cpu_usage:
type: integer
format: int64
description: costs per core
memory_consumption:
type: integer
format: int64
description: costs of computation per GB
data_storage:
type: integer
format: int64
description: cost of data storage GB
data_transfer:
type: integer
format: int64
description: cost of data transfer per GB and 1000 km
TES-cli can be used to communicate with the /update-config
endpoint.
Note that while the
/update-config
endpoint can be accessed via the same root URI (and explored via the Swagger UI), it was not included in the modified TES specifications, but rather is added to it on the fly when the service is started.
This project is a community effort and lives off your contributions, be it in the form of bug reports, feature requests, discussions, or fixes and other code changes. Please read the contributing guidelines if you want to contribute. And please mind the code of conduct for all interactions with the community.
Development of the app is currently still in alpha stage, and current versioning is for internal use only. In the future, we are aiming to adopt semantic versioning that is synchronized to the versioning of TEStribute and TES-cli in order to ensure that these apps will be compatible as long as both their major and minor versions match.
This project is covered by the Apache License 2.0 also available shipped with this repository.
Please contact the project leader for inquiries, proposals, questions etc. that are not covered by the Contributing section.
The project is a collaborative effort under the umbrella of the ELIXIR Cloud and AAI group. It was started during the 2019 Google Summer of Code as part of the Global Alliance for Genomics and Health organization.