Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Integrate optional task distribution logic middleware #16

Open
uniqueg opened this issue Sep 2, 2019 · 4 comments
Open

Integrate optional task distribution logic middleware #16

uniqueg opened this issue Sep 2, 2019 · 4 comments
Labels
status: on hold Won't be fixed for now

Comments

@uniqueg
Copy link
Member

uniqueg commented Sep 2, 2019

Is your feature request related to a problem? Please describe.

One of the main benefits of having TES requests intercepted by proTES is that it allows for the distribution of tasks over a network of TES instances, regardless of whether the workflow engine that emitted the TES request contains such a feature. Through the integration of a task distribution logic, proTES will be able to select the most advantageous TES instance for a given task, according to cost and/or clock time considerations.

Describe the solution you'd like

The proof-of-concept task distribution app TEStribute has recently been developed.

A publicly accessible API service of TEStribute can be integrated by service calls to its single endpoint /rank-services. Refer to TEStribute's documentation for details.

Describe alternatives you've considered

If no publicly accessible TEStribute API service is available, the package can also be imported and used in the following manner (refer to TEStribute documentation for details):

from TEStribute import rank_services

rank_services(...)

Additional context

Note that TEStribute currently works slightly beyond the GA4GH specs, i.e., it makes an assumption of data repository services that is not warranted by their specification and it amends a property in a model of one endpoint and adds another required endpoint to the TES specs, thus making it only with specifically tuned TES implementations for now. For this reason, the use of the task distribution logic middleware must be optional (set a switch and necessary config parameters in the config file.

@vaniisgh
Copy link

vaniisgh commented Oct 12, 2019

Would love to try and tackle this (or #37 after this is done) if it's still unassigned or you haven't started work @uniqueg :)

@uniqueg
Copy link
Member Author

uniqueg commented Oct 12, 2019

Please, go ahead - haven't started! Let me know if you need help

@uniqueg
Copy link
Member Author

uniqueg commented Nov 19, 2019

The controller handling incoming task requests is in pro_tes/ga4gh/tes/endpoints/create_task.py. This would be the place to inject any middleware. Best is to abstract this a bit by adding a module middleware.py somewhere and then add calling of TEStribute as a function or class...

@uniqueg uniqueg added the status: on hold Won't be fixed for now label Mar 8, 2023
@uniqueg
Copy link
Member Author

uniqueg commented Mar 8, 2023

For the time being, we have implemented a (simpler) task distribution logic (which took portions from TEStribute) right in proTES, see #119. Integration of TEStribute will be put on hold until further notice (together with #37).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
status: on hold Won't be fixed for now
Projects
None yet
Development

No branches or pull requests

2 participants