RFC 008: hyper63 work queue service

[twilson63]

Through the discovery process, it has become a parent that hyper63 needs to provide a queue service:

Here is a proposal

What this proposal is not:

• It is not a publish/subscribe service, that is a different set of use cases
• It is not a service for long running background tasks/jobs
• It is not a scheduling service for tasks

Challenge

Background processing in a serverless world.

When data is persisted or if a specific event in the business logic engine occurs, there may
be a need to do a background process event as a result of that change event. This background
process event should not be a part of the same transaction that created the event trigger. It
should occur in its own transaction and be able to notify the client when complete.

With Server-less technology it is not possible to do this within a serverless function, because as
soon as the response of the function is resolved to the client, the process is terminated.

What is a queue?

A queue is a pattern that creates a handoff between one process and another process. You can think of the pattern broken into three logical entities. Provider, Broker, and Consumer. The Provider is responsible for putting tasks or jobs on the queue, the consumer or worker is responsible for taking items off of the queue and the broker is the intermediary that is responsible for keeping track of each tasks or jobs state.

What is a task or job?

What is a task or a job? A task or job is a document, or collection of data elements that have keys and values, these documents are used to pass instructions from the Provider to the Consumer. The Task or Job will contain an id and some structured data provided by the provider.

Queue Patterns

There are several different types of queue patterns, from publish/subscribe to work/task queues, in this proposal we will just focus on work queues for our service implementation. A publish/subscribe queue should be a different service used for different use cases.

What is a work/task queue?

A work/task queue basically allows for 0 to many consumers/workers to listen to the broker for tasks/jobs and as the producer submits a task/job on the queue the broker checks to see if any consumers/workers are available, if one is available the broker assigns the task/job to the worker/consumer and marks it as busy. This pattern is a stateless processing pattern that does not depend on time as a constraint. If there are no workers/consumers running then the queue just holds the task/job until a time that a worker becomes available. Also, the broker can use common rules to properly manage the queue, like retry a task/job for a number of times before discarding the task/job in a dead letter queue.

Why should hyper63 create a worker queue service?

Queue Service

Queue services introduce another level of complexity and monitoring, ie another moving part, but
with the workers pattern, the development team can implement isolated job processing functions
that are distributed work from the queue.

hyper63 Queue Service

hyper63 can provide a common api abstraction to queue services, so that the implementation team does
not have to deal with the complexity burden of managing these services. The hyper63 queue service
is as easy to use as working with any API service.

The proposed work queue service will enable developers using the hyper63 service to create one to many queues, each queue
can be identified by a name. PUT /queue/{name}. This PUT command will create a queue if it does not exist and return a 200 if it does exist, to create an IDEMPOTENT operation.

What is IDEMPOTENT? is the property of certain operations in mathematics and computer science whereby they can be applied multiple times without changing the result beyond the initial application. https://en.wikipedia.org/wiki/Idempotence

When creating a queue, we need to submit some attributes in a JSON document.

PUT /queue/{name} HTTP/1.1

{
  "worker": "https://example.com/worker",
  "secret": "iamasecret"
}

NOTE: The worker is the target url to submit the queued up tasks/jobs, and the secret is the text used to sign a JWT token that will be submit with the task/job document

Now, I have created my queue, how do I submit work?

To place a job on the queue, the development team using the hyper63 service would invoke a POST /queue/{name} with a JSON Payload. The payload will need to include a JSON Object that contains the structured data that should be passed to the worker.

Running a JOB

The hyper63 queue service will invoke a POST command to the worker web service specified when the queue was created.
Once the worker web service responds with a SUCCESS, then the queue service will wait for the next task/job to be delivered.
If a task/job is delivered before the SUCCESS message, then the queue will hold the job until the previous request is completed.

NOTE: What about long running workers? In this iteration, long-running task will not be addressed, the purpose of this service
is to solve multiple short running less than 3 minute tasks that need to run in background of a server-less request.

For example, we create a queue called hooks and instruct the queue to send any jobs posted to the queue to
the worker endpoint of https://api.engine.com/worker with a POST method and a Bearer token generated using the
provided secret.

PUT /queue/hooks

{
  "worker": 'https://api.engine.com/worker',
  "secret": 'ASHAREDSECRETFORJWT'
}

Then, we can trigger a job, by executing a POST to the hyper63 queue:

POST /queue/{name}

{
...data
}

POST /queue/hooks

{
  "type": "outcome",
  "id": "12345"
}

This trigger will post to the hyper63 queue service in which a worker from an internal fleet of workers
will consume the job and invoke the worker endpoint passing the data json document to that endpoint.

Monitoring and management

• Get Jobs
• Cancel Job
• Retry Job

Summary

In a traditional infrastructure environment, you may utilize a service already built by the infrastructure provider or you may use an open source service that you choose to maintain. The purpose of the hyper63 queue service is to create a pattern that you as the application development team does not have worry or think about servers or clouds, just a common API and hyper63 can leverage the ports and adapters pattern to create an abstraction so that your team can focus on building specific stuff.

Adapters worth considering

• Bee-queue + redis [first implementation]
• RabbitMQ
• AWS SQS
• Celery or Sidekiq

[tripott]

Each queue will need a notification target that every job will invoke with a JSON payload of the data of the job.

• Unclear. Consider rewording or breaking it down into simpler, several sentences.
• Define "job"

[TillaTheHun0]

This sentence also confused me. I agree with this point. Is this to say:

Each queue will have a corresponding target that represents a resource that is able to process a job from its respective queue. A job is a compute task to be handled. When a new job is created on a queue, hyper63 will invoke the resource to process the job, and pass the JSON payload it receives as input to the job.

[tripott]

POST method and a Bearer token generated using the provided secret.

• Will this secret be logged or persisted as part of queue processing and therefore vulnerable?

[twilson63]

It will not be logged, but will be persisted using encryption

[tripott]

Then, we can trigger a job, by executing a POST to the hyper63 queue:

• Will the payload sent to the queue adhere to a schema?
• Consider reviewing GCP Pub/Sub message payload structure to inform the design.
• https://cloud.google.com/pubsub/docs/publisher#message_format

[twilson63]

Payload should be JSON
Payload should be less than 1 MB
And each key on the JSON document should follow the following conventions

• must begin with lowercase letter (a-z)
• must be lowercase letters (a-z)
• digits (0-9)
• any of these characters _ $ + -

[tripott]

As with GCP Pub/Sub, will there be a concept of multiple subscriptions?

https://cloud.google.com/pubsub/docs/admin#manage_subs

[twilson63]

No, this is a standard work queue pattern not a pub/sub pattern, in the future we may abstract a pub/sub pattern but it would be a different hyper63 service and not have conflicting functionality, think SQS and SNS from AWS

[tripott]

• _cancel and _retry are verbs. Is this RESTful? .... :(
• instead of _cancel what about the DELETE /queue/{name}/{id}?
• instead of _retry what about PUT /queue/{name}/{id} with a JSON request body of { "retry": true} ?

[twilson63]

It is RESTful, from experience in the wild for specific action that you want to occur to invoke some sort of behavior.

IE: POST /{db}/_compact
IE: POST [base]/Claim/$submit - https://www.hl7.org/fhir/claim-operation-submit.html

Here is a good explanation:

https://www.hl7.org/fhir/operations.html

The above suggestions seem to be bending the conventional meaning of REST principles, for example, what if you wanted to actually DELETE a queue? or PUT (update) a job? I recommend that we leverage operators with either a prefix of _ or $ to manage actions outside the provided scope of the REST Verbs and save the REST verbs for their intended purpose.

[TillaTheHun0]

hyper63 will wait for that endpoint to resolve with a status. if the status is 200 OK, then hyper63 will mark job as successfully
processed...

Some jobs could take quite a long time to run, more time than what is conventional for an http request. I am thinking jobs for ETL, analytics, etc. Which could result is "false" retries of long running jobs

Maybe there could be some sort of asynchronous way to inform hyper63 that the job is complete? Like a PUT /queue/{name}/{id} with request body of { "complete": true } or PUT /queue/{name}/{id}/_complete? This endpoint could be invoked by the job?

[TillaTheHun0]

I've seen services, like Slack webhooks for example, allow the webhook to respond directly to the POST that Slack sends, within say 60 seconds. Additionally, Slack also sends a unique endpoint, as part of the webhook payload, that can be used to send updates equivalent to what would be sent back to the original POST. So if processing takes longer than 60 seconds, the original POST can respond with a 20x, and then send subsequent updates to the Slack-provided unique endpoint, asynchronously.

[twilson63]

That is cool, but I would say for this iteration, this level of complexity is not needed to ship the concept and let it get some mileage on it, but that could be something that could be implemented if needed over time. I would recommend we invoke the YAGNI principle here and focus on the proposed use case of short less than 3 minute processing times.

[TillaTheHun0]

I agree. It is a use case to be mindful in design, so as to not back the api into a corner, but doesn't need to be implemented as part of the first iteration.

[twilson63]

Job is a json document containing direction for a worker process to perform a specific function or process. Like send an email or log to a remote db. In the hyper63 queue service when create a queue you need to provide a worker target url that will be used to direct incoming jobs. You also can pass a secret, that will be used to to sign a bearer token so that your worker can verify the submission. WIP Thx Tom

…

Sent from my iPhone

On Mar 12, 2021, at 8:55 AM, tripott ***@***.***> wrote:  Each queue will need a notification target that every job will invoke with a JSON payload of the data of the job. Unclear. Consider rewording or breaking it down into simpler, several sentences. Define "job" — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or unsubscribe.

[twilson63]

The secret will be stored in a secured database within the secure hyper63 network Thx Tom

…

Sent from my iPhone

On Mar 12, 2021, at 8:58 AM, tripott ***@***.***> wrote:  POST method and a Bearer token generated using the provided secret. Will this secret be logged or persisted as part of queue processing and therefore vulnerable? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or unsubscribe.

[twilson63]

We can reach out to security experts to measure risk and vulnerability, also we can encrypt the secret in the db using pbkdf2 encryption Thx Tom

…

Sent from my iPhone

On Mar 12, 2021, at 8:58 AM, tripott ***@***.***> wrote:  POST method and a Bearer token generated using the provided secret. Will this secret be logged or persisted as part of queue processing and therefore vulnerable? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or unsubscribe.

[twilson63]

I need to better list the goals, strengths and weaknesses of this queue service iteration. The purpose of this queue is to support publish - worker queue, in which the queue has a set of stateless worker clients. Unlike a pub/sub methodology, this queue system will wait until a worker is available before dispatching a job. Also the goal for this iteration is to focus on short job processing and not long job processing workers. The use case is to handle background tasks like dispatching to a third party analytics system, sms system, email system, or other foreign service system. Thx

…

Sent from my iPhone

On Mar 12, 2021, at 9:02 AM, tripott ***@***.***> wrote:  Then, we can trigger a job, by executing a POST to the hyper63 queue: Will the payload sent to the queue adhere to a schema? Consider reviewing GCP Pub/Sub message payload structure to inform the design. https://cloud.google.com/pubsub/docs/publisher#message_format — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or unsubscribe.

[twilson63]

No, I’m the future we may build a pub/sub service to support those use cases. This service is to support background de coupled http requests Thx Tom

…

Sent from my iPhone

On Mar 12, 2021, at 9:04 AM, tripott ***@***.***> wrote:  As with GCP Pub/Sub, will there be a concept of multiple subscriptions? https://cloud.google.com/pubsub/docs/admin#manage_subs — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or unsubscribe.

[twilson63]

Yes, In rest what is the appropriate syntax for commands that invoke actions ? In fhir they do this with $ and in couch _ and then the verb, open to recommendations, if verb is not standard just following practices I have seen in the wild

…

Sent from my iPhone

On Mar 12, 2021, at 9:06 AM, tripott ***@***.***> wrote:  _retry is a verb.... :( — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or unsubscribe.

[twilson63]

Ok I will try to be more clear. Maybe define th Normal queue flow then the abstraction I am recommending. The main use case for this proposal is to create a generic reusable abstraction to allow hyper63 clients to perform background short lived tasks without having to manage any infrastructure The current client has implemented a hooks interface on their api and is running in a serverless environment Their api has a business rules engine that creates matches based on some criteria, if the match can be made in which sever hooks can be set to dispatch rest calls to third party systems. The match response should return and not wait for the 1 to many hook requests, but since the api is serverless the match function can not dispatch these requests in the background So there needs to be a way for the match process to perform an async process. This proposal is a solution for that workflow, but I believe it can be generalized so that adapters can be made to support different queue implementations Thx Tom

…

Sent from my iPhone

On Mar 12, 2021, at 3:07 PM, Tyler Hall ***@***.***> wrote:  This sentence also confused me. I agree with this point. Is this to say: Each queue will have a corresponding target that represents a resource that is able to process a job from its respective queue. A job is a compute task to be handled. When a new job is created on a queue, hyper63 will invoke the resource to process the job, and pass the JSON payload it receives as input to the job. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or unsubscribe.

[twilson63]

This is a great point, and may need to be addressed if we plan to support long running jobs, or explicitly state this service does not support those workflows

…

Sent from my iPhone

On Mar 12, 2021, at 3:13 PM, Tyler Hall ***@***.***> wrote:  hyper63 will wait for that endpoint to resolve with a status. if the status is 200 OK, then hyper63 will mark job as successfully processed... Some jobs could take quite a long time to run, more time than what is conventional for an http request. I am thinking jobs for ETL, analytics, etc. Which could result is "false" retries of long running jobs Maybe there could be some sort of asynchronous way to inform hyper63 that the job is complete? Like a PUT /queue/{name}/{id} with request body of { "complete": true } or PUT /queue/{name}/{id}/_complete? This endpoint could be invoked by the job? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or unsubscribe.

[twilson63]

Would like to learn more about this and see if this pattern supports this case, maybe we can discuss at the tsc meeting on Tuesday Thx Tom

…

Sent from my iPhone

On Mar 12, 2021, at 3:18 PM, Tyler Hall ***@***.***> wrote:  I've seen services, like Slack webhooks for example, allow the webhook to respond directly to the POST that Slack sends, within say 60 seconds. Additionally, Slack also sends a unique endpoint, as part of the webhook payload, that can be used to send updates equivalent to what would be sent back to the original POST. So if processing takes longer than 60 seconds, the original POST can respond with 200, and then send subsequent updates to the Slack-provided unique endpoint, asynchronously. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or unsubscribe.

[twilson63]

Shipped