Communication Protocol

[sharpener6]

Introduction

This issue is meant to standardize and maintain the scaler communication protocol across butlers, clients, scheduler and workers, which will enable scaler become language independent mechanism to talk, following connection should work:
client (python) <-> scheduler (c++) <-> workers (python)

Transportation Layer

The client <-> scheduler and scheduler <-> worker is fully use TCP under ZeroMQ until March, 2025, after March 2025, zmq will get abandoned and pure TCP based communication will be used, any language implementation with TCP connection and re-connect mechanism should work.

Identity

Each connection peer should have an identity and should use some human readable string and include uuid4 string to avoid identity conflicts, identity is important as it will be used to identify peers and scheduler will use it to routing the messages, normally the first message when connecting will send the identity message to scheduler for showing connected peer identity

Message

Encoding/Decoding

All messages are using Cap'np for encoding, a typical Task message format will be like this:

struct Task {
    taskId @0 :Data;
    source @1 :Data;
    metadata @2 :Data;
    funcObjectId @3 :Data;
    functionArgs @4 :List(Argument);

    struct Argument {
        type @0 :ArgumentType;
        data @1 :Data;

        enum ArgumentType {
            task @0;
            objectID @1;
        }
    }
}

please see here for all messages fields, all implementation should use this for encoding/decoding messages

In general, the message type is in 2 big categories: object channel, and task channel,

Object Channel

Currently Object channel is communicated with scheduler, means scheduler is serve as storage services, in the future, this will be isolated from scheduler and working as independent service, but the address of object storage service will be retrieve from scheduler

Task Channel

Task channel includes actual Task Messaging mechanism and state machine