-
Notifications
You must be signed in to change notification settings - Fork 32
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Introduce runners #230
Comments
Overall it looks good to me, I have some remarks though
|
Agree, I think "runner" is a better name for this component. Let's use that. I like gRPC because it's self documenting, it forces us to version the API following best practices, and we can generate clients automatically. |
I'm going to sketch out the runner gRPC Service so that we can start working on an implementation. Indexer resourceTerminology:
message Indexer {
message Status {
}
// Resource name, e.g. `indexers/my-indexer`.
string name = 1;
// Additional labels attached to the indexer.
// Useful to attach application-specific metadata.
map<string, string> labels = 2;
} Open Questions: will edit later.
OperationsCreateThis method creates a new indexer if one with the same Notice that if the client provides a value for fields that are set server-side service IndexerRunner {
rpc CreateIndexer(CreateIndexerRequest) returns (Indexer);
}
message CreateIndexeRequest {
// Indexer id, e.g. `my-indexer`.
string indexer_id = 1;
Indexer indexer = 2;
} DeleteThis method deletes the indexer. If persistence for the indexers is configured, service IndexerRunner {
rpc DeleteIndexer(DeleteIndexerRequest) returns (google.protobuf.Empty);
}
message DeleteIndexerRequest {
// Indexer name, e.g. `indexers/my-indexer`.
string name = 1;
} GetThis method simply gets an indexer by its name. service IndexerRunner {
rpc GetIndexer(GetIndexerRequest) returns (Indexer);
}
message GetIndexerRequest {
// Indexer name, e.g. `indexers/my-indexer`.
string name = 1;
} ListList all indexers according to some criteria. Returns a paginated list of indexers. We implement filtering based on AIP 160. The idea Since filtering is complex, we skip it at first. service IndexerRunner {
rpc ListIndexers(ListIndexersRequest) returns (ListIndexersResponse);
}
message ListIndexersRequest {
// Number of indexers per page.
int32 page_size = 1;
// Continuation token.
string page_token = 2;
// Filter indexers.
string filter = 3;
}
message ListIndexersResponse {
repeated Indexer indexers = 1;
string next_page_token = 2;
} Stream LogsThis method returns a stream of logs for the indexer. It is an infinite stream service IndexerRunner {
rpc StreamLogs(StreamLogsRequest) returns (stream StreamLogsResponse);
}
enum LogLevel {
LOG_LEVEL_UNKNOWN = 0;
LOG_LEVEL_TRACE = 1;
LOG_LEVEL_DEBUG = 2;
LOG_LEVEL_INFO = 3;
LOG_LEVEL_WARNING = 4;
LOG_LEVEL_ERROR = 5;
}
message StreamLogsRequest {
// The name of the indexer, e.g. `indexers/my-indexer`.
string parent = 1;
LogLevel level = 2;
}
message StreamLogsResponse {
LogLevel level = 1;
string content = 2;
} Indexers persistenceThe runner is responsible for setting up the indexers persistence. This is for several reasons:
Runner persistenceIn some cases, the runner needs to keep track of the indexers it created. I
Some runners (like the one based on Kubernetes) can use other persistence mechanism. Other considerationsThis service does not deal with authentication or authorization. Deployments |
Re: how to specify indexer script. We add two new properties to the indexer:
The indexer path is then compute as In practiceapibara up Creates new indexers as defined in the configuration. indexer factory By default, in both cases When they call the api to create an indexer, they forwardsthe current |
re: delete an indexer and its children The easiest solution is to add a On delete, the runner goes through all indexers where Note that we cannot use the name |
This issue has been automatically marked as stale because it has not had activity in the six months. It will be closed in 2 weeks if no further activity occurs. Please feel free to leave a comment if you believe the issue is still relevant. |
Is your feature request related to a problem? Please describe.
As seen in #226 and #227, we need a way to dynamically spinup indexers. This system needs to be robust enough for production usage, and support different deployment models (Kubernetes, Docker, bare metal).
Describe the solution you'd like
The idea is to introduce schedulers. A scheduler is a server (gRPC for now, + REST later) that exposes a CRUD interface for indexers, and schedules them to run in the background. The scheduler is also used to show indexing status (see #229) and logs.
Notice that the
CreateIndexer
operation is idempotent: if an indexer with the same indexerdefinition.id
already exists, it will return the existing indexer without creating a new one.See message below for a first draft of the implementation
The idea is to include at least two implementations of the scheduler:
After we have this, implementing #226 and #227 becomes easy:
apibara up
command to run multiple indexers at the same time #226: for each indexer in the config, callCreateIndexer
. No need to check if the indexer exists since operations are idempotent.apibara down
will callDeleteIndexer
.CreateIndexer
with the returned value. This becomes a sink like any other.Additional context
The idea to use an API to schedule factory indexers comes from a telegram chat with @bigherc18
The text was updated successfully, but these errors were encountered: