Skip to content
Open
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
61 changes: 61 additions & 0 deletions 20251028-RS-workflow-list.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,61 @@
# Workflow List and History RPCs for Durable Task

* Author(s): @joshvanl

## Overview

This proposal adds two new RPCs to the durabletask framework which are used to support observability and discoverability of running and completed workflows in Dapr.
Specifically, adding a `ListInstances` and `GetInstanceHistory` RPC to the durabletask gRPC service.

## Background

Today, there is no ways of discovering the list of workflow instances that are currently running or have completed in the past without using external storage queries.
The Dapr CLI [introduced list and workflow history commands](https://github.com/dapr/cli/pull/1560) to get information about running and completed workflows, however these commands rely on direct queries to the underlying storage provider.
By introducing this functionality into the durabletask framework itself, these commands need only talk to Daprd, removing the requirement for direct access to the storage provider as well as authentication.
Daprd can make these queries itself, and use the Actor State Store component to access the underlying storage.

## Related Items

## Implementation Details

### Design

Two new RPCs will be added to the durabletask gRPC service, which will be available to both the application client, as well as the dapr CLI.
The list RPC will be used to discover the workflow instance IDs, which their metadatda can then be fetched.
The workflow history RPC will be used to fetch the full history of a given workflow instance.

```proto
service TaskHubSidecarService {
rpc ListInstances (ListInstancesRequest) returns (ListInstancesResponse);
rpc GetInstanceHistory (GetInstanceHistoryRequest) returns (GetInstanceHistoryResponse);
}

// ListInstancesRequest is used to list all orchestration instances.
message ListInstancesRequest {
// continuationToken is the continuation token to use for pagination. This
// is the index which the next page should start from. If not given, the first
// page will be returned.
optional uint32 continuationToken = 1;

// pageSize is the maximum number of instances to return for this page. If
// not given, all instances will be attempted to be returned.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it'd be safer to set a maximum allowed value.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What would you consider safe in this context?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

100? completely arbitrary, but seems large enough to be useful and small enough to not stress the system.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How about 1024? 😄

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you think ~1000 items are manageable? It feels like a bit too much for me. Github API normally limits to 100 per page, with a default of 30. But their payloads are big. What do you suggest? No limit and just return them all?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I should think so- we are only returning the key strings, no the value payloads.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In that case I think we can keep it high, fine with the 1000 if you feel it's fine, but I'd enforce a limit just to protect us from situations where there are just too many keys to return.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should it not be the case that the user should enforce that limit? They have the controls with the page size and continuation token.

optional uint32 pageSize = 2;
}

// ListInstancesResponse is the response to executing ListInstances.
message ListInstancesResponse {
// instanceIds is the list of instance IDs returned.
repeated string instanceIds = 1;
}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would it be useful to include something about the instance in this message? Maybe the name of the workflow at least?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also, shouldn't this return a continuationId for the pagination?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Because we store workflows in a key value store, with the instance ID being in the key, we only have access to that in a simple table scan without decoding each key. The users of this API will do follow up GetInstanceIDHistory or metadata lookups for each instance ID.

We don't want to keep any inter-request state on the server side, so it is up to the client to calculate this- it being the continuationToken+pageSize. If len(instanceIds) < pageSize then the client knows it has reached the end of the full list.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

After think about this over the weekend and doing some implementations, I agree it is correct that we return a continuation token on the response! 🙂


// GetInstanceHistoryRequest is used to get the full history of an
// orchestration instance.
message GetInstanceHistoryRequest {
string instanceId = 1;
}

// GetInstanceHistoryResponse is the response to executing GetInstanceHistory.
message GetInstanceHistoryResponse {
repeated HistoryEvent events = 1;
}
```