Adds RPCs for workflow list and get history

[JoshVanL]

No description provided.

[acroca]

I think it'd be safer to set a maximum allowed value.

[JoshVanL]

What would you consider safe in this context?

[acroca]

100? completely arbitrary, but seems large enough to be useful and small enough to not stress the system.

[JoshVanL]

How about 1024? 😄

[acroca]

Do you think ~1000 items are manageable? It feels like a bit too much for me. Github API normally limits to 100 per page, with a default of 30. But their payloads are big. What do you suggest? No limit and just return them all?

[JoshVanL]

I should think so- we are only returning the key strings, no the value payloads.

[acroca]

In that case I think we can keep it high, fine with the 1000 if you feel it's fine, but I'd enforce a limit just to protect us from situations where there are just too many keys to return.

[JoshVanL]

Should it not be the case that the user should enforce that limit? They have the controls with the page size and continuation token.

[acroca]

Would it be useful to include something about the instance in this message? Maybe the name of the workflow at least?

[acroca]

Also, shouldn't this return a continuationId for the pagination?

[JoshVanL]

Because we store workflows in a key value store, with the instance ID being in the key, we only have access to that in a simple table scan without decoding each key. The users of this API will do follow up GetInstanceIDHistory or metadata lookups for each instance ID.

We don't want to keep any inter-request state on the server side, so it is up to the client to calculate this- it being the continuationToken+pageSize. If len(instanceIds) < pageSize then the client knows it has reached the end of the full list.

[JoshVanL]

After think about this over the weekend and doing some implementations, I agree it is correct that we return a continuation token on the response! 🙂

[cicoyle]

I'm mostly good with this proposal and think it's a great value add. Once workflow versioning is supported, will we need any tweaking to include the workflow version in the list or history views?

Also, thoughts on changing instanceIds -> instanceIDs && instanceId -> instanceID?

[olitomlinson]

I would prefer we take the approach of using the Event Streaming proposal over building workflow list capabilities (driven by my initial Workflows lifecycle event feature request)

I appreciate this might be an effective solution for small workloads (A couple of hundred workflow instances) but, paging an unbounded set of Workflow instances is going to be a nightmare at scale, Continuation Tokens are okay, but they're nothing more than the encoding of a page size, and offset to move through a set.

But we know that Wokflow statuses change over time so forward-only traversal of a set via paging is going to miss workflows where the status has changed. At this point you have to start going down the Change Data Capture/Feed route, which is a whole 'nother thing! Look at Cosmosdbs Change Data Feed for example.

Far simpler IMO to move forward with the Event Streaming proposal, which gives people an evented model for handling changes, rather than the proposed polling/list model. And can be used in many different ways to satisify many different use-cases (as described in my Workflows lifecycle event feature request)

Btw is not just running and completed workflows that folks are interested in discovering, it could be workflows that are in any of the available states.

---

I appreciate that both this proposal and the Event Streaming proposal can co-exist and be delivered as two separate things without any problems. I just feel we get more mileage out of the Event Streaming, hence my preference. That said, this proposal still gets my non-binding vote of approval, despite its inherent complexities at scale.

[JoshVanL]

I would prefer we take the approach of using the Event Streaming proposal over building workflow list capabilities (driven by my initial Workflows lifecycle event feature request)

I appreciate this might be an effective solution for small workloads (A couple of hundred workflow instances) but, paging an unbounded set of Workflow instances is going to be a nightmare at scale, Continuation Tokens are okay, but they're nothing more than the encoding of a page size, and offset to move through a set.

But we know that Wokflow statuses change over time so forward-only traversal of a set via paging is going to miss workflows where the status has changed. At this point you have to start going down the Change Data Capture/Feed route, which is a whole 'nother thing! Look at Cosmosdbs Change Data Feed for example.

Far simpler IMO to move forward with the Event Streaming proposal, which gives people an evented model for handling changes, rather than the proposed polling/list model. And can be used in many different ways to satisify many different use-cases (as described in my Workflows lifecycle event feature request)

Btw is not just running and completed workflows that folks are interested in discovering, it could be workflows that are in any of the available states.

I appreciate that both this proposal and the Event Streaming proposal can co-exist and be delivered as two separate things without any problems. I just feel we get more mileage out of the Event Streaming, hence my preference. That said, this proposal still gets my non-binding vote of approval, despite its inherent complexities at scale.

As you say, this proposal is in no way trying to supersede or replace the event streaming feature proposal.
This functionality is specifically being added to support the CLI commands dapr workflow list etc. without needing to give any db connection string to the CLI from the user. Adding event streaming would not solve this problem.

[cicoyle]

+1 binding