Fix multiple workflows and add proper engine persistance

[LuisDuarte1]

Fixes #7127.
Closes WOR-322, WOR-395

Previously, engine did store everything in-memory - this is not expected behaviour since R2/KV/DOs all have persistenece, by default.

Also, multiple workflow definitions were broken since the DO unique key was a constant (therefore it would fail when workerd launched)

---

• Tests
  • TODO (before merge)
  • Tests included
  • Tests not necessary because:
• E2E Tests CI Job required? (Use "e2e" label or ask maintainer to run separately)
  • I don't know
  • Required
  • Not required because:
• Public documentation
  • TODO (before merge)
  • Cloudflare docs PR(s):
  • Documentation not necessary because: there isn't anything to document (except changelog, which will come later) - this is expected behavior

[changeset-bot]

🦋 Changeset detected

Latest commit: 4a17f40

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 5 packages

Name	Type
@cloudflare/workflows-shared	Minor
miniflare	Minor
@cloudflare/pages-shared	Patch
@cloudflare/vitest-pool-workers	Patch
wrangler	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

[github-actions]

A wrangler prerelease is available for testing. You can install this latest build in your project with:

npm install --save-dev https://prerelease-registry.devprod.cloudflare.dev/workers-sdk/runs/11940185264/npm-package-wrangler-7286

You can reference the automatically updated head of this PR with:

npm install --save-dev https://prerelease-registry.devprod.cloudflare.dev/workers-sdk/prs/7286/npm-package-wrangler-7286

Or you can use npx with this latest build directly:

npx https://prerelease-registry.devprod.cloudflare.dev/workers-sdk/runs/11940185264/npm-package-wrangler-7286 dev path/to/script.js

Additional artifacts:

npx https://prerelease-registry.devprod.cloudflare.dev/workers-sdk/runs/11940185264/npm-package-create-cloudflare-7286 --no-auto-update

npm install https://prerelease-registry.devprod.cloudflare.dev/workers-sdk/runs/11940185264/npm-package-cloudflare-kv-asset-handler-7286

npm install https://prerelease-registry.devprod.cloudflare.dev/workers-sdk/runs/11940185264/npm-package-miniflare-7286

npm install https://prerelease-registry.devprod.cloudflare.dev/workers-sdk/runs/11940185264/npm-package-cloudflare-pages-shared-7286

npm install https://prerelease-registry.devprod.cloudflare.dev/workers-sdk/runs/11940185264/npm-package-cloudflare-vitest-pool-workers-7286

npm install https://prerelease-registry.devprod.cloudflare.dev/workers-sdk/runs/11940185264/npm-package-cloudflare-workers-editor-shared-7286

npm install https://prerelease-registry.devprod.cloudflare.dev/workers-sdk/runs/11940185264/npm-package-cloudflare-workers-shared-7286

npm install https://prerelease-registry.devprod.cloudflare.dev/workers-sdk/runs/11940185264/npm-package-cloudflare-workflows-shared-7286

Note that these links will no longer work once the GitHub Actions artifact expires.

---

[email protected] includes the following runtime dependencies:

Package	Constraint	Resolved
miniflare	workspace:*	3.20241106.0
workerd	1.20241106.1	1.20241106.1
workerd --version	1.20241106.1	2024-11-06

Please ensure constraints are pinned, and miniflare/workerd minor versions match.

[emily-shen]

sanity check for me: why does each workflow need its own storage service? as far as i can tell all other plugins use one storage service for all instances.

[LuisDuarte1]

It was personal preference mostly - this way each workflow gets its own folder in .wrangler/workflow and makes it way easier to debug the engine state

[masylum]

Will adding persistence enable workflows to return serializable values like steps do? This would be great to compose workflows.

[LuisDuarte1]

Will adding persistence enable workflows to return serializable values like steps do? This would be great to compose workflows.

I'm sorry, not sure what you mean by that - afaik you shouldn't have any problem composing workflows atm

[masylum]

Will adding persistence enable workflows to return serializable values like steps do? This would be great to compose workflows.

I'm sorry, not sure what you mean by that - afaik you shouldn't have any problem composing workflows atm

I meant being able to call a workflow and get the return value. Right now you can only compose by having them triggering each other, like queues. But it would be great to be able to have them act like steps, with a return value. This is how it's implemented at inngest: https://www.inngest.com/docs/reference/functions/step-invoke

[LuisDuarte1]

I meant being able to call a workflow and get the return value. Right now you can only compose by having them triggering each other, like queues. But it would be great to be able to have them act like steps, with a return value. This is how it's implemented at inngest: https://www.inngest.com/docs/reference/functions/step-invoke

That is unrelated to the PR - but you can already do this - you can pass the workflow output as an event payload to the next. What you are describing is syntactic sugar for that, which we might (or might not) consider in the future 😄 thanks ^^

[emily-shen]

looks good from the wrangler side pending a tiny fixup

[davidhollenbeckx]

@LuisDuarte1 you're my hero man. multiple workflows on local working great now!