Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 22 additions & 2 deletions docs/guide/durability/cosmosdb.md
Original file line number Diff line number Diff line change
Expand Up @@ -147,8 +147,6 @@ Wolverine uses a single CosmosDB container, named `wolverine` by default, with a
`/partitionKey`. The container is automatically created during database migration if it does not exist, but the
database it lives in is not: create the database ahead of time.

<!-- DRAFT (GH-5059, for Jeremy to word): incoming envelope documents are partitioned by their message identity under the default MessageIdentity.IdOnly (one logical partition per message, so the same message id received at two addresses is rejected as a duplicate), and by destination under MessageIdentity.IdAndDestination. Documents written by earlier versions were partitioned by destination; they are still found, updated and deleted, and drain away normally. While they do, a redelivery of one of those older messages is not recognized as a duplicate. -->

All Wolverine document types are stored in the same container, differentiated by a `docType` field:
- `incoming` - Incoming message envelopes
- `outgoing` - Outgoing message envelopes
Expand All @@ -160,6 +158,28 @@ All Wolverine document types are stored in the same container, differentiated by
- `agent-restriction` - Agent restriction documents
- `lock` - Distributed lock documents

### How Incoming Envelopes Are Partitioned

Cosmos DB only enforces a document id as unique *within one logical partition*, so the partition key Wolverine picks for
an incoming envelope document decides what the inbox can recognize as a duplicate. Wolverine follows your
[message identity](/guide/durability/#message-identity) setting here:

* Under the default `MessageIdentity.IdOnly`, an incoming envelope document is partitioned by the message's identity, so
every message gets its own logical partition. The same message id received at two different addresses is the same
message, and Cosmos DB rejects the second copy as a duplicate just like the relational message stores do. As a bonus,
that spreads the inbox evenly across partitions
* Under `MessageIdentity.IdAndDestination`, the destination is part of the message identity, so the incoming envelope
documents are partitioned by destination

::: info
Versions of Wolverine before 6.51 partitioned every incoming envelope document by its destination. If you're upgrading
with messages still sitting in the inbox, you don't need to do anything. Wolverine still finds, updates, and deletes
those older documents where they are, and they drain away through normal processing. The one gap while they do is that a
redelivery of one of those older messages isn't rejected when it's stored. The new copy is written to the new partition,
the older document is still sitting in its destination partition, and Cosmos DB only sees a conflict within the same
partition.
:::

### Several Applications in One Database <Badge type="tip" text="6.45" />

The container is the unit of clustering. The node registry, leadership, agent assignments and the single
Expand Down
40 changes: 39 additions & 1 deletion docs/guide/serverless.md
Original file line number Diff line number Diff line change
Expand Up @@ -215,6 +215,24 @@ expect a lot of rows in that table, you can opt into an index on that column wit
as shown in the sample above. That's off by default so that upgrading Wolverine doesn't change your database schema unless you
ask it to. Every relational message store supports the index. RavenDb builds an auto index for that query on its own and Cosmos DB indexes every property by default, so the flag doesn't change anything for those two stores.

### Moving an Existing Database to ServerlessWithOutbox

That `timestamp` column isn't always there. The relational message stores only add a `timestamp` column to the outgoing
envelope table when `OutboxStaleTime` has a value, and to the incoming envelope table when `InboxStaleTime` has a value.
Since `ServerlessWithOutbox` gives both of those a default, switching an existing application over to this mode is a
schema change to your message storage, even if you never touch either setting yourself.

If Wolverine is allowed to migrate the message storage at startup, you won't notice anything. But a lot of folks
running in serverless functions turn that off with `AutoBuildMessageStorageOnStartup = AutoCreate.None` to keep
migrations out of the cold start, and that's exactly the case to watch out for. With `AutoCreate.None`, Wolverine
only logs a warning at startup that the message storage is out of date and leaves your schema alone. The durability pass
reads and updates that `timestamp` column to find and recover the stale outbox rows, so that part of the pass fails every
single time until the schema is migrated, and the messages whose inline send failed are never recovered.

The fix is to migrate the message storage as part of your deployment, before the new version of the function takes any
traffic. `dotnet run -- resources setup` always applies the message storage migration, even when `AutoCreate` is `None`.
See [Disable Automatic Storage Migration](/guide/durability/managing#disable-automatic-storage-migration) for more on that.

## The Durability Pass

Since there's no background durability agent in `ServerlessWithOutbox`, *something* outside of your function has to periodically
Expand Down Expand Up @@ -444,6 +462,27 @@ is to not `AutoProvision()` the listener subscriptions on the push host at all.
where this is headed.
:::

## Provisioning Broker Topology

`AutoProvision()` is a great convenience for a long running process that starts up once and then runs for days. In a
serverless function though, every cold start is an application startup. With RabbitMQ for example, an application using
`AutoProvision()` declares every exchange, queue, and binding it knows about each time a new instance spins up, and those
are extra broker round trips sitting right in the middle of the cold start you're trying so hard to keep short.

Our recommendation is to leave `AutoProvision()` off inside your functions and to provision the broker topology outside of
them instead. That can be `dotnet run -- resources setup` run as a deployment step, which sets up the broker objects
whether or not `AutoProvision()` is turned on and migrates your message storage at the same time, or it can be Terraform,
Pulumi, or whatever you already use for the rest of your infrastructure.

Under `ServerlessWithOutbox` with RabbitMQ, this isn't just about performance. Since Wolverine publishes to queues as
*mandatory* in that mode, a send to a queue that doesn't exist yet fails, and the message waits in the outbox until a
durability pass runs after the queue finally does exist. So get the topology in place before the function takes traffic.

::: tip
Neither serverless mode starts any listeners, but the RabbitMQ transport still opens a separate listening connection by
default. Add `UseSenderConnectionOnly()` to your `UseRabbitMq()` configuration to skip that connection on every cold start.
:::

## Pre-Generate All Types

The runtime code generation that Wolverine does comes with a potentially non-trivial "cold start" problem with its first
Expand All @@ -457,7 +496,6 @@ Just so you know going in, neither `Serverless` nor `ServerlessWithOutbox` suppo
* Local queues. Under plain `Serverless` a message this application handles is dropped, so use `InvokeAsync()` for in-process work instead or send
the message to an external broker. `ServerlessWithOutbox` replaces local queues with the inbox: a published or cascaded message this application
handles is stored there and handled by the next durability pass, so expect latency up to your pass interval

* Listeners on external brokers. See [Receiving Messages by Push](#receiving-messages-by-push) instead. Provisioning the listening side of a broker
still runs though, so `AutoProvision()` of Pub/Sub subscriptions creates pull subscriptions that nothing ever reads. The push subscriptions themselves
are provisioned outside of Wolverine, as described under [Google Pub/Sub Push Subscriptions](#google-pub-sub-push-subscriptions)
Expand Down