Bring back inbox/outbox metrics

[jeremydmiller]

The metrics sampling for the inbox/outbox is gonna go all to hell with multi-tenancy. Just bring it back per database later.

See:

internal class MetricsCalculator : IDurabilityAction
{
    private readonly ObservableGauge<int> _incoming;
    private readonly ObservableGauge<int> _outgoing;
    private readonly ObservableGauge<int> _scheduled;

    public MetricsCalculator(Meter meter)
    {
        _incoming = meter.CreateObservableGauge(MetricsConstants.InboxCount, () => Counts.Incoming,
            MetricsConstants.Messages, "Inbox messages");
        _outgoing = meter.CreateObservableGauge(MetricsConstants.OutboxCount, () => Counts.Outgoing,
            MetricsConstants.Messages, "Outbox messages");
        _scheduled = meter.CreateObservableGauge(MetricsConstants.ScheduledCount, () => Counts.Scheduled,
            MetricsConstants.Messages, "Scheduled messages");
    }

    public PersistedCounts Counts { get; private set; } = new();

    public string Description { get; } = "Metrics collection of inbox and outbox";

    public async Task ExecuteAsync(IMessageDatabase database, IDurabilityAgent agent,
        IDurableStorageSession session)
    {
        var counts = await database.FetchCountsAsync();
        Counts = counts;
    }
}

[dgrozenok]

Would be great to have those metrics working. Need to monitor the inbox/outbox counters while troubleshooting the messages not being processed timely.

[jeremydmiller]

Sorry Denys, these are getting bounced back to 4.0:(

Also, add this back to the docs:

| wolverine-inbox-count        | [Observable Gauge](https://opentelemetry.io/docs/reference/specification/metrics/api/#asynchronous-gauge) | Samples the number of pending envelopes in the durable inbox (likely to change)                                                                                                                                                                                                        |
| wolverine-outbox-count       | Observable Gauge                                                                                          | Samples the number of pending envelopes in the durable outbox (likely to change)                                                                                                                                                                                                       |
| wolverine-scheduled-count    | Observable Gauge                                                                                          | Samples the number of pending scheduled envelopes in the durable inbox (likely to change)                                                                                                                                                                                              |

[jeremydmiller]

Picking this up a bit today.

• Bury this into each IMessageStore. Use tenant database names if necessary
• Add a UpdateMetricsPeriod to DurabilitySettings with a default value of 5 seconds.
• New class that uses a timer for non-database things like RavenDb?
• Utilize DurabilityAgent for databases