[Databuckets] Improved Reliability and Performance of Databuckets

[Akkadius]

Description

This PR aims to solve the following -

• Performance - Address observed high-scale performance issues of the distributed cache model under higher load scenarios
• Reliability - Address rarer inconsistencies of out of sync distributed cache. Data guaranteed reliability is a high priority
• Code Simplicity - By addressing the two we have big opportunities to simplify some of the code and make it easier to reason about

With this PR, distributed caching has been removed. The distributed caching was beneficial before the scoped caching model was introduced, but also the complexity was not worth the observed performance issues and data reliability issues observed at scale since the initial implementation.

Changes

• Distributed caching has been removed entirely, massively reducing World network strain and (N x zoneserver) network load
• Client databucket cache is now cleared from the zone when a client zones
• Client databucket cache is now cleared from the zone when a client loads in a zone
• Bot databucket cache is now cleared from the zone when a client zones
• Bot databucket cache is cleared from the zone when it is newly spawned
• Simplify the cache code to no longer include an updated_timestamp and update_type wrapper type and now we simply use the native DataBucketsRepository::DataBuckets type
• We no longer bulk load databuckets for NPCs on zone spawn because they no longer get cached local or distributed
• We only cache misses as defined by entities scoped in DataBucket::CanCache

Stays the same

• Bulk load cache for Client and Bot (defined by DataBucket::CanCache)
• Cache local to zone for Client and Bot (defined by DataBucket::CanCache)

Type of change

• Bug fix (non-breaking change which fixes an issue)
• Performance Improvement

Testing

This has been tested on THJ for a few weeks now and has dramatically reduced strain on World sending updates to 700x+ zoneservers. It has posed no issues.

select * from data_buckets where character_id = 1;
+----+----------+-----------+---------+--------------+--------+--------+
| id | key      | value     | expires | character_id | npc_id | bot_id |
+----+----------+-----------+---------+--------------+--------+--------+
|  2 | test_key | something |       0 |            1 |      0 |      0 |
+----+----------+-----------+---------+--------------+--------+--------+

sub EVENT_SAY {
    if ($text=~/hail/i) {
        $client->Message(15, $client->GetBucket("test_key"));
    }
}

Below you can see the first cache miss, returning from database and then subsequent cache return. (No queries)

  Zone | DataBucket | GetData Getting bucket key [test_key] bot_id [0] character_id [1] npc_id [0] -- [overthere] (The Overthere) inst_id [0]
  Zone | DataBucket | GetData Getting bucket key [test_key] bot_id [0] character_id [1] npc_id [0] -- [overthere] (The Overthere) inst_id [0]
  Zone | DataBucket | GetData Returning key [test_key] value [something] from cache -- [overthere] (The Overthere) inst_id [0]
  Zone | DataBucket | GetData Getting bucket key [test_key] bot_id [0] character_id [1] npc_id [0] -- [overthere] (The Overthere) inst_id [0]
  Zone | DataBucket | GetData Returning key [test_key] value [something] from cache -- [overthere] (The Overthere) inst_id [0]
  Zone | DataBucket | GetData Getting bucket key [test_key] bot_id [0] character_id [1] npc_id [0] -- [overthere] (The Overthere) inst_id [0]
  Zone | DataBucket | GetData Returning key [test_key] value [something] from cache -- [overthere] (The Overthere) inst_id [0]
  Zone | DataBucket | GetData Getting bucket key [test_key] bot_id [0] character_id [1] npc_id [0] -- [overthere] (The Overthere) inst_id [0]

When a client camps / zones, the cached keys are purged correctly.

   Zone | DataBucket | DeleteCachedBuckets LoadType [Client] id [1] cache size before [2] after [1] -- [overthere] (The Overthere) inst_id [0]

Below is a test of a global bucket (doesn't cache so performs queries)

  Zone | DataBucket | GetData Getting bucket key [test_key_global] bot_id [0] character_id [0] npc_id [0] -- [overthere] (The Overthere) inst_id [0]
  Zone |   Query    | QueryDatabase SELECT id, `key`, value, expires, character_id, npc_id, bot_id FROM data_buckets WHERE character_id = 0 AND npc_id = 0 AND bot_id = 0 AND `key` = 'test_key_global' LIMIT 1 -- (1 row returned) (0.000061s)-- [overthere] (The Overthere) inst_id [0]

Checklist

• I have tested my changes
• I have performed a self-review of my code. Ensuring variables, functions and methods are named in a human-readable way, comments are added only where naming of variables, functions and methods can't give enough context.
• I own the changes of my code and take responsibility for the potential issues that occur

[catapultam-habeo]

This works as advertised. Noticeable improvement in performance, and several recurring but relatively inconsistent issues around unscoped bucket reliability have disappeared since implementing this, with no clear downsides.