There is a simple fix for this, simply clearing the buffer local keymaps tree by adding the following single line of code

  M.get_tree(mode, buf).tree = Tree:new()

Specifically adding this to the file lua/which-key/keys.lua in the function responsible for updating keys function M.update_keymaps(mode, buf)

NOTE: however that doing this naive change seems to cause nvim to hang and become unresponsive. The cause seems to be that in addition to the reset of the tree, we need to also delete the "hooks". Will be attempting to do this as well.

After some more investigation I've found that this also happens with non local keymaps.

I found something that seems to work , will attempt making a PR soon

function M.update_keymaps(mode, buf)
  ---@type Keymap[]
  local keymaps = buf and vim.api.nvim_buf_get_keymap(buf, mode) or vim.api.nvim_get_keymap(mode)

  -- Clearing up previous tree entries and hooks.
  -- Forgetting to do this will break which-key for users that register and de register
  -- keymaps in different contexts
  do
    local function unhook(node)
      if node.mapping and M.is_hooked(node.mapping.prefix, mode, buf) then
        M.hook_del(node.mapping.prefix, mode, buf)
      end
    end
    M.get_tree(mode, buf).tree:walk(unhook)
    M.get_tree(mode, buf).tree = Tree:new()
  end

I can confirm this issue through trying to resolve LazyVim/LazyVim#3863. Even with the neovim-default keymaps removed through nunmap / vim.keymap.del, which-key retains them and causes conflicts when activated.

I'll see if I can find some time today to fully test that PR.

which-key is quite fragile (was my first plugin), so I'm always a bit hesitant to merge a PR.