Allow hook chaining for the htlc_accepted hook

[cdecker]

This pull request creates the infrastructure for registering multiple plugins
for a given hook and showcases this new capability with the htlc_accepted
hook. The htlc_accepted hook was chosen due to its relative simple semantics
and because it currently is the hook with the most plugins making.

Before you could only use a single plugin that registers the htlc_accepted
hook at a time, which was severely limiting its uses. With this you can
finally run multiple at the same time.

The htlc_accepted hook returns one of the following three outcomes:

• {"result": "continue"}: the plugin signals that it could not handle the
  HTLC internally, so the chain continues. If there are more plugins that
  have registered for this hook they get called, otherwise we handle
  internally.
• {"result": "fail", ...}: the plugin decided that the HTLC should be
  rejected, with the given parameters. This exits the call chain and fails
  the HTLC. If there are more plugins they will not get called.
• {"result": "resolve", ...}: similar to the previous case, the plugin
  knows the preimage and choses to resolve this HTLC. The call chain is
  exited, and no further plugins are called.

Future steps

The following details are as of yet not addressed in this pull request, but I
plan to continue working on these in new PRs:

• Allow plugins to provide a numeric priority which determines the order in
  which the plugins should be called
• Allow more hooks to be chained. We currently only enable htlc_accepted to
  register multiple times, and we will enable more as the need arises and the
  semantics are clarified.
• Potentially we could add a fanout type that calls all hooks in parallel,
  and collected the results again. One potential candidate is the db_write
  hook whose results can be combined into all continues or at least one
  fail.

---

Related issues

• Allow plugin hook chaining (Allow plugin hook chaining #2796)
• Bitcoin backend plugins planning (Bitcoin backend plugins planning #3354)
• Add a sendcustommsg RPC call and a custommsg plugin hook for
  experimental protocol extensions (Add a sendcustommsg RPC call and a custommsg plugin hook for experimental protocol extensions #3315)

[darosior]

ACK 7367597

Tested with hacky jitrebalance.py + noise.py combo tests.

Unrelated Travis spurious failure (actually common on all PRs)

[rustyrussell]

Think we might now be buggy if we remove a plugin while it's supposed to be answering a hook? Request used to be freed previously, what now?

There are several possible fixes for plugins being removed while we're calling hooks.

The most specific would be to re-search hook.plugins for the plugin we were just on, and call the next one (O(n^2) but n is tiny). But the plugin could have been freed (unless we fix the above). We could attach a u64 for each hook.plugins[] entry (assigned off some global counter), and look for the next u64 instead.

[ZmnSCPxj]

Think we might now be buggy if we remove a plugin while it's supposed to be answering a hook? Request used to be freed previously, what now?

What I was planning for this was that invocation of a hook would create a new list of plugins currently registered on that hook. Each list element would have a pointer to the plugin and a pointer to the hook invocation; the list element would be tallocated off the hook invocation object, but would have a tal_destructor2 triggered by detsruction of the plugin object. If the plugin object is destroyed, we check if the hook invocation was currently calling into the plugin, and schedule the hook invocation to call into the next plugin; regardless we would then remove and tal_free the list element from the hook invocation list. When hook chaining completes --- if a plugin returns non-{'result':'continue'} or all plugins have been invoked --- then we unregister the detructors on the plugins and destroy the hook invocation object (could be a tal_destructor on the hook invocation object). This allows multiple parallel invocations of the hook (each invocation has its own list which it traverses independently of the other lists) and removal of plugins while our hook is calling into it as well.

[rustyrussell]

Think we might now be buggy if we remove a plugin while it's supposed to be answering a hook? Request used to be freed previously, what now?

...

If the plugin object is destroyed, we check if the hook invocation was currently calling into the plugin, and schedule the hook invocation to call into the next plugin`

Even with the current scheme, we could walk all plugin_hook_request (we'd need a list somewhere), and if we're about to remove the hook->plugins[req->current_plugin] then we do req->current_plugin--.

We still need to check that we don't crap ourselves if the plugin gets removed while a hook is pending, though. Let me try that...

[rustyrussell]

OK, testing answered my question. If a plugin is freed, the requests are abandoned.

This mean an HTLC is stuck, which is kinda nasty, but OK for now. I've filed a separate issue #3496 to track that.

So there's no issue for now with this PR, please cleanup and apply.

[cdecker]

Yes, I had noticed that as well, and was planning to address this in a separate PR (using current_plugin-- as @rustyrussell suggested). I probably should have mentioned that in the OP to avoid having reviewers need to check.

I'll address the feedback asap and update the PR 👍 Thanks for the review @rustyrussell and @ZmnSCPxj

[cdecker]

Added fixup!s to address the review, rebased and squashed.

You can see the changes between the version reviewed and the current version of the PR here

[cdecker]

Now having a failing assertion:

lightningd: lightningd/plugin_hook.c:135: plugin_hook_call_next: Assertion `ph_req->current_plugin < tal_count(hook->plugins)' failed.

Need to look into this.

Edit: Nevermind, turns out I killed the wrong one of the two overlapping checks. Now it works :-)

[rustyrussell]

Trivial rebase...