Multinetwork/Support remote networks for services with waypoints

[krinkinmu]

Currently build_request when it sees a service with a waypoint resolves the waypoint backend and routes request there using regular HBONE.

In multi network scenario though the waypoint may have workload on a remote network and to reach it we have to go through E/W gateway and use double HBONE.

This change enables handling of services with waypoint on a remote network.

Some of the assumptions that were used when I prepared this change:

1. We assume uniformity of configuration (e.g., if service X in local cluster has a waypoint, then service X in remote network also has a waypoint, if waypoint is service addressable, then it's using service to address waypoint both locally and on remote network)
2. Split-horizon representation of waypoint workloads, just like with
   any regular workloads and services (e.g., in the local cluster
   instead of an actual waypoint workload pointing to a pod on another
   network we will have a "proxy" representation that just has network
   gateway).

Both of those can be in hanled by the controlplane (e.g., controlplane can generate split-horizon workloads and when configuration is non-uniform, just filter out remote configs for remote networks), though we don't yet have a complete implementation.

Tracking issue #1555

[istio-policy-bot]

😊 Welcome @krinkinmu! This is either your first contribution to the Istio ztunnel repo, or it's been
a while since you've been here.

You can learn more about the Istio working groups, Code of Conduct, and contribution guidelines
by referring to Contributing to Istio.

Thanks for contributing!

Courtesy of your friendly welcome wagon.

[istio-testing]

Hi @krinkinmu. Thanks for your PR.

I'm waiting for a istio member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[keithmattix]

/ok-to-test

[keithmattix]

We should create a follow-up issue to address this scenario if the config isn't uniform (for beta). Such a scenario will probably require changes to WDS

[howardjohn]

So is the net effect in traffic just the scenario: local service has "use waypoint" but no local waypoint pods and remote has waypoint pods now sends remote?

Since it seems like most scenarios will behave the same, seeing as waypoints always have "prefer close" so they will never pick a remote waypoint if there is a local one?

[krinkinmu]

So is the net effect in traffic just the scenario: local service has "use waypoint" but no local waypoint pods and remote has waypoint pods now sends remote?

Since it seems like most scenarios will behave the same, seeing as waypoints always have "prefer close" so they will never pick a remote waypoint if there is a local one?

Pretty much, if there is a healthy local waypoint it will be picked unless the user overrides it.

We discussed it internally with @keithmattix and @Stevenjin8 and basically for the initial version the scenario we want to support is failover and not load-balancing between clusters. And current behavior kind of makes sense from this point of view (aside from not using double HBONE).

The reason we didn't go for more is because with L7 policies we just can't know for sure where the requests are going to get routed and because of that it's not easy to consider capacity accurately. We might come up with some heuristic to overcome this, but it's not clear how it would look like and if we want to rely on a heuristic for load balancing between networks.

A side note, unrelated to the above, there is another change in behavior that this PR introduces - it changes hbone destination for double hbone to use the original target port instead of translating it. The desirable behavior for E/W gateway is similar to what waypoint does - we give it the original target we tried to connect, and it decides where to route the request exactly.

[keithmattix]

@howardjohn @ilrudie Can we get some eyes on this? It's pretty small and has been sitting for a month

[ilrudie]

mostly LGTM, I think we should toss in a handful of debug logs just to help clarify what happened when we are troubleshooting.

[ilrudie]

ty ty

[keithmattix]

/cherry-pick release-1.27

[istio-testing]

@keithmattix: new pull request created: #1592

Details

In response to this:

/cherry-pick release-1.27

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.