Double HBONE implementation for ambient multicluster by Stevenjin8 · Pull Request #1429 · istio/ztunnel

Stevenjin8 · 2025-01-17T16:51:37Z

Initial double HBONE implementation

Right now, inner HBONE will only hold one connect tunnel. Once the inner tunnel terminates, so will the outer tunnel (but not the outer HBONE). So when ztunnel receives its first connection to a double HBONE host (E/W gateway), it will perform two TLS handshakes. Subsequent connections to the same host will perform one TLS handshake.

This behavior is not great, but if we put the inner HBONE in the connection pool, then we pin ourselves to a pod in the remote cluster since ztunnel performs connection pooling, but is not aware of the E/W gateway's routing decision.

That being said, I think this is a good place to stop and think about control plane implementation and get some feedback on how I'm approaching this.

Tasks:

Implement double HBONE
Fix TLS code (I don't think I can do this without more control plane info since SANs are set for services)
metrics
Implement proper inner HBONE connection pooling
Tests

Some open questions:

Do I need to make any changes to metrics?
How do should we do inner HBONE pooling? My suggestion is to have up to N inner HBONE connections per E/W or per remote cluster.
Good ways to test for race conditions in connection terminations? Right now, it seems that connections end gracefully without race conditions, but that's just on my machine.

References:

istio-testing · 2025-01-17T16:51:41Z

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

Stevenjin8 · 2025-01-17T17:20:46Z

src/proxy/outbound.rs

                                debug!(component="outbound", dur=?start.elapsed(), "connection completed");
                            }).instrument(span);

-                            assertions::size_between_ref(1000, 1750, &serve_outbound_connection);


How did we get these numbers?

by looking a the current size and adding a small amount of buffer.

I think we covered this live on the WG call, but just to close the loop here - this must not grow (beyond a trivial amount) else it meas every connection will use that much additional memory. Typically the fix here is to Box::pin the futures.

Stevenjin8 · 2025-01-17T17:56:31Z

src/proxy/pool.rs


 // Does nothing but spawn new conns when asked
 impl ConnSpawner {
+    async fn new_unpooled_conn(


Anything here we can do higher up I think, but things might change if we decide to implement pooling in this PR

Yeah if we want double-hbone conns to be unpooled and thus need ~none of this surrounding machinery, then I'd be inclined to just start proxy/double-hbone.rs and use that directly, rather than complicating the purpose of this file.

(Could also just have a common HboneConnMgr trait or something too)

Stevenjin8 · 2025-01-17T19:52:12Z

src/proxy/outbound.rs

+        // This always drops ungracefully
+        // drop(conn_client);
+        // tokio::time::sleep(std::time::Duration::from_secs(1)).await;
+        // drain_tx.send(true).unwrap();
+        // tokio::time::sleep(std::time::Duration::from_secs(1)).await;
+        drain_tx.send(true).unwrap();
+        let _ = driver_task.await;
+        // this sleep is important, so we have a race condition somewhere
+        // tokio::time::sleep(std::time::Duration::from_secs(1)).await;
+        res


Does anybody have any info on how to properly drop/terminate H2 connections over stream with nontrivial drops (e.g. shutting down TLS over HTTP2 CONNECT). Right now, I'm just dropping things/aborting tasks randomly until something works

Are you asking about how to cleanup after, for example, a RST_STREAM to the inner tunnel? Or something else

Kinda. I mostly mean the outer TLS stream because that's what I've looked at. It seems like if I drop conn_client before termination driver_task the TCP connection will close without sending close notifies. So yes, I'm asking if there is a way to explicitly do cleanup rather than relying on implicit drops.

I see the code changed; do you still need help figuring this out?

Im still not confident in it. It works (on my machine), but I couldn't find any docs on proper connection termination/dropping.

Seems like we ignore shutdown errors so I'm not going to worry about this

Stevenjin8 · 2025-01-17T19:52:39Z

src/proxy/outbound.rs

    }

-    async fn send_hbone_request(
+    fn create_hbone_request(


Git merge is getting confused here

keithmattix · 2025-01-17T19:13:20Z

src/config.rs

 const UNSTABLE_ENABLE_SOCKS5: &str = "UNSTABLE_ENABLE_SOCKS5";

-const DEFAULT_WORKER_THREADS: u16 = 2;
+const DEFAULT_WORKER_THREADS: u16 = 40;


I may have missed in the description, but why the change here?

I was hoping it would making debugging async rust easier (it didn't)

(if you haven't already found it, tokio-console can be helpful)

keithmattix · 2025-01-17T20:17:00Z

src/proxy/outbound.rs

+        // This always drops ungracefully
+        // drop(conn_client);
+        // tokio::time::sleep(std::time::Duration::from_secs(1)).await;
+        // drain_tx.send(true).unwrap();
+        // tokio::time::sleep(std::time::Duration::from_secs(1)).await;
+        drain_tx.send(true).unwrap();
+        let _ = driver_task.await;
+        // this sleep is important, so we have a race condition somewhere
+        // tokio::time::sleep(std::time::Duration::from_secs(1)).await;
+        res


Are you asking about how to cleanup after, for example, a RST_STREAM to the inner tunnel? Or something else

Stevenjin8 · 2025-01-21T21:17:31Z

src/proxy/outbound.rs

-        // Inner HBONE
-        let upgraded = TokioH2Stream::new(upgraded);
+        // TODO: dst should take a hostname? and upstream_sans currently contains E/W Gateway certs
        let inner_workload = pool::WorkloadKey {


Will reorganize later.

Stevenjin8 · 2025-01-21T21:24:37Z

src/proxy/outbound.rs

+            Protocol::HBONE | Protocol::DOUBLEHBONE => Some(us.workload_socket_addr()),
            Protocol::TCP => None,
        };
+        let (upstream_sans, final_sans) = match us.workload.protocol {


My understanding from talking to @keithmattix is that Upstream.service_sans will be repurposed to contain the identities of remote pods/waypoints, so I should change the logic of the other protocols to only use us.workload.identity instead of us.workload_and_services_san.

Yes, I think this is correct; only the double hbone codepath needs to be added/changed because there are two sans being considered: the e/w gateway SAN and the SANs of the backends. So what you have looks right to me

Stevenjin8 · 2025-01-21T21:26:22Z

src/proxy/pool.rs

 // send requests over some underlying stream using some underlying http/2 client
-struct ConnClient {
-    sender: H2ConnectClient,
+pub struct ConnClient {


Stevenjin8 · 2025-04-01T15:17:58Z

I think metrics story is clear: only do metrics for inner hbone.

Also for RBAC, its only destination ztunnel that does RBAC, its not relevant here?

Stevenjin8 · 2025-04-02T14:58:54Z

/test test

Stevenjin8 · 2025-04-02T19:13:30Z

src/proxy/h2.rs

        std::io::Error::new(std::io::ErrorKind::Other, e)
    }
 }
+


TODO write tests

tests are in poo.rs

src/proxy/h2/client.rs

src/proxy/outbound.rs

src/state.rs

tests/namespaced.rs

howardjohn · 2025-04-04T21:07:22Z

src/tls/workload.rs

+    {
        let c = tokio_rustls::TlsConnector::from(self.client_config);
-        c.connect(dest, stream).await
+        c.connect(DUMMY_DOMAIN.clone(), stream).await


Can you please revert this. We should NOT be sending an SNI.

rustls makes this kind of annoying, but if you just set an IP it will not send anything. We can put 0.0.0.0 if you want.

Putting a dummy IP != putting a dummy domain

howardjohn · 2025-04-04T21:21:23Z

src/proxy/outbound.rs

+                    .as_ref()
+                    .expect("Workloads with network gateways must be service addressed.");
+
+                let (actual_destination, upstream_sans, final_sans) = match &ew_gtw.destination {


nit: can we just put this all in state.rs like fetch_waypoint but fetch_network_gateway?

howardjohn · 2025-04-04T21:36:12Z

src/proxy/outbound.rs

+        if let Some(ew_gtw) = &us.workload.network_gateway {
+            if us.workload.network != source_workload.network {


IIUC this means that if we find a workload in another network but it does NOT have a gateway, we will just send to the IP, which will not work (or worse, may go to some actual destination unrelated to the target). Do we have something protecting against this

I can add it, but I don't think we ever did this check

howardjohn · 2025-04-04T21:40:10Z

src/proxy/outbound.rs

+                        let us_gtw = self
+                            .pi
+                            .state
+                            .fetch_upstream_by_host(


If we follow the above comment about following fetch_watchpoint this would probably end up calling find_hostname --> find_upstream_from_service

howardjohn

Overall looks good, mostly some minor comments, only blocker is the dummy SNI

* Code layout * Handle mismatched networks with no network gateway * Get rid of a panic

howardjohn · 2025-04-07T19:46:21Z

src/state.rs

+    ) -> Result<(SocketAddr, Vec<Identity>, Vec<Identity>), Error> {
+        match &gtw.destination {
+            Destination::Address(address) => Ok((
+                SocketAddr::from((address.address, gtw.hbone_mtls_port)),


The intent of these APIs is the address are not raw IPs we send to, but lookup keys into workloads/services. We should use the same logic as fetch_waypoint which calls find_upstream

ok, so I can assume that the address has a Workload object behind it

zirain · 2025-04-08T03:06:10Z

the PR title is terrible.

keithmattix · 2025-04-08T03:21:04Z

That's my fault; I removed the hold before he could change it

istio-testing added do-not-merge/work-in-progress Block merging of a PR because it isn't ready yet. needs-rebase Indicates a PR needs to be rebased before being merged size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Jan 17, 2025

Stevenjin8 force-pushed the feature/double-hbone branch from e27a4a8 to 40bbcd0 Compare January 17, 2025 17:00

istio-testing removed the needs-rebase Indicates a PR needs to be rebased before being merged label Jan 17, 2025

Stevenjin8 changed the title ~~Feature/double hbone~~ WIP: Feature/double hbone Jan 17, 2025

Stevenjin8 added the do-not-merge/hold Block automatic merging of a PR. label Jan 17, 2025

Stevenjin8 force-pushed the feature/double-hbone branch from 66db8ae to 99d622f Compare January 17, 2025 17:51

Stevenjin8 commented Jan 17, 2025

View reviewed changes

Stevenjin8 force-pushed the feature/double-hbone branch from 99d622f to 96bb4de Compare January 17, 2025 17:55

Stevenjin8 commented Jan 17, 2025

View reviewed changes

Stevenjin8 requested review from bleggett and howardjohn January 17, 2025 17:59

Stevenjin8 commented Jan 17, 2025

View reviewed changes

keithmattix reviewed Jan 17, 2025

View reviewed changes

Stevenjin8 added 6 commits January 21, 2025 12:07

double hbone (no inner encryption) working!!

8702bd2

hbone-ish

e340d8c

cleanup

b381e34

graceful shutdowns

2192d67

Some cleanup

dff6c92

Add some auth/tls logic back in

f1cc535

Stevenjin8 force-pushed the feature/double-hbone branch from 1ea75fb to f1cc535 Compare January 21, 2025 17:07

inline double hbone code

cab4849

Stevenjin8 commented Jan 21, 2025

View reviewed changes

Use correct(?) identities

565f41f

Stevenjin8 force-pushed the feature/double-hbone branch from a8856a4 to 565f41f Compare January 21, 2025 21:19

Stevenjin8 commented Jan 21, 2025

View reviewed changes

Stevenjin8 commented Apr 2, 2025

View reviewed changes

Stevenjin8 added 2 commits April 2, 2025 15:20

Merge branch 'master' into feature/double-hbone

00aa327

Drop some uneeded changers

d7cb313

Stevenjin8 force-pushed the feature/double-hbone branch from 50cb231 to b4bd1c0 Compare April 2, 2025 19:50

get rid of more stuff

7cd4cf1

Stevenjin8 force-pushed the feature/double-hbone branch from b4bd1c0 to 7cd4cf1 Compare April 2, 2025 19:51

Stevenjin8 added 2 commits April 2, 2025 16:08

lint

fa46e46

lint

0638119

howardjohn reviewed Apr 4, 2025

View reviewed changes

Review:

04e7d30

* Code layout * Handle mismatched networks with no network gateway * Get rid of a panic

Stevenjin8 force-pushed the feature/double-hbone branch from f3fa871 to 04e7d30 Compare April 7, 2025 16:44

Stevenjin8 requested a review from howardjohn April 7, 2025 17:01

howardjohn reviewed Apr 7, 2025

View reviewed changes

Assume existing workload

0fbae22

Stevenjin8 requested a review from howardjohn April 7, 2025 21:49

howardjohn approved these changes Apr 7, 2025

View reviewed changes

keithmattix removed do-not-merge/work-in-progress Block merging of a PR because it isn't ready yet. do-not-merge/hold Block automatic merging of a PR. labels Apr 8, 2025

istio-testing merged commit 5b28370 into istio:master Apr 8, 2025
3 checks passed

keithmattix added do-not-merge/hold Block automatic merging of a PR. and removed do-not-merge/hold Block automatic merging of a PR. labels Apr 8, 2025

keithmattix changed the title ~~WIP: Feature/double hbone~~ Double HBONE implementation for ambient multicluster Apr 8, 2025

Stevenjin8 mentioned this pull request Jan 30, 2026

Add Stevenjin8 as ztunnel maintainer istio/community#1713

Merged

		if let Some(ew_gtw) = &us.workload.network_gateway {
		if us.workload.network != source_workload.network {

Conversation

Stevenjin8 commented Jan 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Initial double HBONE implementation

Uh oh!

istio-testing commented Jan 17, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bleggett Jan 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Stevenjin8 Jan 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Stevenjin8 commented Apr 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Stevenjin8 commented Apr 2, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

howardjohn left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Stevenjin8 commented Jan 17, 2025 •

edited

Loading

bleggett Jan 17, 2025 •

edited

Loading

Stevenjin8 Jan 21, 2025 •

edited

Loading

Stevenjin8 commented Apr 1, 2025 •

edited

Loading