TLS Routing behind ALB: `tsh kube join` by greedy52 · Pull Request #26168 · gravitational/teleport

greedy52 · 2023-05-12T20:16:27Z

Related issue:

TLS Routing ALB Support for all protocols #21870

Rework of #26047 to only include tsh kube join changes.

Updated dialer used for client.NewKubeSession
Moved some helpers from integration/proxy to integration/helpers.
Updated Kube Join integration test to connect kube session with different configurations

smallinsky · 2023-05-15T09:32:12Z

+
+	// Add options for ALPN connection upgrade only if kube is served at Proxy
+	// web address.
+	if tc.WebProxyAddr == kubeAddr && tc.TLSRoutingConnUpgradeRequired {


tc.WebProxyAddr == kubeAddr

I wonder about cases where this condition will be evaluated to false
AFIK kube sessions are using websocket conn so it must be a proxy addr.

The websocket is served by the kube server instead of the web server

teleport/lib/kube/proxy/forwarder.go

Line 332 in 19b193a

router.GET("/api/:ver/teleport/join/:session", fwd.withAuthPassthrough(fwd.join))

where TLS config uses the kube SNI prefix.

So the false case is when proxy in separate port mode and kube has its own public address.

public-teleport-github-review-bot · 2023-05-15T17:02:53Z

@greedy52 See the table below for backport results.

Branch	Result
branch/v13	Failed

public-teleport-github-review-bot · 2023-05-15T19:18:32Z

@greedy52 See the table below for backport results.

Branch	Result
branch/v13	Create PR

PR #26168 extended the inital Kube Join test to add more observers using different upgrade logic. While extending it, it also introduced the following dataraces ``` ================== WARNING: DATA RACE Read at 0x00c00139d548 by goroutine 64284: bytes.(*Buffer).String() /opt/go/src/bytes/buffer.go:65 +0x252 github.com/gravitational/teleport/integration.testKubeJoin.func7() /__w/teleport/teleport/integration/kube_integration_test.go:1702 +0x245 testing.tRunner() /opt/go/src/testing/testing.go:1576 +0x216 testing.(*T).Run.func1() /opt/go/src/testing/testing.go:1629 +0x47 Previous write at 0x00c00139d548 by goroutine 63051: bytes.(*Buffer).grow() /opt/go/src/bytes/buffer.go:145 +0x3c4 bytes.(*Buffer).Write() /opt/go/src/bytes/buffer.go:170 +0xcd github.com/gravitational/teleport/lib/utils.(*SyncWriter).Write() /__w/teleport/teleport/lib/utils/sync_writer.go:38 +0x109 io.copyBuffer() /opt/go/src/io/io.go:429 +0x2de io.Copy() /opt/go/src/io/io.go:386 +0xc5 github.com/gravitational/teleport/lib/client.(*KubeSession).pipeInOut.func1() /__w/teleport/teleport/lib/client/kubesession.go:208 +0x8d Goroutine 64284 (running) created at: testing.(*T).Run() /opt/go/src/testing/testing.go:1629 +0x805 github.com/gravitational/teleport/integration.testKubeJoin() /__w/teleport/teleport/integration/kube_integration_test.go:1691 +0x2b79 github.com/gravitational/teleport/integration.(*KubeSuite).bind.func1() /__w/teleport/teleport/integration/kube_integration_test.go:152 +0x118 testing.tRunner() /opt/go/src/testing/testing.go:1576 +0x216 testing.(*T).Run.func1() /opt/go/src/testing/testing.go:1629 +0x47 Goroutine 63051 (running) created at: github.com/gravitational/teleport/lib/client.(*KubeSession).pipeInOut() /__w/teleport/teleport/lib/client/kubesession.go:206 +0xea github.com/gravitational/teleport/lib/client.NewKubeSession() /__w/teleport/teleport/lib/client/kubesession.go:120 +0x12cb github.com/gravitational/teleport/integration.kubeJoin() /__w/teleport/teleport/integration/kube_integration_test.go:1518 +0x12a github.com/gravitational/teleport/integration.kubeJoinObserverWithSNISet() /__w/teleport/teleport/integration/kube_integration_test.go:1750 +0x34b github.com/gravitational/teleport/integration.kubeJoinByALBAddr() /__w/teleport/teleport/integration/kube_integration_test.go:1740 +0x249 github.com/gravitational/teleport/integration.testKubeJoin.func5() /__w/teleport/teleport/integration/kube_integration_test.go:1663 +0x14e testing.tRunner() /opt/go/src/testing/testing.go:1576 +0x216 testing.(*T).Run.func1() /opt/go/src/testing/testing.go:1629 +0x47 ================== ================== WARNING: DATA RACE Read at 0x00c00139d530 by goroutine 64284: bytes.(*Buffer).String() /opt/go/src/bytes/buffer.go:65 +0x268 github.com/gravitational/teleport/integration.testKubeJoin.func7() /__w/teleport/teleport/integration/kube_integration_test.go:1702 +0x245 testing.tRunner() /opt/go/src/testing/testing.go:1576 +0x216 testing.(*T).Run.func1() /opt/go/src/testing/testing.go:1629 +0x47 Previous write at 0x00c00139d530 by goroutine 63051: bytes.(*Buffer).grow() /opt/go/src/bytes/buffer.go:142 +0x2d6 bytes.(*Buffer).Write() /opt/go/src/bytes/buffer.go:170 +0xcd github.com/gravitational/teleport/lib/utils.(*SyncWriter).Write() /__w/teleport/teleport/lib/utils/sync_writer.go:38 +0x109 io.copyBuffer() /opt/go/src/io/io.go:429 +0x2de io.Copy() /opt/go/src/io/io.go:386 +0xc5 github.com/gravitational/teleport/lib/client.(*KubeSession).pipeInOut.func1() /__w/teleport/teleport/lib/client/kubesession.go:208 +0x8d Goroutine 64284 (running) created at: testing.(*T).Run() /opt/go/src/testing/testing.go:1629 +0x805 github.com/gravitational/teleport/integration.testKubeJoin() /__w/teleport/teleport/integration/kube_integration_test.go:1691 +0x2b79 github.com/gravitational/teleport/integration.(*KubeSuite).bind.func1() /__w/teleport/teleport/integration/kube_integration_test.go:152 +0x118 testing.tRunner() /opt/go/src/testing/testing.go:1576 +0x216 testing.(*T).Run.func1() /opt/go/src/testing/testing.go:1629 +0x47 Goroutine 63051 (running) created at: github.com/gravitational/teleport/lib/client.(*KubeSession).pipeInOut() /__w/teleport/teleport/lib/client/kubesession.go:206 +0xea github.com/gravitational/teleport/lib/client.NewKubeSession() /__w/teleport/teleport/lib/client/kubesession.go:120 +0x12cb github.com/gravitational/teleport/integration.kubeJoin() /__w/teleport/teleport/integration/kube_integration_test.go:1518 +0x12a github.com/gravitational/teleport/integration.kubeJoinObserverWithSNISet() /__w/teleport/teleport/integration/kube_integration_test.go:1750 +0x34b github.com/gravitational/teleport/integration.kubeJoinByALBAddr() /__w/teleport/teleport/integration/kube_integration_test.go:1740 +0x249 github.com/gravitational/teleport/integration.testKubeJoin.func5() /__w/teleport/teleport/integration/kube_integration_test.go:1663 +0x14e testing.tRunner() /opt/go/src/testing/testing.go:1576 +0x216 testing.(*T).Run.func1() /opt/go/src/testing/testing.go:1629 +0x47 ================== ================== WARNING: DATA RACE Read at 0x00c0023c2d00 by goroutine 64284: runtime.slicebytetostring() /opt/go/src/runtime/string.go:81 +0x0 bytes.(*Buffer).String() /opt/go/src/bytes/buffer.go:65 +0x2a6 github.com/gravitational/teleport/integration.testKubeJoin.func7() /__w/teleport/teleport/integration/kube_integration_test.go:1702 +0x245 testing.tRunner() /opt/go/src/testing/testing.go:1576 +0x216 testing.(*T).Run.func1() /opt/go/src/testing/testing.go:1629 +0x47 Previous write at 0x00c0023c2d00 by goroutine 63051: runtime.slicecopy() /opt/go/src/runtime/slice.go:310 +0x0 bytes.growSlice() /opt/go/src/bytes/buffer.go:241 +0x14d bytes.(*Buffer).grow() /opt/go/src/bytes/buffer.go:142 +0x2ba bytes.(*Buffer).Write() /opt/go/src/bytes/buffer.go:170 +0xcd github.com/gravitational/teleport/lib/utils.(*SyncWriter).Write() /__w/teleport/teleport/lib/utils/sync_writer.go:38 +0x109 io.copyBuffer() /opt/go/src/io/io.go:429 +0x2de io.Copy() /opt/go/src/io/io.go:386 +0xc5 github.com/gravitational/teleport/lib/client.(*KubeSession).pipeInOut.func1() /__w/teleport/teleport/lib/client/kubesession.go:208 +0x8d Goroutine 64284 (running) created at: testing.(*T).Run() /opt/go/src/testing/testing.go:1629 +0x805 github.com/gravitational/teleport/integration.testKubeJoin() /__w/teleport/teleport/integration/kube_integration_test.go:1691 +0x2b79 github.com/gravitational/teleport/integration.(*KubeSuite).bind.func1() /__w/teleport/teleport/integration/kube_integration_test.go:152 +0x118 testing.tRunner() /opt/go/src/testing/testing.go:1576 +0x216 testing.(*T).Run.func1() /opt/go/src/testing/testing.go:1629 +0x47 Goroutine 63051 (running) created at: github.com/gravitational/teleport/lib/client.(*KubeSession).pipeInOut() /__w/teleport/teleport/lib/client/kubesession.go:206 +0xea github.com/gravitational/teleport/lib/client.NewKubeSession() /__w/teleport/teleport/lib/client/kubesession.go:120 +0x12cb github.com/gravitational/teleport/integration.kubeJoin() /__w/teleport/teleport/integration/kube_integration_test.go:1518 +0x12a github.com/gravitational/teleport/integration.kubeJoinObserverWithSNISet() /__w/teleport/teleport/integration/kube_integration_test.go:1750 +0x34b github.com/gravitational/teleport/integration.kubeJoinByALBAddr() /__w/teleport/teleport/integration/kube_integration_test.go:1740 +0x249 github.com/gravitational/teleport/integration.testKubeJoin.func5() /__w/teleport/teleport/integration/kube_integration_test.go:1663 +0x14e testing.tRunner() /opt/go/src/testing/testing.go:1576 +0x216 testing.(*T).Run.func1() /opt/go/src/testing/testing.go:1629 +0x47 ================== testing.go:1446: race detected during execution of test ```

* Fix `TestKube/Join` data race PR #26168 extended the inital Kube Join test to add more observers using different upgrade logic. While extending it, it also introduced the following dataraces ``` ================== WARNING: DATA RACE Read at 0x00c00139d548 by goroutine 64284: bytes.(*Buffer).String() /opt/go/src/bytes/buffer.go:65 +0x252 github.com/gravitational/teleport/integration.testKubeJoin.func7() /__w/teleport/teleport/integration/kube_integration_test.go:1702 +0x245 testing.tRunner() /opt/go/src/testing/testing.go:1576 +0x216 testing.(*T).Run.func1() /opt/go/src/testing/testing.go:1629 +0x47 Previous write at 0x00c00139d548 by goroutine 63051: bytes.(*Buffer).grow() /opt/go/src/bytes/buffer.go:145 +0x3c4 bytes.(*Buffer).Write() /opt/go/src/bytes/buffer.go:170 +0xcd github.com/gravitational/teleport/lib/utils.(*SyncWriter).Write() /__w/teleport/teleport/lib/utils/sync_writer.go:38 +0x109 io.copyBuffer() /opt/go/src/io/io.go:429 +0x2de io.Copy() /opt/go/src/io/io.go:386 +0xc5 github.com/gravitational/teleport/lib/client.(*KubeSession).pipeInOut.func1() /__w/teleport/teleport/lib/client/kubesession.go:208 +0x8d Goroutine 64284 (running) created at: testing.(*T).Run() /opt/go/src/testing/testing.go:1629 +0x805 github.com/gravitational/teleport/integration.testKubeJoin() /__w/teleport/teleport/integration/kube_integration_test.go:1691 +0x2b79 github.com/gravitational/teleport/integration.(*KubeSuite).bind.func1() /__w/teleport/teleport/integration/kube_integration_test.go:152 +0x118 testing.tRunner() /opt/go/src/testing/testing.go:1576 +0x216 testing.(*T).Run.func1() /opt/go/src/testing/testing.go:1629 +0x47 Goroutine 63051 (running) created at: github.com/gravitational/teleport/lib/client.(*KubeSession).pipeInOut() /__w/teleport/teleport/lib/client/kubesession.go:206 +0xea github.com/gravitational/teleport/lib/client.NewKubeSession() /__w/teleport/teleport/lib/client/kubesession.go:120 +0x12cb github.com/gravitational/teleport/integration.kubeJoin() /__w/teleport/teleport/integration/kube_integration_test.go:1518 +0x12a github.com/gravitational/teleport/integration.kubeJoinObserverWithSNISet() /__w/teleport/teleport/integration/kube_integration_test.go:1750 +0x34b github.com/gravitational/teleport/integration.kubeJoinByALBAddr() /__w/teleport/teleport/integration/kube_integration_test.go:1740 +0x249 github.com/gravitational/teleport/integration.testKubeJoin.func5() /__w/teleport/teleport/integration/kube_integration_test.go:1663 +0x14e testing.tRunner() /opt/go/src/testing/testing.go:1576 +0x216 testing.(*T).Run.func1() /opt/go/src/testing/testing.go:1629 +0x47 ================== ================== WARNING: DATA RACE Read at 0x00c00139d530 by goroutine 64284: bytes.(*Buffer).String() /opt/go/src/bytes/buffer.go:65 +0x268 github.com/gravitational/teleport/integration.testKubeJoin.func7() /__w/teleport/teleport/integration/kube_integration_test.go:1702 +0x245 testing.tRunner() /opt/go/src/testing/testing.go:1576 +0x216 testing.(*T).Run.func1() /opt/go/src/testing/testing.go:1629 +0x47 Previous write at 0x00c00139d530 by goroutine 63051: bytes.(*Buffer).grow() /opt/go/src/bytes/buffer.go:142 +0x2d6 bytes.(*Buffer).Write() /opt/go/src/bytes/buffer.go:170 +0xcd github.com/gravitational/teleport/lib/utils.(*SyncWriter).Write() /__w/teleport/teleport/lib/utils/sync_writer.go:38 +0x109 io.copyBuffer() /opt/go/src/io/io.go:429 +0x2de io.Copy() /opt/go/src/io/io.go:386 +0xc5 github.com/gravitational/teleport/lib/client.(*KubeSession).pipeInOut.func1() /__w/teleport/teleport/lib/client/kubesession.go:208 +0x8d Goroutine 64284 (running) created at: testing.(*T).Run() /opt/go/src/testing/testing.go:1629 +0x805 github.com/gravitational/teleport/integration.testKubeJoin() /__w/teleport/teleport/integration/kube_integration_test.go:1691 +0x2b79 github.com/gravitational/teleport/integration.(*KubeSuite).bind.func1() /__w/teleport/teleport/integration/kube_integration_test.go:152 +0x118 testing.tRunner() /opt/go/src/testing/testing.go:1576 +0x216 testing.(*T).Run.func1() /opt/go/src/testing/testing.go:1629 +0x47 Goroutine 63051 (running) created at: github.com/gravitational/teleport/lib/client.(*KubeSession).pipeInOut() /__w/teleport/teleport/lib/client/kubesession.go:206 +0xea github.com/gravitational/teleport/lib/client.NewKubeSession() /__w/teleport/teleport/lib/client/kubesession.go:120 +0x12cb github.com/gravitational/teleport/integration.kubeJoin() /__w/teleport/teleport/integration/kube_integration_test.go:1518 +0x12a github.com/gravitational/teleport/integration.kubeJoinObserverWithSNISet() /__w/teleport/teleport/integration/kube_integration_test.go:1750 +0x34b github.com/gravitational/teleport/integration.kubeJoinByALBAddr() /__w/teleport/teleport/integration/kube_integration_test.go:1740 +0x249 github.com/gravitational/teleport/integration.testKubeJoin.func5() /__w/teleport/teleport/integration/kube_integration_test.go:1663 +0x14e testing.tRunner() /opt/go/src/testing/testing.go:1576 +0x216 testing.(*T).Run.func1() /opt/go/src/testing/testing.go:1629 +0x47 ================== ================== WARNING: DATA RACE Read at 0x00c0023c2d00 by goroutine 64284: runtime.slicebytetostring() /opt/go/src/runtime/string.go:81 +0x0 bytes.(*Buffer).String() /opt/go/src/bytes/buffer.go:65 +0x2a6 github.com/gravitational/teleport/integration.testKubeJoin.func7() /__w/teleport/teleport/integration/kube_integration_test.go:1702 +0x245 testing.tRunner() /opt/go/src/testing/testing.go:1576 +0x216 testing.(*T).Run.func1() /opt/go/src/testing/testing.go:1629 +0x47 Previous write at 0x00c0023c2d00 by goroutine 63051: runtime.slicecopy() /opt/go/src/runtime/slice.go:310 +0x0 bytes.growSlice() /opt/go/src/bytes/buffer.go:241 +0x14d bytes.(*Buffer).grow() /opt/go/src/bytes/buffer.go:142 +0x2ba bytes.(*Buffer).Write() /opt/go/src/bytes/buffer.go:170 +0xcd github.com/gravitational/teleport/lib/utils.(*SyncWriter).Write() /__w/teleport/teleport/lib/utils/sync_writer.go:38 +0x109 io.copyBuffer() /opt/go/src/io/io.go:429 +0x2de io.Copy() /opt/go/src/io/io.go:386 +0xc5 github.com/gravitational/teleport/lib/client.(*KubeSession).pipeInOut.func1() /__w/teleport/teleport/lib/client/kubesession.go:208 +0x8d Goroutine 64284 (running) created at: testing.(*T).Run() /opt/go/src/testing/testing.go:1629 +0x805 github.com/gravitational/teleport/integration.testKubeJoin() /__w/teleport/teleport/integration/kube_integration_test.go:1691 +0x2b79 github.com/gravitational/teleport/integration.(*KubeSuite).bind.func1() /__w/teleport/teleport/integration/kube_integration_test.go:152 +0x118 testing.tRunner() /opt/go/src/testing/testing.go:1576 +0x216 testing.(*T).Run.func1() /opt/go/src/testing/testing.go:1629 +0x47 Goroutine 63051 (running) created at: github.com/gravitational/teleport/lib/client.(*KubeSession).pipeInOut() /__w/teleport/teleport/lib/client/kubesession.go:206 +0xea github.com/gravitational/teleport/lib/client.NewKubeSession() /__w/teleport/teleport/lib/client/kubesession.go:120 +0x12cb github.com/gravitational/teleport/integration.kubeJoin() /__w/teleport/teleport/integration/kube_integration_test.go:1518 +0x12a github.com/gravitational/teleport/integration.kubeJoinObserverWithSNISet() /__w/teleport/teleport/integration/kube_integration_test.go:1750 +0x34b github.com/gravitational/teleport/integration.kubeJoinByALBAddr() /__w/teleport/teleport/integration/kube_integration_test.go:1740 +0x249 github.com/gravitational/teleport/integration.testKubeJoin.func5() /__w/teleport/teleport/integration/kube_integration_test.go:1663 +0x14e testing.tRunner() /opt/go/src/testing/testing.go:1576 +0x216 testing.(*T).Run.func1() /opt/go/src/testing/testing.go:1629 +0x47 ================== testing.go:1446: race detected during execution of test ``` * fix multi write * improve blocked case

TLS Routing behind ALB: tsh kube join

f5ee060

greedy52 added kubernetes-access tls-routing Issues related to TLS routing labels May 12, 2023

greedy52 requested review from AntonAM, smallinsky and tigrato May 12, 2023 20:16

greedy52 self-assigned this May 12, 2023

github-actions Bot added the size/md label May 12, 2023

github-actions Bot requested review from marcoandredinis and rosstimothy May 12, 2023 20:16

marcoandredinis removed their request for review May 15, 2023 08:08

tigrato approved these changes May 15, 2023

View reviewed changes

smallinsky approved these changes May 15, 2023

View reviewed changes

public-teleport-github-review-bot Bot removed request for AntonAM and rosstimothy May 15, 2023 09:35

fix lint

1d59ebd

greedy52 enabled auto-merge May 15, 2023 13:57

greedy52 added this pull request to the merge queue May 15, 2023

Merged via the queue into master with commit 89d17b5 May 15, 2023

greedy52 deleted the STeve/21870_kube_join branch May 15, 2023 16:32

greedy52 added the backport/branch/v13 label May 15, 2023

greedy52 mentioned this pull request May 15, 2023

[v13] TLS Routing behind ALB: tsh kube join #26283

Merged

tigrato mentioned this pull request May 19, 2023

Fix TestKube/Join data race #26619

Merged

GavinFrazar mentioned this pull request May 20, 2023

TestKube data race #26652

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

TLS Routing behind ALB: `tsh kube join`#26168

TLS Routing behind ALB: `tsh kube join`#26168
greedy52 merged 2 commits intomasterfrom
STeve/21870_kube_join

greedy52 commented May 12, 2023

Uh oh!

Uh oh!

smallinsky May 15, 2023

Uh oh!

greedy52 May 15, 2023

Uh oh!

public-teleport-github-review-bot Bot commented May 15, 2023

Uh oh!

public-teleport-github-review-bot Bot commented May 15, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

greedy52 commented May 12, 2023

Uh oh!

Uh oh!

smallinsky May 15, 2023

Choose a reason for hiding this comment

Uh oh!

greedy52 May 15, 2023

Choose a reason for hiding this comment

Uh oh!

public-teleport-github-review-bot Bot commented May 15, 2023

Uh oh!

public-teleport-github-review-bot Bot commented May 15, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants