Fix `TestKube/Join` data race by tigrato · Pull Request #26619 · gravitational/teleport

tigrato · 2023-05-19T16:41:45Z

PR #26168 extended the inital Kube Join test to add more observers using different upgrade logic.
While extending it, it also introduced the following dataraces

==================
WARNING: DATA RACE
Read at 0x00c00139d548 by goroutine 64284:
  bytes.(*Buffer).String()
      /opt/go/src/bytes/buffer.go:65 +0x252
  github.com/gravitational/teleport/integration.testKubeJoin.func7()
      /__w/teleport/teleport/integration/kube_integration_test.go:1702 +0x245
  testing.tRunner()
      /opt/go/src/testing/testing.go:1576 +0x216
  testing.(*T).Run.func1()
      /opt/go/src/testing/testing.go:1629 +0x47

Previous write at 0x00c00139d548 by goroutine 63051:
  bytes.(*Buffer).grow()
      /opt/go/src/bytes/buffer.go:145 +0x3c4
  bytes.(*Buffer).Write()
      /opt/go/src/bytes/buffer.go:170 +0xcd
  github.com/gravitational/teleport/lib/utils.(*SyncWriter).Write()
      /__w/teleport/teleport/lib/utils/sync_writer.go:38 +0x109
  io.copyBuffer()
      /opt/go/src/io/io.go:429 +0x2de
  io.Copy()
      /opt/go/src/io/io.go:386 +0xc5
  github.com/gravitational/teleport/lib/client.(*KubeSession).pipeInOut.func1()
      /__w/teleport/teleport/lib/client/kubesession.go:208 +0x8d

Goroutine 64284 (running) created at:
  testing.(*T).Run()
      /opt/go/src/testing/testing.go:1629 +0x805
  github.com/gravitational/teleport/integration.testKubeJoin()
      /__w/teleport/teleport/integration/kube_integration_test.go:1691 +0x2b79
  github.com/gravitational/teleport/integration.(*KubeSuite).bind.func1()
      /__w/teleport/teleport/integration/kube_integration_test.go:152 +0x118
  testing.tRunner()
      /opt/go/src/testing/testing.go:1576 +0x216
  testing.(*T).Run.func1()
      /opt/go/src/testing/testing.go:1629 +0x47

Goroutine 63051 (running) created at:
  github.com/gravitational/teleport/lib/client.(*KubeSession).pipeInOut()
      /__w/teleport/teleport/lib/client/kubesession.go:206 +0xea
  github.com/gravitational/teleport/lib/client.NewKubeSession()
      /__w/teleport/teleport/lib/client/kubesession.go:120 +0x12cb
  github.com/gravitational/teleport/integration.kubeJoin()
      /__w/teleport/teleport/integration/kube_integration_test.go:1518 +0x12a
  github.com/gravitational/teleport/integration.kubeJoinObserverWithSNISet()
      /__w/teleport/teleport/integration/kube_integration_test.go:1750 +0x34b
  github.com/gravitational/teleport/integration.kubeJoinByALBAddr()
      /__w/teleport/teleport/integration/kube_integration_test.go:1740 +0x249
  github.com/gravitational/teleport/integration.testKubeJoin.func5()
      /__w/teleport/teleport/integration/kube_integration_test.go:1663 +0x14e
  testing.tRunner()
      /opt/go/src/testing/testing.go:1576 +0x216
  testing.(*T).Run.func1()
      /opt/go/src/testing/testing.go:1629 +0x47
==================
==================
WARNING: DATA RACE
Read at 0x00c00139d530 by goroutine 64284:
  bytes.(*Buffer).String()
      /opt/go/src/bytes/buffer.go:65 +0x268
  github.com/gravitational/teleport/integration.testKubeJoin.func7()
      /__w/teleport/teleport/integration/kube_integration_test.go:1702 +0x245
  testing.tRunner()
      /opt/go/src/testing/testing.go:1576 +0x216
  testing.(*T).Run.func1()
      /opt/go/src/testing/testing.go:1629 +0x47

Previous write at 0x00c00139d530 by goroutine 63051:
  bytes.(*Buffer).grow()
      /opt/go/src/bytes/buffer.go:142 +0x2d6
  bytes.(*Buffer).Write()
      /opt/go/src/bytes/buffer.go:170 +0xcd
  github.com/gravitational/teleport/lib/utils.(*SyncWriter).Write()
      /__w/teleport/teleport/lib/utils/sync_writer.go:38 +0x109
  io.copyBuffer()
      /opt/go/src/io/io.go:429 +0x2de
  io.Copy()
      /opt/go/src/io/io.go:386 +0xc5
  github.com/gravitational/teleport/lib/client.(*KubeSession).pipeInOut.func1()
      /__w/teleport/teleport/lib/client/kubesession.go:208 +0x8d

Goroutine 64284 (running) created at:
  testing.(*T).Run()
      /opt/go/src/testing/testing.go:1629 +0x805
  github.com/gravitational/teleport/integration.testKubeJoin()
      /__w/teleport/teleport/integration/kube_integration_test.go:1691 +0x2b79
  github.com/gravitational/teleport/integration.(*KubeSuite).bind.func1()
      /__w/teleport/teleport/integration/kube_integration_test.go:152 +0x118
  testing.tRunner()
      /opt/go/src/testing/testing.go:1576 +0x216
  testing.(*T).Run.func1()
      /opt/go/src/testing/testing.go:1629 +0x47

Goroutine 63051 (running) created at:
  github.com/gravitational/teleport/lib/client.(*KubeSession).pipeInOut()
      /__w/teleport/teleport/lib/client/kubesession.go:206 +0xea
  github.com/gravitational/teleport/lib/client.NewKubeSession()
      /__w/teleport/teleport/lib/client/kubesession.go:120 +0x12cb
  github.com/gravitational/teleport/integration.kubeJoin()
      /__w/teleport/teleport/integration/kube_integration_test.go:1518 +0x12a
  github.com/gravitational/teleport/integration.kubeJoinObserverWithSNISet()
      /__w/teleport/teleport/integration/kube_integration_test.go:1750 +0x34b
  github.com/gravitational/teleport/integration.kubeJoinByALBAddr()
      /__w/teleport/teleport/integration/kube_integration_test.go:1740 +0x249
  github.com/gravitational/teleport/integration.testKubeJoin.func5()
      /__w/teleport/teleport/integration/kube_integration_test.go:1663 +0x14e
  testing.tRunner()
      /opt/go/src/testing/testing.go:1576 +0x216
  testing.(*T).Run.func1()
      /opt/go/src/testing/testing.go:1629 +0x47
==================
==================
WARNING: DATA RACE
Read at 0x00c0023c2d00 by goroutine 64284:
  runtime.slicebytetostring()
      /opt/go/src/runtime/string.go:81 +0x0
  bytes.(*Buffer).String()
      /opt/go/src/bytes/buffer.go:65 +0x2a6
  github.com/gravitational/teleport/integration.testKubeJoin.func7()
      /__w/teleport/teleport/integration/kube_integration_test.go:1702 +0x245
  testing.tRunner()
      /opt/go/src/testing/testing.go:1576 +0x216
  testing.(*T).Run.func1()
      /opt/go/src/testing/testing.go:1629 +0x47

Previous write at 0x00c0023c2d00 by goroutine 63051:
  runtime.slicecopy()
      /opt/go/src/runtime/slice.go:310 +0x0
  bytes.growSlice()
      /opt/go/src/bytes/buffer.go:241 +0x14d
  bytes.(*Buffer).grow()
      /opt/go/src/bytes/buffer.go:142 +0x2ba
  bytes.(*Buffer).Write()
      /opt/go/src/bytes/buffer.go:170 +0xcd
  github.com/gravitational/teleport/lib/utils.(*SyncWriter).Write()
      /__w/teleport/teleport/lib/utils/sync_writer.go:38 +0x109
  io.copyBuffer()
      /opt/go/src/io/io.go:429 +0x2de
  io.Copy()
      /opt/go/src/io/io.go:386 +0xc5
  github.com/gravitational/teleport/lib/client.(*KubeSession).pipeInOut.func1()
      /__w/teleport/teleport/lib/client/kubesession.go:208 +0x8d

Goroutine 64284 (running) created at:
  testing.(*T).Run()
      /opt/go/src/testing/testing.go:1629 +0x805
  github.com/gravitational/teleport/integration.testKubeJoin()
      /__w/teleport/teleport/integration/kube_integration_test.go:1691 +0x2b79
  github.com/gravitational/teleport/integration.(*KubeSuite).bind.func1()
      /__w/teleport/teleport/integration/kube_integration_test.go:152 +0x118
  testing.tRunner()
      /opt/go/src/testing/testing.go:1576 +0x216
  testing.(*T).Run.func1()
      /opt/go/src/testing/testing.go:1629 +0x47

Goroutine 63051 (running) created at:
  github.com/gravitational/teleport/lib/client.(*KubeSession).pipeInOut()
      /__w/teleport/teleport/lib/client/kubesession.go:206 +0xea
  github.com/gravitational/teleport/lib/client.NewKubeSession()
      /__w/teleport/teleport/lib/client/kubesession.go:120 +0x12cb
  github.com/gravitational/teleport/integration.kubeJoin()
      /__w/teleport/teleport/integration/kube_integration_test.go:1518 +0x12a
  github.com/gravitational/teleport/integration.kubeJoinObserverWithSNISet()
      /__w/teleport/teleport/integration/kube_integration_test.go:1750 +0x34b
  github.com/gravitational/teleport/integration.kubeJoinByALBAddr()
      /__w/teleport/teleport/integration/kube_integration_test.go:1740 +0x249
  github.com/gravitational/teleport/integration.testKubeJoin.func5()
      /__w/teleport/teleport/integration/kube_integration_test.go:1663 +0x14e
  testing.tRunner()
      /opt/go/src/testing/testing.go:1576 +0x216
  testing.(*T).Run.func1()
      /opt/go/src/testing/testing.go:1629 +0x47
==================
    testing.go:1446: race detected during execution of test

Fixes #26652

PR #26168 extended the inital Kube Join test to add more observers using different upgrade logic. While extending it, it also introduced the following dataraces ``` ================== WARNING: DATA RACE Read at 0x00c00139d548 by goroutine 64284: bytes.(*Buffer).String() /opt/go/src/bytes/buffer.go:65 +0x252 github.com/gravitational/teleport/integration.testKubeJoin.func7() /__w/teleport/teleport/integration/kube_integration_test.go:1702 +0x245 testing.tRunner() /opt/go/src/testing/testing.go:1576 +0x216 testing.(*T).Run.func1() /opt/go/src/testing/testing.go:1629 +0x47 Previous write at 0x00c00139d548 by goroutine 63051: bytes.(*Buffer).grow() /opt/go/src/bytes/buffer.go:145 +0x3c4 bytes.(*Buffer).Write() /opt/go/src/bytes/buffer.go:170 +0xcd github.com/gravitational/teleport/lib/utils.(*SyncWriter).Write() /__w/teleport/teleport/lib/utils/sync_writer.go:38 +0x109 io.copyBuffer() /opt/go/src/io/io.go:429 +0x2de io.Copy() /opt/go/src/io/io.go:386 +0xc5 github.com/gravitational/teleport/lib/client.(*KubeSession).pipeInOut.func1() /__w/teleport/teleport/lib/client/kubesession.go:208 +0x8d Goroutine 64284 (running) created at: testing.(*T).Run() /opt/go/src/testing/testing.go:1629 +0x805 github.com/gravitational/teleport/integration.testKubeJoin() /__w/teleport/teleport/integration/kube_integration_test.go:1691 +0x2b79 github.com/gravitational/teleport/integration.(*KubeSuite).bind.func1() /__w/teleport/teleport/integration/kube_integration_test.go:152 +0x118 testing.tRunner() /opt/go/src/testing/testing.go:1576 +0x216 testing.(*T).Run.func1() /opt/go/src/testing/testing.go:1629 +0x47 Goroutine 63051 (running) created at: github.com/gravitational/teleport/lib/client.(*KubeSession).pipeInOut() /__w/teleport/teleport/lib/client/kubesession.go:206 +0xea github.com/gravitational/teleport/lib/client.NewKubeSession() /__w/teleport/teleport/lib/client/kubesession.go:120 +0x12cb github.com/gravitational/teleport/integration.kubeJoin() /__w/teleport/teleport/integration/kube_integration_test.go:1518 +0x12a github.com/gravitational/teleport/integration.kubeJoinObserverWithSNISet() /__w/teleport/teleport/integration/kube_integration_test.go:1750 +0x34b github.com/gravitational/teleport/integration.kubeJoinByALBAddr() /__w/teleport/teleport/integration/kube_integration_test.go:1740 +0x249 github.com/gravitational/teleport/integration.testKubeJoin.func5() /__w/teleport/teleport/integration/kube_integration_test.go:1663 +0x14e testing.tRunner() /opt/go/src/testing/testing.go:1576 +0x216 testing.(*T).Run.func1() /opt/go/src/testing/testing.go:1629 +0x47 ================== ================== WARNING: DATA RACE Read at 0x00c00139d530 by goroutine 64284: bytes.(*Buffer).String() /opt/go/src/bytes/buffer.go:65 +0x268 github.com/gravitational/teleport/integration.testKubeJoin.func7() /__w/teleport/teleport/integration/kube_integration_test.go:1702 +0x245 testing.tRunner() /opt/go/src/testing/testing.go:1576 +0x216 testing.(*T).Run.func1() /opt/go/src/testing/testing.go:1629 +0x47 Previous write at 0x00c00139d530 by goroutine 63051: bytes.(*Buffer).grow() /opt/go/src/bytes/buffer.go:142 +0x2d6 bytes.(*Buffer).Write() /opt/go/src/bytes/buffer.go:170 +0xcd github.com/gravitational/teleport/lib/utils.(*SyncWriter).Write() /__w/teleport/teleport/lib/utils/sync_writer.go:38 +0x109 io.copyBuffer() /opt/go/src/io/io.go:429 +0x2de io.Copy() /opt/go/src/io/io.go:386 +0xc5 github.com/gravitational/teleport/lib/client.(*KubeSession).pipeInOut.func1() /__w/teleport/teleport/lib/client/kubesession.go:208 +0x8d Goroutine 64284 (running) created at: testing.(*T).Run() /opt/go/src/testing/testing.go:1629 +0x805 github.com/gravitational/teleport/integration.testKubeJoin() /__w/teleport/teleport/integration/kube_integration_test.go:1691 +0x2b79 github.com/gravitational/teleport/integration.(*KubeSuite).bind.func1() /__w/teleport/teleport/integration/kube_integration_test.go:152 +0x118 testing.tRunner() /opt/go/src/testing/testing.go:1576 +0x216 testing.(*T).Run.func1() /opt/go/src/testing/testing.go:1629 +0x47 Goroutine 63051 (running) created at: github.com/gravitational/teleport/lib/client.(*KubeSession).pipeInOut() /__w/teleport/teleport/lib/client/kubesession.go:206 +0xea github.com/gravitational/teleport/lib/client.NewKubeSession() /__w/teleport/teleport/lib/client/kubesession.go:120 +0x12cb github.com/gravitational/teleport/integration.kubeJoin() /__w/teleport/teleport/integration/kube_integration_test.go:1518 +0x12a github.com/gravitational/teleport/integration.kubeJoinObserverWithSNISet() /__w/teleport/teleport/integration/kube_integration_test.go:1750 +0x34b github.com/gravitational/teleport/integration.kubeJoinByALBAddr() /__w/teleport/teleport/integration/kube_integration_test.go:1740 +0x249 github.com/gravitational/teleport/integration.testKubeJoin.func5() /__w/teleport/teleport/integration/kube_integration_test.go:1663 +0x14e testing.tRunner() /opt/go/src/testing/testing.go:1576 +0x216 testing.(*T).Run.func1() /opt/go/src/testing/testing.go:1629 +0x47 ================== ================== WARNING: DATA RACE Read at 0x00c0023c2d00 by goroutine 64284: runtime.slicebytetostring() /opt/go/src/runtime/string.go:81 +0x0 bytes.(*Buffer).String() /opt/go/src/bytes/buffer.go:65 +0x2a6 github.com/gravitational/teleport/integration.testKubeJoin.func7() /__w/teleport/teleport/integration/kube_integration_test.go:1702 +0x245 testing.tRunner() /opt/go/src/testing/testing.go:1576 +0x216 testing.(*T).Run.func1() /opt/go/src/testing/testing.go:1629 +0x47 Previous write at 0x00c0023c2d00 by goroutine 63051: runtime.slicecopy() /opt/go/src/runtime/slice.go:310 +0x0 bytes.growSlice() /opt/go/src/bytes/buffer.go:241 +0x14d bytes.(*Buffer).grow() /opt/go/src/bytes/buffer.go:142 +0x2ba bytes.(*Buffer).Write() /opt/go/src/bytes/buffer.go:170 +0xcd github.com/gravitational/teleport/lib/utils.(*SyncWriter).Write() /__w/teleport/teleport/lib/utils/sync_writer.go:38 +0x109 io.copyBuffer() /opt/go/src/io/io.go:429 +0x2de io.Copy() /opt/go/src/io/io.go:386 +0xc5 github.com/gravitational/teleport/lib/client.(*KubeSession).pipeInOut.func1() /__w/teleport/teleport/lib/client/kubesession.go:208 +0x8d Goroutine 64284 (running) created at: testing.(*T).Run() /opt/go/src/testing/testing.go:1629 +0x805 github.com/gravitational/teleport/integration.testKubeJoin() /__w/teleport/teleport/integration/kube_integration_test.go:1691 +0x2b79 github.com/gravitational/teleport/integration.(*KubeSuite).bind.func1() /__w/teleport/teleport/integration/kube_integration_test.go:152 +0x118 testing.tRunner() /opt/go/src/testing/testing.go:1576 +0x216 testing.(*T).Run.func1() /opt/go/src/testing/testing.go:1629 +0x47 Goroutine 63051 (running) created at: github.com/gravitational/teleport/lib/client.(*KubeSession).pipeInOut() /__w/teleport/teleport/lib/client/kubesession.go:206 +0xea github.com/gravitational/teleport/lib/client.NewKubeSession() /__w/teleport/teleport/lib/client/kubesession.go:120 +0x12cb github.com/gravitational/teleport/integration.kubeJoin() /__w/teleport/teleport/integration/kube_integration_test.go:1518 +0x12a github.com/gravitational/teleport/integration.kubeJoinObserverWithSNISet() /__w/teleport/teleport/integration/kube_integration_test.go:1750 +0x34b github.com/gravitational/teleport/integration.kubeJoinByALBAddr() /__w/teleport/teleport/integration/kube_integration_test.go:1740 +0x249 github.com/gravitational/teleport/integration.testKubeJoin.func5() /__w/teleport/teleport/integration/kube_integration_test.go:1663 +0x14e testing.tRunner() /opt/go/src/testing/testing.go:1576 +0x216 testing.(*T).Run.func1() /opt/go/src/testing/testing.go:1629 +0x47 ================== testing.go:1446: race detected during execution of test ```

camscale · 2023-05-22T06:45:46Z

 	}

-	headers := req.Header
+	headers := http.Header{}


Is this change meant to be in this PR? I'm not sure how it is related to the kube session data races - it doesn't appear in the data race traces, and there are no tests updated for it.

This is a typo I introduce in #25202 but is now covered by this PR and by #26657.

Only one will deliver the change but both need them.

public-teleport-github-review-bot · 2023-05-22T21:06:18Z

@tigrato See the table below for backport results.

Branch	Result
branch/v13	Failed

nklaassen · 2023-05-30T16:49:01Z

@tigrato I got a hit on v13, can we backport this? https://github.com/gravitational/teleport/actions/runs/5113646678/jobs/9193093441?pr=26970

* Fix `TestKube/Join` data race PR #26168 extended the inital Kube Join test to add more observers using different upgrade logic. While extending it, it also introduced the following dataraces ``` ================== WARNING: DATA RACE Read at 0x00c00139d548 by goroutine 64284: bytes.(*Buffer).String() /opt/go/src/bytes/buffer.go:65 +0x252 github.com/gravitational/teleport/integration.testKubeJoin.func7() /__w/teleport/teleport/integration/kube_integration_test.go:1702 +0x245 testing.tRunner() /opt/go/src/testing/testing.go:1576 +0x216 testing.(*T).Run.func1() /opt/go/src/testing/testing.go:1629 +0x47 Previous write at 0x00c00139d548 by goroutine 63051: bytes.(*Buffer).grow() /opt/go/src/bytes/buffer.go:145 +0x3c4 bytes.(*Buffer).Write() /opt/go/src/bytes/buffer.go:170 +0xcd github.com/gravitational/teleport/lib/utils.(*SyncWriter).Write() /__w/teleport/teleport/lib/utils/sync_writer.go:38 +0x109 io.copyBuffer() /opt/go/src/io/io.go:429 +0x2de io.Copy() /opt/go/src/io/io.go:386 +0xc5 github.com/gravitational/teleport/lib/client.(*KubeSession).pipeInOut.func1() /__w/teleport/teleport/lib/client/kubesession.go:208 +0x8d Goroutine 64284 (running) created at: testing.(*T).Run() /opt/go/src/testing/testing.go:1629 +0x805 github.com/gravitational/teleport/integration.testKubeJoin() /__w/teleport/teleport/integration/kube_integration_test.go:1691 +0x2b79 github.com/gravitational/teleport/integration.(*KubeSuite).bind.func1() /__w/teleport/teleport/integration/kube_integration_test.go:152 +0x118 testing.tRunner() /opt/go/src/testing/testing.go:1576 +0x216 testing.(*T).Run.func1() /opt/go/src/testing/testing.go:1629 +0x47 Goroutine 63051 (running) created at: github.com/gravitational/teleport/lib/client.(*KubeSession).pipeInOut() /__w/teleport/teleport/lib/client/kubesession.go:206 +0xea github.com/gravitational/teleport/lib/client.NewKubeSession() /__w/teleport/teleport/lib/client/kubesession.go:120 +0x12cb github.com/gravitational/teleport/integration.kubeJoin() /__w/teleport/teleport/integration/kube_integration_test.go:1518 +0x12a github.com/gravitational/teleport/integration.kubeJoinObserverWithSNISet() /__w/teleport/teleport/integration/kube_integration_test.go:1750 +0x34b github.com/gravitational/teleport/integration.kubeJoinByALBAddr() /__w/teleport/teleport/integration/kube_integration_test.go:1740 +0x249 github.com/gravitational/teleport/integration.testKubeJoin.func5() /__w/teleport/teleport/integration/kube_integration_test.go:1663 +0x14e testing.tRunner() /opt/go/src/testing/testing.go:1576 +0x216 testing.(*T).Run.func1() /opt/go/src/testing/testing.go:1629 +0x47 ================== ================== WARNING: DATA RACE Read at 0x00c00139d530 by goroutine 64284: bytes.(*Buffer).String() /opt/go/src/bytes/buffer.go:65 +0x268 github.com/gravitational/teleport/integration.testKubeJoin.func7() /__w/teleport/teleport/integration/kube_integration_test.go:1702 +0x245 testing.tRunner() /opt/go/src/testing/testing.go:1576 +0x216 testing.(*T).Run.func1() /opt/go/src/testing/testing.go:1629 +0x47 Previous write at 0x00c00139d530 by goroutine 63051: bytes.(*Buffer).grow() /opt/go/src/bytes/buffer.go:142 +0x2d6 bytes.(*Buffer).Write() /opt/go/src/bytes/buffer.go:170 +0xcd github.com/gravitational/teleport/lib/utils.(*SyncWriter).Write() /__w/teleport/teleport/lib/utils/sync_writer.go:38 +0x109 io.copyBuffer() /opt/go/src/io/io.go:429 +0x2de io.Copy() /opt/go/src/io/io.go:386 +0xc5 github.com/gravitational/teleport/lib/client.(*KubeSession).pipeInOut.func1() /__w/teleport/teleport/lib/client/kubesession.go:208 +0x8d Goroutine 64284 (running) created at: testing.(*T).Run() /opt/go/src/testing/testing.go:1629 +0x805 github.com/gravitational/teleport/integration.testKubeJoin() /__w/teleport/teleport/integration/kube_integration_test.go:1691 +0x2b79 github.com/gravitational/teleport/integration.(*KubeSuite).bind.func1() /__w/teleport/teleport/integration/kube_integration_test.go:152 +0x118 testing.tRunner() /opt/go/src/testing/testing.go:1576 +0x216 testing.(*T).Run.func1() /opt/go/src/testing/testing.go:1629 +0x47 Goroutine 63051 (running) created at: github.com/gravitational/teleport/lib/client.(*KubeSession).pipeInOut() /__w/teleport/teleport/lib/client/kubesession.go:206 +0xea github.com/gravitational/teleport/lib/client.NewKubeSession() /__w/teleport/teleport/lib/client/kubesession.go:120 +0x12cb github.com/gravitational/teleport/integration.kubeJoin() /__w/teleport/teleport/integration/kube_integration_test.go:1518 +0x12a github.com/gravitational/teleport/integration.kubeJoinObserverWithSNISet() /__w/teleport/teleport/integration/kube_integration_test.go:1750 +0x34b github.com/gravitational/teleport/integration.kubeJoinByALBAddr() /__w/teleport/teleport/integration/kube_integration_test.go:1740 +0x249 github.com/gravitational/teleport/integration.testKubeJoin.func5() /__w/teleport/teleport/integration/kube_integration_test.go:1663 +0x14e testing.tRunner() /opt/go/src/testing/testing.go:1576 +0x216 testing.(*T).Run.func1() /opt/go/src/testing/testing.go:1629 +0x47 ================== ================== WARNING: DATA RACE Read at 0x00c0023c2d00 by goroutine 64284: runtime.slicebytetostring() /opt/go/src/runtime/string.go:81 +0x0 bytes.(*Buffer).String() /opt/go/src/bytes/buffer.go:65 +0x2a6 github.com/gravitational/teleport/integration.testKubeJoin.func7() /__w/teleport/teleport/integration/kube_integration_test.go:1702 +0x245 testing.tRunner() /opt/go/src/testing/testing.go:1576 +0x216 testing.(*T).Run.func1() /opt/go/src/testing/testing.go:1629 +0x47 Previous write at 0x00c0023c2d00 by goroutine 63051: runtime.slicecopy() /opt/go/src/runtime/slice.go:310 +0x0 bytes.growSlice() /opt/go/src/bytes/buffer.go:241 +0x14d bytes.(*Buffer).grow() /opt/go/src/bytes/buffer.go:142 +0x2ba bytes.(*Buffer).Write() /opt/go/src/bytes/buffer.go:170 +0xcd github.com/gravitational/teleport/lib/utils.(*SyncWriter).Write() /__w/teleport/teleport/lib/utils/sync_writer.go:38 +0x109 io.copyBuffer() /opt/go/src/io/io.go:429 +0x2de io.Copy() /opt/go/src/io/io.go:386 +0xc5 github.com/gravitational/teleport/lib/client.(*KubeSession).pipeInOut.func1() /__w/teleport/teleport/lib/client/kubesession.go:208 +0x8d Goroutine 64284 (running) created at: testing.(*T).Run() /opt/go/src/testing/testing.go:1629 +0x805 github.com/gravitational/teleport/integration.testKubeJoin() /__w/teleport/teleport/integration/kube_integration_test.go:1691 +0x2b79 github.com/gravitational/teleport/integration.(*KubeSuite).bind.func1() /__w/teleport/teleport/integration/kube_integration_test.go:152 +0x118 testing.tRunner() /opt/go/src/testing/testing.go:1576 +0x216 testing.(*T).Run.func1() /opt/go/src/testing/testing.go:1629 +0x47 Goroutine 63051 (running) created at: github.com/gravitational/teleport/lib/client.(*KubeSession).pipeInOut() /__w/teleport/teleport/lib/client/kubesession.go:206 +0xea github.com/gravitational/teleport/lib/client.NewKubeSession() /__w/teleport/teleport/lib/client/kubesession.go:120 +0x12cb github.com/gravitational/teleport/integration.kubeJoin() /__w/teleport/teleport/integration/kube_integration_test.go:1518 +0x12a github.com/gravitational/teleport/integration.kubeJoinObserverWithSNISet() /__w/teleport/teleport/integration/kube_integration_test.go:1750 +0x34b github.com/gravitational/teleport/integration.kubeJoinByALBAddr() /__w/teleport/teleport/integration/kube_integration_test.go:1740 +0x249 github.com/gravitational/teleport/integration.testKubeJoin.func5() /__w/teleport/teleport/integration/kube_integration_test.go:1663 +0x14e testing.tRunner() /opt/go/src/testing/testing.go:1576 +0x216 testing.(*T).Run.func1() /opt/go/src/testing/testing.go:1629 +0x47 ================== testing.go:1446: race detected during execution of test ``` * fix multi write * improve blocked case

tigrato · 2023-05-30T16:56:50Z

@nklaassen
Sure. I completely forgot to backport it

#27124

* Fix `TestKube/Join` data race PR #26168 extended the inital Kube Join test to add more observers using different upgrade logic. While extending it, it also introduced the following dataraces ``` ================== WARNING: DATA RACE Read at 0x00c00139d548 by goroutine 64284: bytes.(*Buffer).String() /opt/go/src/bytes/buffer.go:65 +0x252 github.com/gravitational/teleport/integration.testKubeJoin.func7() /__w/teleport/teleport/integration/kube_integration_test.go:1702 +0x245 testing.tRunner() /opt/go/src/testing/testing.go:1576 +0x216 testing.(*T).Run.func1() /opt/go/src/testing/testing.go:1629 +0x47 Previous write at 0x00c00139d548 by goroutine 63051: bytes.(*Buffer).grow() /opt/go/src/bytes/buffer.go:145 +0x3c4 bytes.(*Buffer).Write() /opt/go/src/bytes/buffer.go:170 +0xcd github.com/gravitational/teleport/lib/utils.(*SyncWriter).Write() /__w/teleport/teleport/lib/utils/sync_writer.go:38 +0x109 io.copyBuffer() /opt/go/src/io/io.go:429 +0x2de io.Copy() /opt/go/src/io/io.go:386 +0xc5 github.com/gravitational/teleport/lib/client.(*KubeSession).pipeInOut.func1() /__w/teleport/teleport/lib/client/kubesession.go:208 +0x8d Goroutine 64284 (running) created at: testing.(*T).Run() /opt/go/src/testing/testing.go:1629 +0x805 github.com/gravitational/teleport/integration.testKubeJoin() /__w/teleport/teleport/integration/kube_integration_test.go:1691 +0x2b79 github.com/gravitational/teleport/integration.(*KubeSuite).bind.func1() /__w/teleport/teleport/integration/kube_integration_test.go:152 +0x118 testing.tRunner() /opt/go/src/testing/testing.go:1576 +0x216 testing.(*T).Run.func1() /opt/go/src/testing/testing.go:1629 +0x47 Goroutine 63051 (running) created at: github.com/gravitational/teleport/lib/client.(*KubeSession).pipeInOut() /__w/teleport/teleport/lib/client/kubesession.go:206 +0xea github.com/gravitational/teleport/lib/client.NewKubeSession() /__w/teleport/teleport/lib/client/kubesession.go:120 +0x12cb github.com/gravitational/teleport/integration.kubeJoin() /__w/teleport/teleport/integration/kube_integration_test.go:1518 +0x12a github.com/gravitational/teleport/integration.kubeJoinObserverWithSNISet() /__w/teleport/teleport/integration/kube_integration_test.go:1750 +0x34b github.com/gravitational/teleport/integration.kubeJoinByALBAddr() /__w/teleport/teleport/integration/kube_integration_test.go:1740 +0x249 github.com/gravitational/teleport/integration.testKubeJoin.func5() /__w/teleport/teleport/integration/kube_integration_test.go:1663 +0x14e testing.tRunner() /opt/go/src/testing/testing.go:1576 +0x216 testing.(*T).Run.func1() /opt/go/src/testing/testing.go:1629 +0x47 ================== ================== WARNING: DATA RACE Read at 0x00c00139d530 by goroutine 64284: bytes.(*Buffer).String() /opt/go/src/bytes/buffer.go:65 +0x268 github.com/gravitational/teleport/integration.testKubeJoin.func7() /__w/teleport/teleport/integration/kube_integration_test.go:1702 +0x245 testing.tRunner() /opt/go/src/testing/testing.go:1576 +0x216 testing.(*T).Run.func1() /opt/go/src/testing/testing.go:1629 +0x47 Previous write at 0x00c00139d530 by goroutine 63051: bytes.(*Buffer).grow() /opt/go/src/bytes/buffer.go:142 +0x2d6 bytes.(*Buffer).Write() /opt/go/src/bytes/buffer.go:170 +0xcd github.com/gravitational/teleport/lib/utils.(*SyncWriter).Write() /__w/teleport/teleport/lib/utils/sync_writer.go:38 +0x109 io.copyBuffer() /opt/go/src/io/io.go:429 +0x2de io.Copy() /opt/go/src/io/io.go:386 +0xc5 github.com/gravitational/teleport/lib/client.(*KubeSession).pipeInOut.func1() /__w/teleport/teleport/lib/client/kubesession.go:208 +0x8d Goroutine 64284 (running) created at: testing.(*T).Run() /opt/go/src/testing/testing.go:1629 +0x805 github.com/gravitational/teleport/integration.testKubeJoin() /__w/teleport/teleport/integration/kube_integration_test.go:1691 +0x2b79 github.com/gravitational/teleport/integration.(*KubeSuite).bind.func1() /__w/teleport/teleport/integration/kube_integration_test.go:152 +0x118 testing.tRunner() /opt/go/src/testing/testing.go:1576 +0x216 testing.(*T).Run.func1() /opt/go/src/testing/testing.go:1629 +0x47 Goroutine 63051 (running) created at: github.com/gravitational/teleport/lib/client.(*KubeSession).pipeInOut() /__w/teleport/teleport/lib/client/kubesession.go:206 +0xea github.com/gravitational/teleport/lib/client.NewKubeSession() /__w/teleport/teleport/lib/client/kubesession.go:120 +0x12cb github.com/gravitational/teleport/integration.kubeJoin() /__w/teleport/teleport/integration/kube_integration_test.go:1518 +0x12a github.com/gravitational/teleport/integration.kubeJoinObserverWithSNISet() /__w/teleport/teleport/integration/kube_integration_test.go:1750 +0x34b github.com/gravitational/teleport/integration.kubeJoinByALBAddr() /__w/teleport/teleport/integration/kube_integration_test.go:1740 +0x249 github.com/gravitational/teleport/integration.testKubeJoin.func5() /__w/teleport/teleport/integration/kube_integration_test.go:1663 +0x14e testing.tRunner() /opt/go/src/testing/testing.go:1576 +0x216 testing.(*T).Run.func1() /opt/go/src/testing/testing.go:1629 +0x47 ================== ================== WARNING: DATA RACE Read at 0x00c0023c2d00 by goroutine 64284: runtime.slicebytetostring() /opt/go/src/runtime/string.go:81 +0x0 bytes.(*Buffer).String() /opt/go/src/bytes/buffer.go:65 +0x2a6 github.com/gravitational/teleport/integration.testKubeJoin.func7() /__w/teleport/teleport/integration/kube_integration_test.go:1702 +0x245 testing.tRunner() /opt/go/src/testing/testing.go:1576 +0x216 testing.(*T).Run.func1() /opt/go/src/testing/testing.go:1629 +0x47 Previous write at 0x00c0023c2d00 by goroutine 63051: runtime.slicecopy() /opt/go/src/runtime/slice.go:310 +0x0 bytes.growSlice() /opt/go/src/bytes/buffer.go:241 +0x14d bytes.(*Buffer).grow() /opt/go/src/bytes/buffer.go:142 +0x2ba bytes.(*Buffer).Write() /opt/go/src/bytes/buffer.go:170 +0xcd github.com/gravitational/teleport/lib/utils.(*SyncWriter).Write() /__w/teleport/teleport/lib/utils/sync_writer.go:38 +0x109 io.copyBuffer() /opt/go/src/io/io.go:429 +0x2de io.Copy() /opt/go/src/io/io.go:386 +0xc5 github.com/gravitational/teleport/lib/client.(*KubeSession).pipeInOut.func1() /__w/teleport/teleport/lib/client/kubesession.go:208 +0x8d Goroutine 64284 (running) created at: testing.(*T).Run() /opt/go/src/testing/testing.go:1629 +0x805 github.com/gravitational/teleport/integration.testKubeJoin() /__w/teleport/teleport/integration/kube_integration_test.go:1691 +0x2b79 github.com/gravitational/teleport/integration.(*KubeSuite).bind.func1() /__w/teleport/teleport/integration/kube_integration_test.go:152 +0x118 testing.tRunner() /opt/go/src/testing/testing.go:1576 +0x216 testing.(*T).Run.func1() /opt/go/src/testing/testing.go:1629 +0x47 Goroutine 63051 (running) created at: github.com/gravitational/teleport/lib/client.(*KubeSession).pipeInOut() /__w/teleport/teleport/lib/client/kubesession.go:206 +0xea github.com/gravitational/teleport/lib/client.NewKubeSession() /__w/teleport/teleport/lib/client/kubesession.go:120 +0x12cb github.com/gravitational/teleport/integration.kubeJoin() /__w/teleport/teleport/integration/kube_integration_test.go:1518 +0x12a github.com/gravitational/teleport/integration.kubeJoinObserverWithSNISet() /__w/teleport/teleport/integration/kube_integration_test.go:1750 +0x34b github.com/gravitational/teleport/integration.kubeJoinByALBAddr() /__w/teleport/teleport/integration/kube_integration_test.go:1740 +0x249 github.com/gravitational/teleport/integration.testKubeJoin.func5() /__w/teleport/teleport/integration/kube_integration_test.go:1663 +0x14e testing.tRunner() /opt/go/src/testing/testing.go:1576 +0x216 testing.(*T).Run.func1() /opt/go/src/testing/testing.go:1629 +0x47 ================== testing.go:1446: race detected during execution of test ``` * fix multi write * improve blocked case

tigrato requested a review from greedy52 May 19, 2023 16:41

tigrato added the backport/branch/v13 label May 19, 2023

github-actions Bot requested review from camscale and r0mant May 19, 2023 16:42

github-actions Bot added the size/sm label May 19, 2023

greedy52 approved these changes May 19, 2023

View reviewed changes

fix multi write

6fafa24

tigrato force-pushed the tigrato/fix-kube-integration-datarace branch from e105536 to 6fafa24 Compare May 19, 2023 18:28

camscale approved these changes May 22, 2023

View reviewed changes

tigrato force-pushed the tigrato/fix-kube-integration-datarace branch from e7d7b35 to 6d639d3 Compare May 22, 2023 08:38

tigrato mentioned this pull request May 22, 2023

TestKube data race #26652

Closed

improve blocked case

d62e311

tigrato force-pushed the tigrato/fix-kube-integration-datarace branch from 6d639d3 to d62e311 Compare May 22, 2023 09:34

r0mant approved these changes May 22, 2023

View reviewed changes

tigrato added this pull request to the merge queue May 22, 2023

Merged via the queue into master with commit 41ed53e May 22, 2023

tigrato deleted the tigrato/fix-kube-integration-datarace branch May 22, 2023 21:05

tigrato mentioned this pull request May 30, 2023

[v13] Fix TestKube/Join data race #27124

Merged

r0mant mentioned this pull request Jul 14, 2023

Release 13.2.2 #29158

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix `TestKube/Join` data race#26619

Fix `TestKube/Join` data race#26619
tigrato merged 3 commits intomasterfrom
tigrato/fix-kube-integration-datarace

tigrato commented May 19, 2023 •

edited

Loading

Uh oh!

Uh oh!

camscale May 22, 2023

Uh oh!

tigrato May 22, 2023

Uh oh!

public-teleport-github-review-bot Bot commented May 22, 2023

Uh oh!

nklaassen commented May 30, 2023

Uh oh!

tigrato commented May 30, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Conversation

tigrato commented May 19, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

camscale May 22, 2023

Choose a reason for hiding this comment

Uh oh!

tigrato May 22, 2023

Choose a reason for hiding this comment

Uh oh!

public-teleport-github-review-bot Bot commented May 22, 2023

Uh oh!

nklaassen commented May 30, 2023

Uh oh!

tigrato commented May 30, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

tigrato commented May 19, 2023 •

edited

Loading