Skip to content

roachtest: cdc/pubsub-sink failed #110637

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
cockroach-teamcity opened this issue Sep 14, 2023 · 1 comment · Fixed by #110859
Closed

roachtest: cdc/pubsub-sink failed #110637

cockroach-teamcity opened this issue Sep 14, 2023 · 1 comment · Fixed by #110859
Assignees
@jayshrivastava
Labels
A-cdc Change Data Capture branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. T-cdc
Milestone
23.2

Comments

@cockroach-teamcity
Copy link
Member

cockroach-teamcity commented Sep 14, 2023 •
edited by cockroach-jira-scripts
Loading

roachtest.cdc/pubsub-sink failed with artifacts on master @ 0cde11b885a5e156b970c003505acd081fb4f326:

(latency_verifier.go:192).assertValid: latency never dropped to acceptable steady level: 1m0s
(monitor.go:153).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/cdc/pubsub-sink/run_1

Parameters: ROACHTEST_arch=amd64 , ROACHTEST_cloud=gce , ROACHTEST_cpu=16 , ROACHTEST_encrypted=false , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=true , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

/cc @cockroachdb/cdc

This test on roachdash | Improve this report!

Jira issue: CRDB-31537
@cockroach-teamcity cockroach-teamcity added branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. T-cdc labels Sep 14, 2023
@cockroach-teamcity cockroach-teamcity added this to the 23.2 milestone Sep 14, 2023
@blathers-crl blathers-crl bot added the A-cdc Change Data Capture label Sep 14, 2023
@jayshrivastava
Copy link
Contributor

Initial scan finishes at 19:19

changefeed: 11:19:19 latency_verifier.go:96: initial scan completed: latency 4m15.464334652s, highwater 2023-09-14 11:15:03.782606239 +0000 UTC

After the initial scan finishes, there is a transient error.

W230914 11:19:22.977526 18931 ccl/changefeedccl/changefeed_stmt.go:1310 ⋮ [T1,Vsystem,n1,job=‹CHANGEFEED id=899873845399322625›] 328  Changefeed job 899873845399322625 encountered transient error: grpc: ‹Invalid resource name given (name=projects/cockroach-ephemeral/topics/). Refer to https://cloud.google.com/pubsub/docs/admin#resource_names for more information.› [code 3/InvalidArgument] (attempt 1)

This happens several times and prevents the changefeed from advancing.

jayshrivastava added a commit to jayshrivastava/cockroach that referenced this issue Sep 18, 2023
@jayshrivastava
changefeedccl: ensure pubsub sink emits resolved messages to all topics
6654989 
Previously, the pubsub sink would emit one resolved message with an empty
topic, causing an error. This change ensures that it emits resolved
messages to each topic and the topic string is not empty. This behavior
aligns with what the old, now deprecated, pubsub sink.

Fixes: cockroachdb#110637
Release note: None
@jayshrivastava jayshrivastava mentioned this issue Sep 18, 2023
Merged
craig bot pushed a commit that referenced this issue Sep 19, 2023
@nvanbenschoten @jayshrivastava
Merge #110395 #110859
baeffe5 
110395: storage: check for conflicting replicated locks in mvccPutInternal r=nvanbenschoten a=nvanbenschoten

Fixes #109647.
Informs #100193.

This commit uses the `lockTableKeyScanner` introduced in #110323 to scan for conflicting replicated locks at the same time as we can for intents (the writer's or others') when performing mutations in mvccPutInternal. In doing so, the commit removes use of `intentInterleavingIter` on the mutation path, opting for the explicit scan of the lock table instead.

Use of the `lockTableKeyScanner` to both check for conflicting locks and to scan for intents ensures that in the common case where there are no locks on a key, we only perform one (prefix) seek. This should avoid any regression. I'll collect some benchmark results to verify that this reasoning holds and that this does not cause a performance regression for write-heavy workloads.

Release note: None

110859: changefeedccl: ensure pubsub sink emits resolved messages to all topics r=jayshrivastava a=jayshrivastava

Previously, the pubsub sink would emit one resolved message with an empty
topic, causing an error. This change ensures that it emits resolved
messages to each topic and the topic string is not empty. This behavior
aligns with what the old, now deprecated, pubsub sink.

Fixes: #110637
Release note: None

Co-authored-by: Nathan VanBenschoten <[email protected]>
Co-authored-by: Jayant Shrivastava <[email protected]>
@craig craig bot closed this as completed in b92e292 Sep 19, 2023
blathers-crl bot pushed a commit that referenced this issue Sep 19, 2023
@jayshrivastava
changefeedccl: ensure pubsub sink emits resolved messages to all topics
a042242 
Previously, the pubsub sink would emit one resolved message with an empty
topic, causing an error. This change ensures that it emits resolved
messages to each topic and the topic string is not empty. This behavior
aligns with what the old, now deprecated, pubsub sink.

Fixes: #110637
Release note: None
@blathers-crl blathers-crl bot mentioned this issue Sep 19, 2023
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@jayshrivastava jayshrivastava

Labels
A-cdc Change Data Capture branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. T-cdc
Projects
None yet
Milestone
23.2
Development

Successfully merging a pull request may close this issue.

changefeedccl: ensure pubsub sink emits resolved messages to all topics jayshrivastava/cockroach
2 participants
@jayshrivastava @cockroach-teamcity