Skip to content

Local backend: handle canceled steps case#6008

Merged
6543 merged 5 commits into
woodpecker-ci:mainfrom
6543-forks:local-backend_handle-canceled-steps-case
Jan 25, 2026
Merged

Local backend: handle canceled steps case#6008
6543 merged 5 commits into
woodpecker-ci:mainfrom
6543-forks:local-backend_handle-canceled-steps-case

Conversation

@6543

@6543 6543 commented Jan 25, 2026

Copy link
Copy Markdown
Member

currently if you manually cancel or a new pipeline cancels a pipeline automatically and you use the local backend, the agent just crashes:

woodpecker-agent[2524]: panic: runtime error: invalid memory address or nil pointer dereference
woodpecker-agent[2524]: [signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x1be0abc]
woodpecker-agent[2524]: goroutine 119 [running]:
woodpecker-agent[2524]: go.woodpecker-ci.org/woodpecker/v3/pipeline/backend/local.(*local).WaitStep(0x37d6040, {0x1?, 0xc000252be0?}, 0xc00029af00, {0xc0003c23e5, 0x5})
woodpecker-agent[2524]:         go.woodpecker-ci.org/woodpecker/v3/pipeline/backend/local/local.go:193 +0x13c
woodpecker-agent[2524]: go.woodpecker-ci.org/woodpecker/v3/pipeline.(*Runtime).exec(0xc000336a80, 0xc00029af00)
woodpecker-agent[2524]:         go.woodpecker-ci.org/woodpecker/v3/pipeline/pipeline.go:284 +0x182
woodpecker-agent[2524]: go.woodpecker-ci.org/woodpecker/v3/pipeline.(*Runtime).execAll.func1()
woodpecker-agent[2524]:         go.woodpecker-ci.org/woodpecker/v3/pipeline/pipeline.go:229 +0x1ba
woodpecker-agent[2524]: golang.org/x/sync/errgroup.(*Group).Go.func1()
woodpecker-agent[2524]:         golang.org/x/sync@v0.18.0/errgroup/errgroup.go:93 +0x50
woodpecker-agent[2524]: created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 142
woodpecker-agent[2524]:         golang.org/x/sync@v0.18.0/errgroup/errgroup.go:78 +0x95
systemd[1]: woodpecker-agent-local.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
systemd[1]: woodpecker-agent-local.service: Failed with result 'exit-code'.
systemd[1]: Stopped Woodpecker-Agent Service - local.

(log of v3.12.0)


Sponsored by Kithara Software GmbH

@6543 6543 added bug Something isn't working backend/local labels Jan 25, 2026
@6543

6543 commented Jan 25, 2026

Copy link
Copy Markdown
Member Author

c8c7626 just adds the other case back where a command just runs too fast to have it even Wait for it ...

@codecov

codecov Bot commented Jan 25, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 69.04762% with 13 lines in your changes missing coverage. Please review.
✅ Project coverage is 21.87%. Comparing base (bd8e2f3) to head (eff2665).
⚠️ Report is 10 commits behind head on main.

Files with missing lines Patch % Lines
pipeline/backend/local/local.go 69.04% 8 Missing and 5 partials ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #6008      +/-   ##
==========================================
+ Coverage   21.85%   21.87%   +0.02%     
==========================================
  Files         432      432              
  Lines       39203    39229      +26     
==========================================
+ Hits         8567     8581      +14     
- Misses      29831    29840       +9     
- Partials      805      808       +3     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@6543

6543 commented Jan 25, 2026

Copy link
Copy Markdown
Member Author

error on c663941:

5:22PM DBG cmd/agent/core/agent.go:299 > polling new steps
5:22PM DBG agent/runner.go:54 > request next execution
5:23PM DBG agent/runner.go:90 > received execution pipeline=2 repo=test/test-bash-ci workflow_id=2
5:23PM DBG agent/runner.go:104 > listen for cancel signal pipeline=2 repo=test/test-bash-ci workflow_id=2
5:23PM DBG pipeline/pipeline.go:95 > executing 1 stages, in order of: pipeline_number=2 repo=test/test-bash-ci workflow_id=2
5:23PM DBG pipeline/pipeline.go:105 > stage StagePos=0 Steps="sleep 30s" pipeline_number=2 repo=test/test-bash-ci workflow_id=2
5:23PM DBG pipeline/pipeline.go:200 > prepare pipeline_number=2 repo=test/test-bash-ci step="sleep 30s" workflow_id=2
5:23PM DBG agent/tracer.go:54 > update step status exit_code=0 exited=false image=bash pipeline=2 repo=test/test-bash-ci workflow_id=2
5:23PM DBG agent/tracer.go:62 > update step status complete exit_code=0 exited=false image=bash pipeline=2 repo=test/test-bash-ci workflow_id=2
5:23PM DBG pipeline/pipeline.go:227 > executing pipeline_number=2 repo=test/test-bash-ci step="sleep 30s" workflow_id=2
5:23PM DBG agent/logger.go:44 > log stream opened image=bash pipeline=2 repo=test/test-bash-ci workflow_id=2
5:23PM DBG agent/rpc/client_grpc.go:407 > log drain: sending queued logs bytes=128 entries=3
5:23PM ERR agent/rpc/client_grpc.go:180 > grpc error: wait(): code: Unknown error="rpc error: code = Unknown desc = queue: task canceled"
5:23PM WRN agent/runner.go:108 > cancel signal received error="rpc error: code = Unknown desc = queue: task canceled" pipeline=2 repo=test/test-bash-ci workflow_id=2
5:23PM DBG agent/runner.go:119 > pipeline done pipeline=2 repo=test/test-bash-ci workflow_id=2
5:23PM ERR agent/logger.go:48 > copy limited logStream part error="read |0: file already closed" image=bash pipeline=2 repo=test/test-bash-ci workflow_id=2
5:23PM DBG agent/logger.go:51 > log stream copied, close ... image=bash pipeline=2 repo=test/test-bash-ci workflow_id=2
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0xa0 pc=0x1aef071]

goroutine 97 [running]:
go.woodpecker-ci.org/woodpecker/v3/pipeline/backend/local.(*local).WaitStep(0x35a4d80, {0x1?, 0xc00016b400?}, 0xc00017f860, {0x247bd90, 0x1})
	woodpecker/pipeline/backend/local/local.go:185 +0xf1
go.woodpecker-ci.org/woodpecker/v3/pipeline.(*Runtime).exec(0xc0002300e0, 0xc00017f860)
	woodpecker/pipeline/pipeline.go:284 +0x182
go.woodpecker-ci.org/woodpecker/v3/pipeline.(*Runtime).execAll.func1()
	woodpecker/pipeline/pipeline.go:229 +0x1ba
golang.org/x/sync/errgroup.(*Group).Go.func1()
	woodpecker/vendor/golang.org/x/sync/errgroup/errgroup.go:93 +0x50
created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 47
	woodpecker/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x95

with this patch:

5:25PM DBG agent/runner.go:90 > received execution pipeline=3 repo=test/test-bash-ci workflow_id=3
5:25PM DBG agent/runner.go:104 > listen for cancel signal pipeline=3 repo=test/test-bash-ci workflow_id=3
5:25PM DBG pipeline/pipeline.go:95 > executing 1 stages, in order of: pipeline_number=3 repo=test/test-bash-ci workflow_id=3
5:25PM DBG pipeline/pipeline.go:105 > stage StagePos=0 Steps="sleep 30s" pipeline_number=3 repo=test/test-bash-ci workflow_id=3
5:25PM DBG pipeline/pipeline.go:200 > prepare pipeline_number=3 repo=test/test-bash-ci step="sleep 30s" workflow_id=3
5:25PM DBG agent/tracer.go:54 > update step status exit_code=0 exited=false image=bash pipeline=3 repo=test/test-bash-ci workflow_id=3
5:25PM DBG agent/tracer.go:62 > update step status complete exit_code=0 exited=false image=bash pipeline=3 repo=test/test-bash-ci workflow_id=3
5:25PM DBG pipeline/pipeline.go:227 > executing pipeline_number=3 repo=test/test-bash-ci step="sleep 30s" workflow_id=3
5:25PM DBG agent/logger.go:44 > log stream opened image=bash pipeline=3 repo=test/test-bash-ci workflow_id=3
5:25PM DBG agent/rpc/client_grpc.go:407 > log drain: sending queued logs bytes=128 entries=3
5:25PM ERR agent/rpc/client_grpc.go:180 > grpc error: wait(): code: Unknown error="rpc error: code = Unknown desc = queue: task canceled"
5:25PM WRN agent/runner.go:108 > cancel signal received error="rpc error: code = Unknown desc = queue: task canceled" pipeline=3 repo=test/test-bash-ci workflow_id=3
5:25PM DBG agent/runner.go:119 > pipeline done pipeline=3 repo=test/test-bash-ci workflow_id=3
5:25PM ERR agent/logger.go:48 > copy limited logStream part error="read |0: file already closed" image=bash pipeline=3 repo=test/test-bash-ci workflow_id=3
5:25PM DBG agent/logger.go:51 > log stream copied, close ... image=bash pipeline=3 repo=test/test-bash-ci workflow_id=3
5:25PM DBG pipeline/pipeline.go:233 > complete pipeline_number=3 repo=test/test-bash-ci step="sleep 30s" workflow_id=3
5:25PM DBG agent/runner.go:170 > workflow finished error=Canceled canceled=true pipeline=3 repo=test/test-bash-ci workflow_id=3
5:25PM DBG agent/tracer.go:54 > update step status error="workflow state not found" exit_code=126 exited=true image=bash pipeline=3 repo=test/test-bash-ci workflow_id=3
5:25PM DBG agent/runner.go:172 > uploading logs and traces / states ... pipeline=3 repo=test/test-bash-ci workflow_id=3
5:25PM DBG agent/tracer.go:62 > update step status complete error="workflow state not found" exit_code=126 exited=true image=bash pipeline=3 repo=test/test-bash-ci workflow_id=3
5:25PM DBG agent/runner.go:174 > uploaded logs and traces / states pipeline=3 repo=test/test-bash-ci workflow_id=3
5:25PM DBG agent/runner.go:178 > updating workflow status error=Canceled pipeline=3 repo=test/test-bash-ci workflow_id=3
5:25PM DBG agent/runner.go:187 > updating workflow status complete pipeline=3 repo=test/test-bash-ci workflow_id=3
5:25PM DBG cmd/agent/core/agent.go:299 > polling new steps
5:25PM DBG agent/runner.go:54 > request next execution`
image

@6543

6543 commented Jan 25, 2026

Copy link
Copy Markdown
Member Author

will merge now but I think there should be a other error message :D

@6543 6543 merged commit 9d21e09 into woodpecker-ci:main Jan 25, 2026
9 checks passed
@6543 6543 deleted the local-backend_handle-canceled-steps-case branch January 25, 2026 16:27
@6543

6543 commented Jan 25, 2026

Copy link
Copy Markdown
Member Author

-> will work on that now

@woodpecker-bot woodpecker-bot mentioned this pull request Jan 25, 2026
1 task
@6543 6543 changed the title Local backend handle canceled steps case Local backend: handle canceled steps case Jan 28, 2026
@woodpecker-bot woodpecker-bot mentioned this pull request Apr 1, 2026
1 task
@woodpecker-bot woodpecker-bot mentioned this pull request Apr 15, 2026
1 task
@woodpecker-bot woodpecker-bot mentioned this pull request Apr 27, 2026
1 task
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backend/local bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants