Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Hatchet run failed with Hatchet::Reaper::AlreadyDeletedError #199

Closed
edmorley opened this issue Mar 2, 2023 · 2 comments · Fixed by #201
Closed

Hatchet run failed with Hatchet::Reaper::AlreadyDeletedError #199

edmorley opened this issue Mar 2, 2023 · 2 comments · Fixed by #201

Comments

@edmorley
Copy link
Member

edmorley commented Mar 2, 2023 •
edited
Loading

This CI run (which was using Hatchet 8.0.1):
https://dashboard.heroku.com/pipelines/ac057663-170b-4bdd-99d0-87560eb3a570/tests/1820
heroku/heroku-buildpack-ruby#1365

...failed with:

-----> Running test command `bundle exec rspec-queue --max-requeues=3 --timeout 180 --queue $REDIS_URL || { cat log/test_order.log;  $(exit 1); }`...
-----> Detecting rails configuration
.       Running: rake assets:precompile
       success!
.Hatchet setup: "hatchet-t-10f87460ec" for "default_ruby"
Destroying "hatchet-t-10f87460ec": 12c81d2b-3f12-4d0f-99de-6df74142fe5d, (0/100) reason: teardown
....-----> Installing bundler 1.17.3
-----> Removing BUNDLED WITH version in the Gemfile.lock
.....Hatchet setup: "hatchet-t-85b2997121" for "ruby-getting-started"
..Destroying "hatchet-t-85b2997121": 89cea7be-2537-4afe-b1a7-1e19870dcc72, (0/100) reason: teardown
RSpec::Retry: 2nd try ./spec/hatchet/rails5_spec.rb:4
Hatchet setup: "hatchet-t-da38ef5878" for "ruby-getting-started"
..Destroying "hatchet-t-da38ef5878": 1802ece3-3e93-4003-ac40-fd2845ace628, (0/100) reason: teardown
F
Failures:
  1) Rails 5 behaves correctly, not related to Rails 5 at all
     Failure/Error:
           Hatchet::Runner.new("ruby-getting-started", stack: "heroku-18").deploy do |app, heroku|
             # Test BUNDLE_DISABLE_VERSION_CHECK works
             expect(app.output).not_to include("The latest bundler is")
       
             # Test worker task only appears if the app has that rake task
             worker_task = worker_task_for_app(app)
             expect(worker_task).to be_nil
       
             run!(%Q{echo "task 'jobs:work' do ; end" >> Rakefile})
             app.commit!
     Hatchet::Reaper::AlreadyDeletedError:
       Duplicate destroy attempted "hatchet-t-da38ef5878": 1802ece3-3e93-4003-ac40-fd2845ace628, status: 404, request_id: 6a428543-0fd2-4b4a-8563-c14ee69e5941,2338922a-a077-b88b-a465-2bdd808980d1,a3062abd-3611-25f2-9e47-7c821df541e6
     # ./vendor/bundle/ruby/3.1.0/gems/heroku_hatchet-8.0.1/lib/hatchet/reaper.rb:195:in `rescue in destroy_with_log'
     # ./vendor/bundle/ruby/3.1.0/gems/heroku_hatchet-8.0.1/lib/hatchet/reaper.rb:184:in `destroy_with_log'
     # ./vendor/bundle/ruby/3.1.0/gems/heroku_hatchet-8.0.1/lib/hatchet/app.rb:381:in `teardown!'
     # ./vendor/bundle/ruby/3.1.0/gems/heroku_hatchet-8.0.1/lib/hatchet/app.rb:433:in `deploy'
     # ./spec/hatchet/rails5_spec.rb:5:in `block (2 levels) in <top (required)>'
     # ./vendor/bundle/ruby/3.1.0/gems/rspec-core-3.12.1/lib/rspec/core/example.rb:263:in `instance_exec'
     # ./vendor/bundle/ruby/3.1.0/gems/rspec-core-3.12.1/lib/rspec/core/example.rb:263:in `block in run'
     # ./vendor/bundle/ruby/3.1.0/gems/rspec-core-3.12.1/lib/rspec/core/example.rb:511:in `block in with_around_and_singleton_context_hooks'
     # ./vendor/bundle/ruby/3.1.0/gems/rspec-core-3.12.1/lib/rspec/core/example.rb:468:in `block in with_around_example_hooks'
     # ./vendor/bundle/ruby/3.1.0/gems/rspec-core-3.12.1/lib/rspec/core/hooks.rb:486:in `block in run'
     # ./vendor/bundle/ruby/3.1.0/gems/rspec-core-3.12.1/lib/rspec/core/hooks.rb:626:in `block in run_around_example_hooks_for'
     # ./vendor/bundle/ruby/3.1.0/gems/rspec-core-3.12.1/lib/rspec/core/example.rb:352:in `call'
     # ./vendor/bundle/ruby/3.1.0/gems/rspec-retry-0.6.2/lib/rspec/retry.rb:124:in `block in run'
     # ./vendor/bundle/ruby/3.1.0/gems/rspec-retry-0.6.2/lib/rspec/retry.rb:110:in `loop'
     # ./vendor/bundle/ruby/3.1.0/gems/rspec-retry-0.6.2/lib/rspec/retry.rb:110:in `run'
     # ./vendor/bundle/ruby/3.1.0/gems/rspec-retry-0.6.2/lib/rspec_ext/rspec_ext.rb:12:in `run_with_retry'
     # ./vendor/bundle/ruby/3.1.0/gems/rspec-retry-0.6.2/lib/rspec/retry.rb:37:in `block (2 levels) in setup'
     # ./vendor/bundle/ruby/3.1.0/gems/rspec-core-3.12.1/lib/rspec/core/example.rb:457:in `instance_exec'
     # ./vendor/bundle/ruby/3.1.0/gems/rspec-core-3.12.1/lib/rspec/core/example.rb:457:in `instance_exec'
     # ./vendor/bundle/ruby/3.1.0/gems/rspec-core-3.12.1/lib/rspec/core/hooks.rb:390:in `execute_with'
     # ./vendor/bundle/ruby/3.1.0/gems/rspec-core-3.12.1/lib/rspec/core/hooks.rb:628:in `block (2 levels) in run_around_example_hooks_for'
     # ./vendor/bundle/ruby/3.1.0/gems/rspec-core-3.12.1/lib/rspec/core/example.rb:352:in `call'
     # ./vendor/bundle/ruby/3.1.0/gems/rspec-core-3.12.1/lib/rspec/core/hooks.rb:629:in `run_around_example_hooks_for'
     # ./vendor/bundle/ruby/3.1.0/gems/rspec-core-3.12.1/lib/rspec/core/hooks.rb:486:in `run'
     # ./vendor/bundle/ruby/3.1.0/gems/rspec-core-3.12.1/lib/rspec/core/example.rb:468:in `with_around_example_hooks'
     # ./vendor/bundle/ruby/3.1.0/gems/rspec-core-3.12.1/lib/rspec/core/example.rb:511:in `with_around_and_singleton_context_hooks'
     # ./vendor/bundle/ruby/3.1.0/gems/rspec-core-3.12.1/lib/rspec/core/example.rb:259:in `run'
     # ./vendor/bundle/ruby/3.1.0/gems/ci-queue-0.24.2/lib/rspec/queue.rb:267:in `run'
     # ./vendor/bundle/ruby/3.1.0/gems/ci-queue-0.24.2/lib/rspec/queue.rb:404:in `block (3 levels) in run_specs'
     # ./vendor/bundle/ruby/3.1.0/gems/ci-queue-0.24.2/lib/ci/queue/redis/worker.rb:51:in `poll'
     # ./vendor/bundle/ruby/3.1.0/gems/ci-queue-0.24.2/lib/rspec/queue.rb:403:in `block (2 levels) in run_specs'
     # ./vendor/bundle/ruby/3.1.0/gems/rspec-core-3.12.1/lib/rspec/core/configuration.rb:2070:in `with_suite_hooks'
     # ./vendor/bundle/ruby/3.1.0/gems/ci-queue-0.24.2/lib/rspec/queue.rb:401:in `block in run_specs'
     # ./vendor/bundle/ruby/3.1.0/gems/rspec-core-3.12.1/lib/rspec/core/reporter.rb:74:in `report'
     # ./vendor/bundle/ruby/3.1.0/gems/ci-queue-0.24.2/lib/rspec/queue.rb:396:in `run_specs'
     # ./vendor/bundle/ruby/3.1.0/gems/rspec-core-3.12.1/lib/rspec/core/runner.rb:89:in `run'
     # ./vendor/bundle/ruby/3.1.0/gems/rspec-core-3.12.1/lib/rspec/core/runner.rb:71:in `run'
     # ./vendor/bundle/ruby/3.1.0/gems/rspec-core-3.12.1/lib/rspec/core/runner.rb:45:in `invoke'
     # ./vendor/bundle/ruby/3.1.0/gems/ci-queue-0.24.2/exe/rspec-queue:5:in `<top (required)>'
     # ./vendor/bundle/ruby/3.1.0/bin/rspec-queue:25:in `load'
     # ./vendor/bundle/ruby/3.1.0/bin/rspec-queue:25:in `<top (required)>'
     # ./vendor/bundle/ruby/3.1.0/gems/bundler-2.3.25/lib/bundler/cli/exec.rb:58:in `load'
     # ./vendor/bundle/ruby/3.1.0/gems/bundler-2.3.25/lib/bundler/cli/exec.rb:58:in `kernel_load'
     # ./vendor/bundle/ruby/3.1.0/gems/bundler-2.3.25/lib/bundler/cli/exec.rb:23:in `run'
     # ./vendor/bundle/ruby/3.1.0/gems/bundler-2.3.25/lib/bundler/cli.rb:486:in `exec'
     # ./vendor/bundle/ruby/3.1.0/gems/bundler-2.3.25/lib/bundler/vendor/thor/lib/thor/command.rb:27:in `run'
     # ./vendor/bundle/ruby/3.1.0/gems/bundler-2.3.25/lib/bundler/vendor/thor/lib/thor/invocation.rb:127:in `invoke_command'
     # ./vendor/bundle/ruby/3.1.0/gems/bundler-2.3.25/lib/bundler/vendor/thor/lib/thor.rb:392:in `dispatch'
     # ./vendor/bundle/ruby/3.1.0/gems/bundler-2.3.25/lib/bundler/cli.rb:31:in `dispatch'
     # ./vendor/bundle/ruby/3.1.0/gems/bundler-2.3.25/lib/bundler/vendor/thor/lib/thor/base.rb:485:in `start'
     # ./vendor/bundle/ruby/3.1.0/gems/bundler-2.3.25/lib/bundler/cli.rb:25:in `start'
     # ./vendor/bundle/ruby/3.1.0/gems/bundler-2.3.25/exe/bundle:48:in `block in <top (required)>'
     # ./vendor/bundle/ruby/3.1.0/gems/bundler-2.3.25/lib/bundler/friendly_errors.rb:120:in `with_friendly_errors'
     # ./vendor/bundle/ruby/3.1.0/gems/bundler-2.3.25/exe/bundle:36:in `<top (required)>'
     # ./vendor/bundle/bin/bundle:113:in `load'
     # ./vendor/bundle/bin/bundle:113:in `<main>'
     # ------------------
     # --- Caused by: ---
     # Excon::Error::NotFound:
     #   Expected([200, 201, 202, 204, 206, 304, 429]) <=> Actual(404 Not Found)
     #   ./vendor/bundle/ruby/3.1.0/gems/excon-0.99.0/lib/excon/middlewares/expects.rb:13:in `response_call'
Finished in 4 minutes 57.8 seconds (files took 0.41851 seconds to load)
12 examples, 1 failure
Failed examples:
rspec ./spec/hatchet/rails5_spec.rb:4 # Rails 5 behaves correctly, not related to Rails 5 at all
./spec/helpers/rails_runner_spec.rb[1:4]
./spec/helpers/rake_runner_spec.rb[1:1]
./spec/hatchet/ruby_spec.rb[1:1:3]
./spec/helpers/download_presence_spec.rb[1:7]
./spec/unit/bash_functions_spec.rb[1:2]
./spec/helpers/rake_runner_spec.rb[1:5]
./spec/helpers/ruby_version_spec.rb[1:7]
./spec/rake/deploy_check_spec.rb[1:1:1]
./spec/installers/rbx_installer_spec.rb[1:1:1]
./spec/installers/heroku_ruby_installer_spec.rb[1:1:2:1]
./spec/helpers/shell_spec.rb[1:3:1]
./spec/hatchet/rails5_spec.rb[1:1]
-----> test command `bundle exec rspec-queue --max-requeues=3 --timeout 180 --queue $REDIS_URL || { cat log/test_order.log;  $(exit 1); }` failed with exit status 1

Seems like a regression from #195 / #198?

cc @schneems
@edmorley
Copy link
Member Author

edmorley commented Mar 2, 2023

This is perma-failing for heroku-buildpack-ruby, eg:
https://dashboard.heroku.com/pipelines/ac057663-170b-4bdd-99d0-87560eb3a570/tests/1822

@edmorley edmorley mentioned this issue Mar 2, 2023
Closed
@edmorley edmorley linked a pull request Mar 13, 2023 that will close this issue
Only call teardown once per outer block #201
Merged
@edmorley
Copy link
Member Author

This was fixed by #201.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Only call teardown once per outer block heroku/hatchet
1 participant
@edmorley