Allow multiple concurrent one-off dynos to run

[schneems]

By default free apps can only run one "one-off" dyno at a time. For example if you run heroku run rails c you cannot run heroku run ls until the first command finishes (on a free app). There's a quirk about the distributed nature of the heroku platform in that once heroku run <command> returns, it does not guarantee that the dyno has completely been shutdown, but only that shutdown has started. This means that if you're wanting to do multiple app.run calls in a test, you need to add sleeps to be safe:

Hatchet::Runner.new("default_ruby").deploy do |app|
  expect(app.run("ls -a Gemfile 'foo bar #baz'")).to match(/ls: cannot access 'foo bar #baz': No such file or directory\s+Gemfile/)
  sleep 4
  expect((0 != $?.exitstatus)).to be_truthy
  app.run("ls erpderp", heroku: ({ "exit-code" => (Hatchet::App::SkipDefaultOption) }))
  sleep 4
  expect((0 == $?.exitstatus)).to be_truthy
  app.run("ls erpderp", heroku: ({ "no-tty" => nil }))
end

Which is not great.

Enable multiple one-off dynos

This PR allows an individual application to flag itself into being able to run multiple concurrent dynos by setting the run_multi: true config at initialization time. However, to use this setting you must set the env var HEROKU_EXPENSIVE_MODE=1 as a safety gate to ensure you understand the risks.

Now the above tests will look like this (without the sleep):

Hatchet::Runner.new("default_ruby", run_multi: true).deploy do |app|
  expect(app.run("ls -a Gemfile 'foo bar #baz'")).to match(/ls: cannot access 'foo bar #baz': No such file or directory\s+Gemfile/)
  expect((0 != $?.exitstatus)).to be_truthy

  app.run("ls erpderp", heroku: ({ "exit-code" => (Hatchet::App::SkipDefaultOption) }))
  expect((0 == $?.exitstatus)).to be_truthy

  app.run("ls erpderp", heroku: ({ "no-tty" => nil }))
end

NEAT!

Run multi feature

The REALLY cool feature this opens up is that you no longer have to wait for each of your heroku run commands to return. They can now be executed concurrently using the new run_multi command:

Hatchet::Runner.new("default_ruby", run_multi: true).deploy do |app|
  app.run_multi("ls") do |out, status|
    expect(status.success?).to be_truthy
    expect(out).to include("Gemfile")
  end
  app.run_multi("ruby -v") do |out, status|
    expect(status.success?).to be_truthy
    expect(out).to include("ruby")
  end
end

In this example the heroku run ls and heroku run ruby -v will be executed concurrently. The order that the run_multi blocks execute is not guaranteed.

You can toggle this run_multi setting on globally by using HATCHET_RUN_MULTI=1. Without this setting enabled, you might need to add a sleep between multiple app.run invocations. WARNING: Enabling this run_multi setting will charge your application account. To work, this requires your application to have a web process associated with it.

[schneems]

Related issue: grosser/parallel_split_test#7 (comment)