jdbc input errors at start-up prevent pipeline termination/reloading

[jsvd]

Connectivity issues in during startup of jdbc input such as an invalid password are retried forever.
During these retries the status of stop? is never checked so the pipeline blocks reloadings.

The issue comes from the rufus scheduler by default catching exceptions in the jobs it executes. It's necessary to catch these exceptions and terminate the scheduler if the pipeline is terminating.

[kaisecheng]

A deadlock related to ReentrantLock is involved in the issue with invalid password setting.

Input plugin start scheduler to connect database with lock, however, open_jdbc_connection() fail and it is not in begin ensure. The plugin retried forever.
User updates central pipeline management with the correct password, the reload path in other thread require the same lock. This is also the case in shutdown path.

thread run
execute_statement [jdbc.rb:211] (LogStash::PluginMixins::Jdbc::Jdbc)
execute_query [jdbc.rb:335] (LogStash::Inputs::Jdbc)
run [jdbc.rb:298] (LogStash::Inputs::Jdbc)
do_call [jobs.rb:234] (Rufus::Scheduler::Job)
do_trigger [jobs.rb:258] (Rufus::Scheduler::Job)
start_work_thread [jobs.rb:300] (Rufus::Scheduler::Job)
start_work_thread [jobs.rb:299] (Rufus::Scheduler::Job)
start_work_thread [jobs.rb:289] (Rufus::Scheduler::Job)

thread reload (update Central pipeline managment)
close_jdbc_connection [jdbc.rb:196] (LogStash::PluginMixins::Jdbc::Jdbc)
stop [jdbc.rb:312] (LogStash::Inputs::Jdbc)
do_stop [base.rb:103] (LogStash::Inputs::Base)
stop_inputs [java_pipeline.rb:458] (LogStash::JavaPipeline)
shutdown [java_pipeline.rb:447] (LogStash::JavaPipeline)
execute [reload.rb:67] (LogStash::PipelineAction::Reload)
reload_pipeline [pipelines_registry.rb:213] (LogStash::PipelinesRegistry)
execute [reload.rb:62] (LogStash::PipelineAction::Reload)
converge_state [agent.rb:384] (LogStash::Agent)

thread stop (^C)
close_jdbc_connection [jdbc.rb:199] (LogStash::PluginMixins::Jdbc::Jdbc)
stop [jdbc.rb:312] (LogStash::Inputs::Jdbc)
do_stop [base.rb:103] (LogStash::Inputs::Base)
stop_inputs [java_pipeline.rb:458] (LogStash::JavaPipeline)
shutdown [java_pipeline.rb:447] (LogStash::JavaPipeline)
execute [stop.rb:30] (LogStash::PipelineAction::Stop)
terminate_pipeline [pipelines_registry.rb:186] (LogStash::PipelinesRegistry)
execute [stop.rb:29] (LogStash::PipelineAction::Stop)
converge_state [agent.rb:384] (LogStash::Agent)