The problem occurs when broker URL in the Connection Factory bean is configured with failover transport, i.e.: failover:(nio://192.168.64.2:61617)?randomize=false&maxReconnectAttempts=3.
With that configuration in place connectionFactory.createConnection() doesn't throw an exception even when Broker cannot be reached.
As a workaround, I've created my own JMS health indicator that calls connection.start(); just before fetching information about JMS provider and it works as expected.
Spring boot version 1.3.7