Timestamp-verified restart

[kford-oconnors]

A restart function that verifies the restart was successful. Convenient for build scripts. Based on this example.

  def verified_restart
    old_timestamp = go(%Q{http://#{@properties["ml.server"]}:8001/admin/v1/timestamp}, "get").body
    restart
    retry_count = 0
    retry_max = 5
    retry_interval= 10
    new_timestamp = old_timestamp
    while retry_count < retry_max do
      begin
        new_timestamp = go(%Q{http://#{@properties["ml.server"]}:8001/admin/v1/timestamp}, "get").body
      rescue
        logger.debug 'Retry attempt ' + retry_count.to_s + ' failed'
      end
      if new_timestamp != old_timestamp
        logger.info 'Verfied restart.'
        break
      end
      logger.debug "Verifying restart..."
      sleep retry_interval
      retry_count += 1
    end
    if new_timestamp == old_timestamp
      logger.warn "Could not verify restart"
    end
  end
end

[RobertSzkutak]

I am tempted to rename your function to restart and rename our restart to something else.

What do you think, @dmcassel and @grtjn ?

[dmcassel]

I like the idea of this functionality being the default restart behavior.

[dmcassel]

This is partially implemented in dev. The verified restart will work for the target host, but doesn't yet check that a cluster has fully restarted.

[grtjn]

We might need to check what restart actually does. It takes some params, so it might look for timestamps of bootstrap-host, but not actually restarting bootstrap-host.

Also wondering if it would make sense to check all hosts participating in a cluster for having completed restart before continuing. Would be more straight-forward if the (original) restart function would return a list of hosts that will restart, and then the verify routine would simply check them all (one after the other perhaps)..

[divino]

I already have a code for this in one of our project. What I did is to get the hosts names using “http://#{@properties["ml.server"]}:8002/manage/v2/hosts?format=json” and check each one if the timestamp changed after invoking restart.

[grtjn]

Roxy supports restarting a specific group (e.g. subset of all hosts) or entire cluster (e.g. all hosts in cluster). You'd need to distinguish which were involved in restart first. But I'd be interested in seeing what you did. Can you send that offline?