After recovering, a primary should update its knowledge of its own local checkpoint

[jasontedor]

A primary shard should update its knowledge of its own local checkpoint so that the global checkpoint does not otherwise know the local checkpoint of the shard until an indexing operation.

Relates #10708, relates #25355

[droberts195]

Given that #25467 was closed as a duplicate of this, I think this test failure is probably due to this issue too: https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+periodic/3369/console

The log contains:

  1> [2017-06-30T13:44:13,802][WARN ][o.e.i.c.IndicesClusterStateService] [node_s0] [[test][0]] marking and sending shard failed due to [failed recovery]
  1> org.elasticsearch.indices.recovery.RecoveryFailedException: [test][0]: Recovery failed from {node_s1}{vZ1YOynsRFyd8glTybKDyQ}{41U494MHT5-rtjThD9JsHg}{127.0.0.1}{127.0.0.1:9601} into {node_s0}{44s7-YEXR22bpT5ZXVfzdw}{GUjuSoRfSH6cDgL05tevTw}{127.0.0.1}{127.0.0.1:9600} (failed to retry recovery)
  1> 	at org.elasticsearch.indices.recovery.RecoveriesCollection.resetRecovery(RecoveriesCollection.java:122) ~[main/:?]
  1> 	at org.elasticsearch.indices.recovery.PeerRecoveryTargetService.retryRecovery(PeerRecoveryTargetService.java:153) ~[main/:?]
  1> 	at org.elasticsearch.indices.recovery.PeerRecoveryTargetService.retryRecovery(PeerRecoveryTargetService.java:149) ~[main/:?]
  1> 	at org.elasticsearch.indices.recovery.PeerRecoveryTargetService.doRecovery(PeerRecoveryTargetService.java:256) ~[main/:?]
  1> 	at org.elasticsearch.indices.recovery.PeerRecoveryTargetService.access$900(PeerRecoveryTargetService.java:75) ~[main/:?]
  1> 	at org.elasticsearch.indices.recovery.PeerRecoveryTargetService$RecoveryRunner.doRun(PeerRecoveryTargetService.java:616) ~[main/:?]
  1> 	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:638) ~[main/:?]
  1> 	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[main/:?]
  1> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[?:1.8.0_131]
  1> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[?:1.8.0_131]
  1> 	at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_131]
  1> Caused by: java.lang.IllegalStateException: cannot reset recovery as previous attempt made it past finalization step
  1> 	at org.elasticsearch.indices.recovery.RecoveryTarget.resetRecovery(RecoveryTarget.java:216) ~[main/:?]
  1> 	at org.elasticsearch.indices.recovery.RecoveriesCollection.resetRecovery(RecoveriesCollection.java:108) ~[main/:?]

and:

  2> WARNING: Uncaught exception in thread: Thread[elasticsearch[node_s0][generic][T#1],5,TGRP-OpenCloseIndexIT]
  2> java.lang.AssertionError: local checkpoint [-2] for allocation ID [aCk3bTylTs2tN5utGruRzw] violates being at least the global checkpoint [21]
  2> 	at __randomizedtesting.SeedInfo.seed([4ECF08D198E60282]:0)
  2> 	at org.elasticsearch.index.seqno.GlobalCheckpointTracker.updateAllocationIdsFromPrimaryContext(GlobalCheckpointTracker.java:411)
  2> 	at org.elasticsearch.index.seqno.SequenceNumbersService.updateAllocationIdsFromPrimaryContext(SequenceNumbersService.java:186)
  2> 	at org.elasticsearch.index.shard.IndexShard.updateAllocationIdsFromPrimaryContext(IndexShard.java:1707)
  2> 	at org.elasticsearch.indices.recovery.RecoveryTarget.handoffPrimaryContext(RecoveryTarget.java:383)
  2> 	at org.elasticsearch.indices.recovery.PeerRecoveryTargetService$HandoffPrimaryContextRequestHandler.messageReceived(PeerRecoveryTargetService.java:425)
  2> 	at org.elasticsearch.indices.recovery.PeerRecoveryTargetService$HandoffPrimaryContextRequestHandler.messageReceived(PeerRecoveryTargetService.java:420)
  2> 	at org.elasticsearch.transport.TransportRequestHandler.messageReceived(TransportRequestHandler.java:30)
  2> 	at org.elasticsearch.transport.AssertingTransportInterceptor$1.messageReceived(AssertingTransportInterceptor.java:76)
  2> 	at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:66)
  2> 	at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1525)
  2> 	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:638)
  2> 	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
  2> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
  2> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
  2> 	at java.lang.Thread.run(Thread.java:748)

[jasontedor]

Closed by #25468