HDDS-13112. [Docs] OM Bootstrap can also happen when follower falls behind too much.

[jojochuang]

What changes were proposed in this pull request?

HDDS-13112. [Docs] Add a section "Automatic Snapshot Installation for Stale Ozone Managers" to OM HA doc

Please describe your PR in detail:

• Generated-by: ChatGPT, prompt:

https://ozone.apache.org/docs/edge/feature/om-ha.html

 

It is written in the user doc that OM bootstrap happens when adding a new OM. Ratis would trigger notifyInstallSnapshotFromLeader() if a follower OM falls behind the leader OM. We should update the doc to include this condition too.

https://github.com/apache/ratis/blob/c1da37cb455bbf94da267b3f2b9bf3884017e1ca/ratis-server-api/src/main/java/org/apache/ratis/server/leader/LogAppender.java#L109

/**
 * Should this {@link LogAppender} send a snapshot to the follower?
 *
 * @return the snapshot if it should install a snapshot; otherwise, return null.
 */
default SnapshotInfo shouldInstallSnapshot() {
  // we should install snapshot if the follower needs to catch up and:
  // 1. there is no local log entry but there is snapshot
  // 2. or the follower's next index is smaller than the log start index
  // 3. or the follower is bootstrapping and has not installed any snapshot yet 

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-13112

How was this patch tested?

User doc.

[adoroszlai]

Thanks @jojochuang for the patch. I find the content generated by ChatGPT too verbose. It keeps repeating the same information from prior paragraphs.

[jojochuang]

Thanks @adoroszlai accepted the suggestions.

[ivandika3]

@jojochuang Thanks for the documentation.

I think it is worth adding how to handle an issue that we encountered before that because of the a huge OM DB, while the follower's install the snapshot from leader (i.e. download the OM DB), the leader's Raft log for the particular snapshot index has already been purged. So after the OM follower finished downloading the OM DB, the leader cannot send the purged logs and request to follower to redownload the OM DB.

Currently, this can be handled by setting the configurations introduced in HDDS-8131. So either:

1. Set ozone.om.ratis.log.purge.preservation.log.num to a high enough value (e.g. 1000000) so that the OM leader will not the purge the last N logs.
2. Set ozone.om.ratis.log.purge.upto.snapshot.index to false, which causes OM leader logs to never be purged until all the follower already catch up.

You can refer to the ticket for a full explanation and tradeoffs.

[szetszwo]

@jojochuang , thanks for working on this! Please see the comments inlined.

[szetszwo]

... the latest snapshot from the leader.

Side note: we should support install snapshot from another follower in order to reduce the load of the leader.

[jojochuang]

good idea!

[jojochuang]

https://issues.apache.org/jira/browse/HDDS-13267

[jojochuang]

@ivandika3 thanks for the tips for huge OM DB. I think it's useful information.
I'd like to think it's an advanced topic and we should create an advanced/troubleshooting section for these kind of info.

Thoughts?

[ivandika3]

I'd like to think it's an advanced topic and we should create an advanced/troubleshooting section for these kind of info.

Yes, this can be updated in a separate ticket / as part of the website v2.

[ivandika3]

I think we can also a one-liner that this snapshot (Raft snapshot) and Ozone snapshot is related but different.

Again, this can be addressed in a separate ticket if needed.

[jojochuang]

https://issues.apache.org/jira/browse/HDDS-13268 for creating a troubleshooting user doc.

[szetszwo]

Thanks for the update!

+1 The change looks good.

[jojochuang]

Thanks @ivandika3 @szetszwo @adoroszlai @sadanand48 merged.

[adoroszlai]

@jojochuang So in the end we removed all changes made by Gemini, but it was still credited in the commit. However, @sadanand48, who contributed a completely new paragraph, was not. I don't think that's fair.