Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

RATIS-1524. Optional DataStreamManagement#startTransaction configuration #601

Open
wants to merge 2 commits into
base: master
Choose a base branch
from

Conversation

guohao-rosicky
Copy link
Contributor

What changes were proposed in this pull request?

Optional DataStreamManagement#startTransaction configuration

What is the link to the Apache JIRA

see:

https://issues.apache.org/jira/browse/RATIS-1524

https://issues.apache.org/jira/browse/RATIS-1513

@guohao-rosicky
Copy link
Contributor Author

@szetszwo What do you think of this change?

@guohao-rosicky
Copy link
Contributor Author

guohao-rosicky commented Feb 15, 2022

@szetszwo I want to discuss a question about ozone BCSID.

Does BCSID still make sense when you use stream to transfer data and put A Block into the stream process? Because raft transport is deprecated, playback of raft log is eliminated.

I can open a JIRA in Ozone and discuss this

@szetszwo
Copy link
Contributor

@guohao-rosicky , without BCSID and the Ratis log, the current Ozone design won't work for recovery. Why we need Ratis (or Raft) in Ozone? It is because Ratis provides a consistent view of the data among the servers. Without Ratis, the data in the servers may diverge. How could you tell which one to trust? And how to do data recovery?

@guohao-rosicky
Copy link
Contributor Author

Thank you @szetszwo , I have understood BCSID.

In one of my test versions of ozone, I generated raft logs by sending raft async RPC through DataStreamManagement#startTransaction to get the BCSID with the raft log index of ozone, Throughput is very small because raft needs to sort.

In another version of my test, if DataStreamManagement#startTransaction throughput was doubled by skipping it, could we get a BCSID in another way than raft log?

@szetszwo
Copy link
Contributor

... could we get a BCSID in another way than raft log?

We need a BCSID and also the ability to commit transactions. I guess there are no easy ways. Otherwise, we can use it to replace the Raft Consensus Algorithm in general.

@guohao-rosicky
Copy link
Contributor Author

guohao-rosicky commented Feb 17, 2022

... could we get a BCSID in another way than raft log?

We need a BCSID and also the ability to commit transactions. I guess there are no easy ways. Otherwise, we can use it to replace the Raft Consensus Algorithm in general.

package org.apache.ratis.io;

public enum StandardWriteOption implements WriteOption {
  /** Sync the data to the underlying storage. */
  SYNC,
  /** Close the data to the underlying storage. */
  CLOSE,

  /** Returns a unique ID **/
  UNIQUE_ID,
  
}

@szetszwo
Can we add a new WriteOption? When the primary node receives this request, it generates a unique ID of type long, synchronizes it to the other nodes, and returns it to the client

In other words, generating an ID on the Primary node and passing it to the other nodes as a stream can improve throughput without raft requests internally.

@szetszwo
Copy link
Contributor

..., generating an ID on the Primary node ...

How to make sure the ID is unique? All the nodes could be the Primary node at some point of time.

@guohao-rosicky
Copy link
Contributor Author

guohao-rosicky commented Feb 17, 2022

I can open a JIRA in Ratis and discuss this @szetszwo @captainzmc

https://issues.apache.org/jira/browse/RATIS-1513

@szetszwo
Copy link
Contributor

https://issues.apache.org/jira/browse/RATIS-1513

@guohao-rosicky, @captainzmc, I really hope that we could fix "TimeoutIOException: Timeout 3000ms". How about we fix it first?

@guohao-rosicky
Copy link
Contributor Author

guohao-rosicky commented Feb 17, 2022

https://issues.apache.org/jira/browse/RATIS-1513

@guohao-rosicky, @captainzmc, I really hope that we could fix "TimeoutIOException: Timeout 3000ms". How about we fix it first?

We changed this configuration to fix this problem because DataStreamManagement#startTransaction was taking too long.

We have a test report showing that DataStreamManagement#startTransaction caused "TimeoutIOException: Timeout 3000ms".

@szetszwo I was hoping you could help us come up with a solution.
The solution I can think of so far is to skip DataStreamManagement#startTransaction that triggers the timeout

@guohao-rosicky
Copy link
Contributor Author

@szetszwo https://docs.google.com/document/d/1mS3GqovQ3D1b7V0L3--VF9xhl5jdId1mSL0cQNb7uHo/edit

This is the process of testing reports and locating problems

@szetszwo
Copy link
Contributor

We changed this configuration to fix this problem because DataStreamManagement#startTransaction was taking too long.

This is not really a fix since it changes the functionality.

commit f349fdf2488e254d0fa2eeb17bbaf44de2b0932d
Author: hao guo <[email protected]>
Date:   Wed Dec 15 10:29:13 2021 +0800

    RATIS-1438. Add request timeout to ratis Streaming (#563)

"TimeoutIOException: Timeout 3000ms" probably started happening after RATIS-1438. How about we increase the timeout value, say to 10 seconds?

@guohao-rosicky
Copy link
Contributor Author

We changed this configuration to fix this problem because DataStreamManagement#startTransaction was taking too long.

This is not really a fix since it changes the functionality.

commit f349fdf2488e254d0fa2eeb17bbaf44de2b0932d
Author: hao guo <[email protected]>
Date:   Wed Dec 15 10:29:13 2021 +0800

    RATIS-1438. Add request timeout to ratis Streaming (#563)

"TimeoutIOException: Timeout 3000ms" probably started happening after RATIS-1438. How about we increase the timeout value, say to 10 seconds?

DataStreamManagement#startTransaction

It can be changed to 10 seconds and I will submit a new PR for this

@guohao-rosicky
Copy link
Contributor Author

guohao-rosicky commented Feb 17, 2022

@szetszwo
This is the process of testing reports and locating problems
https://docs.google.com/document/d/1mS3GqovQ3D1b7V0L3--VF9xhl5jdId1mSL0cQNb7uHo/edit

Can you take a look at our test report and consider further optimizing the performance of ratis stream over ozone based on the problems identified in the test report.

Discuss how to optimize the scheme.

@captainzmc and I can participate in the development of optimization

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants