Skip to content

HDDS-10573. [hsync] Add a freon tool to benchmark hsync/write concurrency #6925

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
jojochuang merged 5 commits into from
Jul 12, 2024
Merged

HDDS-10573. [hsync] Add a freon tool to benchmark hsync/write concurrency #6925

jojochuang merged 5 commits into from
Jul 12, 2024

Conversation

@duongkame
Copy link
Contributor

@duongkame duongkame commented Jul 11, 2024 •
edited
Loading

What changes were proposed in this pull request?

A simple Freon tool to generate concurrent write and hsync workload to one or multiple files.

Sample usage and output:

ozone freon hsync-generator -t 20 --bytes-per-write=1024 --path=ofs://ozone1/vol1/bucket1/p -n 500000 --number-of-files=2

             count = 500000
         mean rate = 2868.67 calls/second
     1-minute rate = 2969.64 calls/second
     5-minute rate = 2734.69 calls/second
    15-minute rate = 2644.31 calls/second
               min = 1.63 milliseconds
               max = 145.83 milliseconds
              mean = 6.21 milliseconds
            stddev = 8.77 milliseconds
            median = 4.07 milliseconds
              75% <= 6.22 milliseconds
              95% <= 12.60 milliseconds
              98% <= 37.96 milliseconds
              99% <= 53.12 milliseconds
            99.9% <= 111.50 milliseconds

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-10573

How was this patch tested?

Unit test.

jojochuang and others added 3 commits July 10, 2024 14:11
@jojochuang @duongkame
Add a freon test ms that creates a thread pool of syncer threads that…
eb84870 
… does hsync nonstop.

Change-Id: I27d72c4412fe47cf862691515dd0f5fc98623a33
@jojochuang @duongkame
Add to ozone freon CLI.
4e48367 
Change-Id: Ie44fa1518a171b70782035d0af0d79a455439954
@duongkame
HDDS-10573. [hsync] Add a freon tool to benchmark hsync/write concurr…
9a3fdae 
…ency
@duongkame duongkame requested a review from jojochuang July 11, 2024 01:23
@jojochuang jojochuang requested a review from smengcl July 11, 2024 07:31
jojochuang
jojochuang reviewed Jul 12, 2024
View reviewed changes
Copy link
Contributor

@jojochuang jojochuang left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Mostly looks good just a few small nits

jojochuang
jojochuang approved these changes Jul 12, 2024
View reviewed changes
Copy link
Contributor

@jojochuang jojochuang left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1 thanks @duongkame

ivandika3
ivandika3 reviewed Jul 12, 2024
View reviewed changes
Copy link
Contributor

@ivandika3 ivandika3 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@duongkame Thanks for the patch. Overall looks good. Left one comment.

hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/freon/HsyncGenerator.java Outdated
configuration = freon.createOzoneConfiguration();
}
URI uri = URI.create(rootPath);
fileSystem = FileSystem.get(uri, configuration);
Copy link
Contributor

@ivandika3 ivandika3 Jul 12, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you think we should incorporate thread-local instance of FileSystem to prevent bottleneck (as done in #1479)?

Note: I'm currently working to unify freon tests using the FileSystem instance https://issues.apache.org/jira/browse/HDDS-11081, hopefully a patch will be raised soon.

Copy link
Contributor Author

@duongkame duongkame Jul 12, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ivandika3 We unlikely need more than one FileSystem in this tool. In #1479, the HadoopFsGenerator creates many files, (thus a big load to OM to create file metadata).
This Freon tool creates one or a few files upfront.

Copy link
Contributor Author

@duongkame duongkame Jul 12, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Anyway, I separated FileSystem per outputstream.

@jojochuang jojochuang merged commit aa3c24e into apache:HDDS-7593 Jul 12, 2024
@jojochuang
Copy link
Contributor

jojochuang commented Jul 12, 2024

Merged. Thanks @smengcl @duongkame and @ivandika3

@jojochuang jojochuang mentioned this pull request Jul 12, 2024
Closed
@duongkame duongkame deleted the HDDS-10573 branch April 12, 2025 00:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@ivandika3 ivandika3 ivandika3 left review comments

@jojochuang jojochuang jojochuang approved these changes

@smengcl smengcl smengcl approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@duongkame @jojochuang @ivandika3 @smengcl