Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

HDDS-11994. Convert Freon to pluggable model #7620

Merged
merged 2 commits into from
Jan 1, 2025

Conversation

adoroszlai
Copy link
Contributor

What changes were proposed in this pull request?

ozone freon has several unrelated subcommands, which currently are enumareted in the class Freon. This PR changes subcommands to register with the parent command, instead of the other way around.

https://issues.apache.org/jira/browse/HDDS-11994

How was this patch tested?

Verified that the number of freon subcommands is the same as previously:

$ bin/ozone freon 2>&1 | grep '^  [^ -]' | wc -l
36

Output when run without specific subcommand:

$ bin/ozone freon
...
Incomplete command
Usage: ozone freon [-hV] [--server] [--verbose] [-conf=<configurationPath>]
                   [-D=<String=String>]... [COMMAND]
Load generator and tester tool for ozone
      -conf=<configurationPath>

  -D, --set=<String=String>

  -h, --help      Show this help message and exit.
      --server    Enable internal http server to provide metric and profile
                    endpoint
  -V, --version   Print version information and exit.
      --verbose   More verbose output. Show the stack trace of the errors.
Commands:
  cgdn                                      Offline container metadata
                                              generator for Ozone Datanodes.
  cgom                                      Offline container metadata
                                              generator for Ozone Manager
  cgscm                                     Offline container metadata
                                              generator for Storage Conainer
                                              Manager
  cmdw, chunk-manager-disk-write            Write chunks as fast as possible.
  cr, container-replicator                  Replicate / download closed
                                              containers.
  dbp, datanode-block-putter                Issues putBlock commands to a Ratis
                                              pipeline.  The blocks are
                                              associated with a list of fake
                                              chunks, which do not really exist.
  dcg, datanode-chunk-generator             Create as many chunks as possible
                                              with pure XCeiverClient.
  dcv, datanode-chunk-validator             Validate generated Chunks are the
                                              same
  ddsg, dfs-directory-generator             Create nested directories to the
                                              any dfs compatible file system.
  dfsg, dfs-file-generator                  Create random files to the any dfs
                                              compatible file system.
  dfsv, dfs-file-validator                  Validate if the generated files
                                              have the same hash.
  dn-echo, dne                              Generate echo RPC request to
                                              DataNode
  dtsg, dfs-tree-generator                  Create nested directories and
                                              create given number of files in
                                              each dir in any dfs compatible
                                              file system.
  falg, follower-append-log-generator       Generate append log entries to a
                                              follower server
  hg, hsync-generator                       Generate writes and hsync traffic
                                              on one or multiple files.
  lalg, leader-append-log-generator         Generate append log entries to a
                                              leader server
  obrwf, om-bucket-read-write-file-ops      Creates files, performs respective
                                              read/write operations to measure
                                              lock performance.
  obrwk, om-bucket-read-write-key-ops       Creates keys, performs respective
                                              read/write operations to measure
                                              lock performance.
  occ, ozone-client-creator                 Create and close Ozone clients
                                              without doing anything useful
  ockg, ozone-client-key-generator          Generate keys with the help of the
                                              ozone clients.
  ockr, ozone-client-key-remover            Remove keys with the help of the
                                              ozone clients.
  ockrw, ozone-client-key-read-write-list-ops
                                            Generate keys with a fixed name and
                                              ranges that can be written, read
                                              and listed as sub-ranges from
                                              multiple clients.
  ockv, ozone-client-key-validator          Validate keys with the help of the
                                              ozone clients.
  ocokr, ozone-client-one-key-reader        Read the same key from multiple
                                              threads.
  om-echo, ome                              Generate echo RPC request to the OM
                                              with or without payload. Max
                                              payload size is 2097151 KB
  ombg, om-bucket-generator                 Generate ozone buckets on OM side.
  ombr, om-bucket-remover                   Remove ozone buckets on OM side.
  omkg, om-key-generator                    Create keys to the om metadata
                                              table.
  ommg, om-metadata-generator               Create metadata operation to the OM.
  ork                                       write range keys with the help of
                                              the ozone clients.
  randomkeys, rk                            Generate volumes/buckets and put
                                              generated keys.
  s3bg, s3-bucket-generator                 Create buckets via the s3 interface.
                                            Command requires AWS authentication
                                              environment variables set,
                                              AWS_ACCESS_KEY_ID and
                                              AWS_SECRET_ACCESS_KEY
  s3kg, s3-key-generator                    Create random keys via the s3
                                              interface.Command requires AWS
                                              authentication environment
                                              variables set, AWS_ACCESS_KEY_ID
                                              and AWS_SECRET_ACCESS_KEY
  scm-throughput-benchmark, stb             Benchmark for scm throughput.
  simulate-datanode                         Simulate one or many datanodes and
                                              register them to SCM.This is used
                                              to stress test SCM handling a
                                              massive cluster.
  strmg, streaming-generator                Create directory structure and
                                              stream them multiple times.

CI:
https://github.com/adoroszlai/ozone/actions/runs/12506264423

@adoroszlai adoroszlai added the tools Tools that helps with debugging label Dec 26, 2024
@adoroszlai adoroszlai self-assigned this Dec 26, 2024
Copy link
Contributor

@jojochuang jojochuang left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks good.

@SuppressWarnings("java:S2245") // no need for secure random
public final class RandomKeyGenerator implements Callable<Void> {
public final class RandomKeyGenerator implements Callable<Void>, FreonSubcommand {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like to me that RandomKeyGenerator should be rewritten by extending BaseFreonGenerator.
Of course that's a much larger rewrite that's out of the scope here.

@SuppressWarnings("java:S2245") // no need for secure random
public final class SCMThroughputBenchmark implements Callable<Void> {
public final class SCMThroughputBenchmark implements Callable<Void>, FreonSubcommand {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah I see. SCMThroughputBenchmark doesn't follow the typical freon class model.

@adoroszlai adoroszlai merged commit f2e6d38 into apache:master Jan 1, 2025
42 checks passed
@adoroszlai adoroszlai deleted the HDDS-11994 branch January 1, 2025 10:17
@adoroszlai
Copy link
Contributor Author

Thanks @jojochuang for the review.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
tools Tools that helps with debugging
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants