Skip to content

HDDS-8921. Add support for EC in Freon SCM block generator #4982

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
sodonnel merged 1 commit into from
Jun 26, 2023
Merged

HDDS-8921. Add support for EC in Freon SCM block generator #4982

sodonnel merged 1 commit into from
Jun 26, 2023

Conversation

@adoroszlai
Copy link
Contributor

@adoroszlai adoroszlai commented Jun 26, 2023

What changes were proposed in this pull request?

Add replication-related options to the ozone freon stb SCM benchmark.

https://issues.apache.org/jira/browse/HDDS-8921

How was this patch tested?

Replaced config in ozone environment:

OZONE-SITE.XML_ozone.scm.stale.node.interval=1d
OZONE-SITE.XML_ozone.scm.dead.node.interval=2d
OZONE-SITE.XML_hdds.heartbeat.interval=1h
OZONE-SITE.XML_ozone.scm.heartbeat.thread.interval=1h

$ OZONE_DATANODES=5 ./run.sh -d
...

$ docker-compose exec -T s3g ozone freon stb --scm-host scm --benchmark AllocateBlocks --num-threads 1 --num-blocks 1 --type EC --replication rs-3-2-1024k
...
Total allocated blocks: 1
Total failed blocks: 0
Execution Time: 00:00:05,000
Throughput: 0.200000 (ops)

$ docker-compose exec -T s3g ozone freon stb --scm-host scm --benchmark AllocateBlocks --num-threads 8 --num-blocks 1024 --type EC --replication rs-3-2-1024k
...
Total allocated blocks: 1024
Total failed blocks: 0
Execution Time: 00:00:05,006
Throughput: 204.800003 (ops)

@adoroszlai adoroszlai self-assigned this Jun 26, 2023
@adoroszlai adoroszlai added the EC label Jun 26, 2023
@adoroszlai adoroszlai requested a review from sodonnel June 26, 2023 13:19
sodonnel
sodonnel approved these changes Jun 26, 2023
View reviewed changes
Copy link
Contributor

@sodonnel sodonnel left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

@sodonnel sodonnel merged commit 7797b46 into apache:master Jun 26, 2023
@adoroszlai adoroszlai deleted the HDDS-8921-simple branch June 26, 2023 17:04
@adoroszlai
Copy link
Contributor Author

adoroszlai commented Jun 26, 2023

Thanks @sodonnel for reviewing and merging this.

errose28 added a commit to errose28/ozone that referenced this pull request Jun 26, 2023
@errose28
Merge branch 'master' into HDDS-7100-scanner-shutdown
7726336 
* master: (79 commits)
  HDDS-8914. Datanode may fail to start due to duplicate VolumeInfoMetrics (apache#4966)
  HDDS-8921. Add support for EC in Freon SCM block generator (apache#4982)
  HDDS-8927. Metadata scanner should not scan unhealthy containers. (apache#4976)
  HDDS-8929. Avoid list allocation for pipeline search (apache#4980)
  HDDS-8778. Support recursive volume delete using Ozone sh command. (apache#4842)
  HDDS-8885. Quota repair count enable quota feature for old bucket/volume. (apache#4941)
  HDDS-8771. Refactor volume level tmp directory for generic usage. (apache#4838)
  HDDS-8922. Random EC read pipeline ID causes XceiverClient cache churn (apache#4971)
  HDDS-8586 Recon. - API for Count of deletePending keys and amount of data mapped to such keys. (apache#4923)
  HDDS-8908. Intermittent failure in TestBlockDeletion#testBlockDeletion (apache#4958)
  HDDS-8910. Replace LockManager with striped lock in ContainerStateManager (apache#4962)
  HDDS-8917. Move protobuf conversion out of the lock in PipelineStateManagerImpl (apache#4965)
  HDDS-8825. Use apache/hadoop 3.3.5 docker image (apache#4963)
  HDDS-8906. Avoid stream when getting in-service healthy nodes (apache#4960)
  HDDS-8907. Store volume count when storage report is updated (apache#4957)
  HDDS-8905. PipelineManager metrics should not be synchronized (apache#4959)
  HDDS-8553. Improve scanner integration tests. (apache#4936)
  HDDS-8854. Avoid unnecessary DatanodeDetails creation for NodeStateManager lookup (apache#4925)
  HDDS-8315. [Snapshot] Added unit tests for SnapshotDiffManager (apache#4716)
  HDDS-7968. [Snapshot] Improve KeyDeletingService to reclaim eligible key blocks in snapshot's deletedTable (apache#4935)
  ...
errose28 added a commit to errose28/ozone that referenced this pull request Jun 26, 2023
@errose28
Merge branch 'master' into improve-volume-scanner
d27093b 
* master:
  HDDS-8914. Datanode may fail to start due to duplicate VolumeInfoMetrics (apache#4966)
  HDDS-8921. Add support for EC in Freon SCM block generator (apache#4982)
  HDDS-8927. Metadata scanner should not scan unhealthy containers. (apache#4976)
  HDDS-8929. Avoid list allocation for pipeline search (apache#4980)
  HDDS-8778. Support recursive volume delete using Ozone sh command. (apache#4842)
  HDDS-8885. Quota repair count enable quota feature for old bucket/volume. (apache#4941)
  HDDS-8771. Refactor volume level tmp directory for generic usage. (apache#4838)
  HDDS-8922. Random EC read pipeline ID causes XceiverClient cache churn (apache#4971)
vtutrinov pushed a commit to Cyrill/ozone that referenced this pull request Jul 3, 2023
@adoroszlai
HDDS-8921. Add support for EC in Freon SCM block generator (apache#4982)
b0ab1d3
vtutrinov pushed a commit to Cyrill/ozone that referenced this pull request Jul 3, 2023
@adoroszlai
HDDS-8921. Add support for EC in Freon SCM block generator (apache#4982)
4913d22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@sodonnel sodonnel sodonnel approved these changes

Assignees

@adoroszlai adoroszlai

Labels

EC

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@adoroszlai @sodonnel