Skip to content

HDDS-4461. Reuse compiled binaries in acceptance test #1588

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
elek merged 9 commits into from
Nov 30, 2020
Merged

HDDS-4461. Reuse compiled binaries in acceptance test #1588

elek merged 9 commits into from
Nov 30, 2020

Conversation

@adoroszlai
Copy link
Contributor

@adoroszlai adoroszlai commented Nov 13, 2020

What changes were proposed in this pull request?

Save Ozone binaries from compile check, and use these in acceptance and kubernetes checks, instead of each check performing its own full build. Total execution time is similar, but the dependent checks are started later, so we save GitHub Actions cycles.

https://issues.apache.org/jira/browse/HDDS-4461

How was this patch tested?

Tested in fork:

Artifact ozone-bin has been successfully uploaded!

https://github.com/adoroszlai/hadoop-ozone/runs/1395732068#step:8:52

Artifact ozone-bin was downloaded to /home/runner/work/hadoop-ozone/hadoop-ozone

https://github.com/adoroszlai/hadoop-ozone/runs/1395772736#step:4:11

Also verified coverage is the same (79%) as with the previous commit (760c1e8).

@adoroszlai adoroszlai self-assigned this Nov 13, 2020
@adoroszlai adoroszlai added build Pull request that modifies the build process enhancement New feature or request labels Nov 13, 2020
@adoroszlai adoroszlai requested a review from elek November 13, 2020 19:19
@elek
Copy link
Member

elek commented Nov 16, 2020

Great idea @adoroszlai, thanks the patch.

What do you think about using -Pdist instead of the custom tar execution (-Pdist supposed to create the tar file).

With this approach we will test exactly the same tar file which will be released...

@adoroszlai
Copy link
Contributor Author

adoroszlai commented Nov 16, 2020 •
edited
Loading

What do you think about using -Pdist instead of the custom tar execution (-Pdist supposed to create the tar file).

With this approach we will test exactly the same tar file which will be released...

@elek Testing part of the release process is a good idea, thanks. Updated the patch with this.

@elek
Copy link
Member

elek commented Nov 23, 2020

Thanks the update @adoroszlai

I am thinking about one thing, but I couldn't find any answer:

Isn't it a problem to store ~300M after EACH of the builds (including PR check). I couldn't find any official information about the available storage quota (there could be a limit: https://docs.github.com/en/free-pro-team@latest/github/setting-up-and-managing-billing-and-payments-on-github/viewing-your-github-actions-usage).

With this modification we not only share the artifact between jobs but also archive them for 90 days. Which can be hundred of gigabytes or even more.

But I am not sure if it's a problem or not.

What do you think?

@adoroszlai
Copy link
Contributor Author

adoroszlai commented Nov 23, 2020

With this modification we not only share the artifact between jobs but also archive them for 90 days.

Good point. Updated to keep it only for one day.

@elek
Copy link
Member

elek commented Nov 30, 2020

Good point. Updated to keep it only for one day.

TIL it's possible to define retention per upload. Thanks.

@elek elek merged commit 536a6e7 into apache:master Nov 30, 2020
@adoroszlai adoroszlai deleted the HDDS-4461 branch November 30, 2020 14:33
@adoroszlai
Copy link
Contributor Author

adoroszlai commented Nov 30, 2020

Thanks @elek for reviewing and committing it.

errose28 added a commit to errose28/ozone that referenced this pull request Dec 1, 2020
@errose28
Merge branch 'HDDS-3698-upgrade' into HDDS-4480
ced3350 
* HDDS-3698-upgrade:
  HDDS-4429. Create unit test for SimpleContainerDownloader. (apache#1551)
  HDDS-4461. Reuse compiled binaries in acceptance test (apache#1588)
  HDDS-4511: Avoiding StaleNodeHandler to take effect in TestDeleteWithSlowFollower. (apache#1625)
  HDDS-4510. SCM can avoid creating RetriableDatanodeEventWatcher for deletion command ACK (apache#1626)
  HDDS-3363. Intermittent failure in testContainerImportExport (apache#1618)
  HDDS-4370. Datanode deletion service can avoid storing deleted blocks. (apache#1620)
  HDDS-4512. Remove unused netty3 transitive dependency (apache#1627)
  HDDS-4481. With HA OM can send deletion blocks to SCM multiple times. (apache#1608)
  HDDS-4487. SCM can avoid using RETRIABLE_DATANODE_COMMAND for datanode deletion commands. (apache#1621)
  HDDS-4471. GrpcOutputStream length can overflow (apache#1617)
  HDDS-4308. Fix issue with quota update (apache#1489)
  HDDS-4392. [DOC] Add Recon architecture to docs (apache#1602)
  HDDS-4501. Reload OM State fail should terminate OM for any exceptions. (apache#1622)
  HDDS-4492. CLI flag --quota should default to 'spaceQuota' to preserve backward compatibility. (apache#1609)
  HDDS-3689. Add various profiles to MiniOzoneChaosCluster to run different modes. (apache#1420)
  HDDS-4497. Recon File Size Count task throws SQL Exception. (apache#1612)
errose28 added a commit to errose28/ozone that referenced this pull request Dec 1, 2020
@errose28
Merge branch 'HDDS-3698-upgrade' into HDDS-4463
5014ec9 
* HDDS-3698-upgrade:
  HDDS-4429. Create unit test for SimpleContainerDownloader. (apache#1551)
  HDDS-4461. Reuse compiled binaries in acceptance test (apache#1588)
  HDDS-4511: Avoiding StaleNodeHandler to take effect in TestDeleteWithSlowFollower. (apache#1625)
  HDDS-4510. SCM can avoid creating RetriableDatanodeEventWatcher for deletion command ACK (apache#1626)
  HDDS-3363. Intermittent failure in testContainerImportExport (apache#1618)
  HDDS-4370. Datanode deletion service can avoid storing deleted blocks. (apache#1620)
  HDDS-4512. Remove unused netty3 transitive dependency (apache#1627)
  HDDS-4481. With HA OM can send deletion blocks to SCM multiple times. (apache#1608)
  HDDS-4487. SCM can avoid using RETRIABLE_DATANODE_COMMAND for datanode deletion commands. (apache#1621)
  HDDS-4471. GrpcOutputStream length can overflow (apache#1617)
  HDDS-4308. Fix issue with quota update (apache#1489)
  HDDS-4392. [DOC] Add Recon architecture to docs (apache#1602)
  HDDS-4501. Reload OM State fail should terminate OM for any exceptions. (apache#1622)
  HDDS-4492. CLI flag --quota should default to 'spaceQuota' to preserve backward compatibility. (apache#1609)
  HDDS-3689. Add various profiles to MiniOzoneChaosCluster to run different modes. (apache#1420)
  HDDS-4497. Recon File Size Count task throws SQL Exception. (apache#1612)
errose28 added a commit to errose28/ozone that referenced this pull request Jan 5, 2021
@errose28
Merge branch 'master' into HDDS-4123
3dc26ad 
* master: (40 commits)
  HDDS-4473. Reduce number of sortDatanodes RPC calls (apache#1610)
  HDDS-4485. [DOC] add the authentication rules of the Ozone Ranger. (apache#1603)
  HDDS-4528. Upgrade slf4j to 1.7.30 (apache#1639)
  HDDS-4424. Update README with information how to report security issues (apache#1548)
  HDDS-4484. Use RaftServerImpl isLeader instead of periodic leader update logic in OM and isLeaderReady for read/write requests (apache#1638)
  HDDS-4429. Create unit test for SimpleContainerDownloader. (apache#1551)
  HDDS-4461. Reuse compiled binaries in acceptance test (apache#1588)
  HDDS-4511: Avoiding StaleNodeHandler to take effect in TestDeleteWithSlowFollower. (apache#1625)
  HDDS-4510. SCM can avoid creating RetriableDatanodeEventWatcher for deletion command ACK (apache#1626)
  HDDS-3363. Intermittent failure in testContainerImportExport (apache#1618)
  HDDS-4370. Datanode deletion service can avoid storing deleted blocks. (apache#1620)
  HDDS-4512. Remove unused netty3 transitive dependency (apache#1627)
  HDDS-4481. With HA OM can send deletion blocks to SCM multiple times. (apache#1608)
  HDDS-4487. SCM can avoid using RETRIABLE_DATANODE_COMMAND for datanode deletion commands. (apache#1621)
  HDDS-4471. GrpcOutputStream length can overflow (apache#1617)
  HDDS-4308. Fix issue with quota update (apache#1489)
  HDDS-4392. [DOC] Add Recon architecture to docs (apache#1602)
  HDDS-4501. Reload OM State fail should terminate OM for any exceptions. (apache#1622)
  HDDS-4492. CLI flag --quota should default to 'spaceQuota' to preserve backward compatibility. (apache#1609)
  HDDS-3689. Add various profiles to MiniOzoneChaosCluster to run different modes. (apache#1420)
  ...
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@elek elek elek approved these changes

Assignees

@adoroszlai adoroszlai

Labels

build Pull request that modifies the build process enhancement New feature or request

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@adoroszlai @elek