HDDS-2115 Add acceptance test for createPipeline CLI and datanode list CLI #375

timmylicheng · 2019-12-18T15:31:09Z

What changes were proposed in this pull request?

#HDDS-2115 Add acceptance test for createPipeline CLI and datanode list CLI

(Please fill in changes proposed in this fix)

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-2115

(Please create an issue in ASF JIRA before opening a pull request,
and you need to set the title of the pull request which starts with
the corresponding JIRA issue number. (e.g. HDDS-XXXX. Fix a typo in YYY.)

Please replace this section with the link to the Apache JIRA)

How was this patch tested?

Run test-single.sh locally and all passed.

(Please explain how this patch was tested. Ex: unit tests, manual tests)
(If this patch involves UI changes, please attach a screen-shot; otherwise, remove this)

timmylicheng · 2019-12-19T06:52:56Z

../test-single.sh scm scmcli/datanode.robot
==============================================================================
ozone-datanode :: Smoketest ozone cluster startup
==============================================================================
Run list pipeline                                                     | PASS |
------------------------------------------------------------------------------
ozone-datanode :: Smoketest ozone cluster startup                     | PASS |
1 critical test, 1 passed, 0 failed
1 test total, 1 passed, 0 failed
==============================================================================
Output:  /tmp/smoketest/ozone/result/robot-ozone-ozone-datanode-scm.xml

timmylicheng · 2019-12-19T06:53:36Z

../test-single.sh scm scmcli/pipeline.robot
==============================================================================
ozone-pipeline :: Smoketest ozone cluster startup
==============================================================================
Run list pipeline                                                     | PASS |
------------------------------------------------------------------------------
Run create pipeline                                                   | PASS |
------------------------------------------------------------------------------
ozone-pipeline :: Smoketest ozone cluster startup                     | PASS |
2 critical tests, 2 passed, 0 failed
2 tests total, 2 passed, 0 failed
==============================================================================
Output:  /tmp/smoketest/ozone/result/robot-ozone-ozone-pipeline-scm.xml
Robot framework is not installed, the reports can be generated (sudo pip install robotframework).

adoroszlai

Thanks @timmylicheng for working on this. LGTM, except two typos. Please also consider two suggestions.

hadoop-ozone/dist/src/main/smoketest/scmcli/datanode.robot

...ds/tools/src/main/java/org/apache/hadoop/hdds/scm/cli/pipeline/CreatePipelineSubcommand.java

hadoop-ozone/dist/src/main/smoketest/scmcli/pipeline.robot

timmylicheng · 2019-12-19T11:42:09Z

ERROR: Test execution of /home/runner/work/hadoop-ozone/hadoop-ozone/hadoop-ozone/dist/target/ozone-0.5.0-SNAPSHOT/compose/ozonesecure is FAILED!!!!

Failed acceptance test is irrelevant.

adoroszlai

Thanks @timmylicheng for updating the patch.

hanishakoneru · 2019-12-21T00:10:47Z

Thanks @timmylicheng for working on this.
The patch LGTM overall. The acceptance test failure you are seeing is because of another issue which was fixed by HDDS-2774. I have rerun the checks again.

hanishakoneru · 2019-12-21T00:19:25Z

@timmylicheng can you please rebase with latest master. We wouldn't get a clean acceptance test run without the fix in HDDS-2774.

timmylicheng · 2019-12-23T02:17:50Z

@timmylicheng can you please rebase with latest master. We wouldn't get a clean acceptance test run without the fix in HDDS-2774.

Yea sure. We are going merge tip master to HDDS-1564.

timmylicheng · 2019-12-23T10:59:40Z

After merge with master, acceptance fails at s3 gateway test. @hanishakoneru

elek · 2020-01-06T14:56:38Z

After merge with master, acceptance fails at s3 gateway test.

Can't see the exact error message. The acceptance test log files are missing from the archive.

Let me try to retrigger it....

hanishakoneru · 2020-01-06T22:32:12Z

I am not able to expand the acceptance test logs. Going to re-trigger the checks again to see if this is a one time problem or persistent.

timmylicheng · 2020-01-07T07:27:12Z

Removing network ozonesecure_default
cp: cannot stat '/home/runner/work/hadoop-ozone/hadoop-ozone/hadoop-ozone/dist/target/ozone-0.5.0-SNAPSHOT/compose/ozonesecure/result/robot-.xml': No such file or directory
[ ERROR ] Reading XML source '/home/runner/work/hadoop-ozone/hadoop-ozone/hadoop-ozone/dist/target/ozone-0.5.0-SNAPSHOT/compose/result/robot-.xml' failed: No such file or directory

Try --help for usage information.
cp: cannot stat '/home/runner/work/hadoop-ozone/hadoop-ozone/hadoop-ozone/dev-support/checks/../../../target/acceptance/log.html': No such file or directory
##[error]Process completed with exit code 1.

@hanishakoneru @elek

adoroszlai · 2020-01-07T11:32:24Z

@timmylicheng all acceptance tests failed because SCM could not exit safe mode:

2020-01-07T03:46:31.5337614Z SCM is in safe mode.
2020-01-07T03:46:41.4364121Z SCM is in safe mode.
2020-01-07T03:46:50.4562615Z SCM is in safe mode.
2020-01-07T03:47:00.1458169Z SCM is in safe mode.
2020-01-07T03:47:08.0186479Z SCM is in safe mode.
2020-01-07T03:47:17.1032118Z SCM is in safe mode.
2020-01-07T03:47:26.6599830Z SCM is in safe mode.
2020-01-07T03:47:28.6637806Z WARNING! Safemode is still on. Please check the docker-compose files

Docker container logs are full of messages like group-9415212EE472 not found. My guess is that this is the same problem as fixed in HDDS-2679 and RATIS-783. Can you please try merging from master again?

hanishakoneru · 2020-01-07T17:26:59Z

Thanks @adoroszlai. I am running CI with @timmylicheng 's changes on top of current master to see if rebasing to master would resolve the issue. I ll post the results when it completes.

…t CLI.

timmylicheng · 2020-01-08T07:02:12Z

I merged with master and acceptance failed again with a different message. @hanishakoneru @adoroszlai

hadoop27-mapreduce :: Execute MR jobs

Output: /tmp/smoketest/hadoop27/result/robot-hadoop27-hadoop27-mapreduce-rm.xml
/home/runner/work/hadoop-ozone/hadoop-ozone/hadoop-ozone/dist/target/ozone-0.5.0-SNAPSHOT/compose/ozone-mr/hadoop27/../../testlib.sh: line 125: 97441 Killed docker-compose -f "$COMPOSE_FILE" --no-ansi logs > "$RESULT_DIR/docker-$OUTPUT_NAME.log"
ERROR: Test execution of /home/runner/work/hadoop-ozone/hadoop-ozone/hadoop-ozone/dist/target/ozone-0.5.0-SNAPSHOT/compose/ozone-mr/hadoop27 is FAILED!!!!
##[error]Process completed with exit code 137.

elek · 2020-01-08T16:32:14Z

Acceptance tests on HDDS-1564 branch seems to be unstable

https://github.com/apache/hadoop-ozone/actions?query=branch%3AHDDS-1564

compared to the master:

https://github.com/apache/hadoop-ozone/actions?query=branch%3Amaster

I think it's a generic problem with the branch as I have seen same failures on different PRs (eg. #406).

I would suggest to merge this PR as is. It wouldn't break the master (feature branch) and the failure is clearly not the problem with this PR.

And after committing the issues in the queue, we need to find the problem and fix the tests on this branch...

hanishakoneru · 2020-01-08T18:57:15Z

Thanks @elek . I also tested the patch extensively. Most times the acceptance test fails without any logs. Tested the robot tests locally and they work fine. Also tested with just this change on top of master and the results are as expected.
I will merge this PR shortly.

hanishakoneru · 2020-01-08T21:17:00Z

Thank you @timmylicheng for the contributions, @adoroszlai and @elek for the reviews.
+1 to merge into branch HDDS-1564.

…t CLI (apache#375) * HDDS-2115 Add acceptance test for createPipeline CLI and datanode list CLI.

* HDDS-1577. Add default pipeline placement policy implementation. (#1366) (cherry picked from commit b640a5f6d53830aee4b9c2a7d17bf57c987962cd) * HDDS-1571. Create an interface for pipeline placement policy to support network topologies. (#1395) (cherry picked from commit 753fc6703a39154ed6013e44dbae572391748906) * HDDS-2089: Add createPipeline CLI. (#1418) (cherry picked from commit 326b5acd4a63fe46821919322867f5daff30750c) * HDDS-1569 Support creating multiple pipelines with same datanode. Contributed by Li Cheng. This closes #28 * HDDS-1572 Implement a Pipeline scrubber to clean up non-OPEN pipeline. (#237) * Rebase Fix * HDDS-2650 Fix createPipeline CLI. (#340) * HDDS-2035 Implement datanode level CLI to reveal pipeline relation. (#348) * Revert "HDDS-2650 Fix createPipeline CLI. (#340)" This reverts commit 7c71710. * HDDS-2650 Fix createPipeline CLI and make it message based. (#370) * HDDS-1574 Average out pipeline allocation on datanodes and add metrcs/test (#291) * Resolve rebase conflict. * HDDS-2756. Handle pipeline creation failure in different way when it exceeds pipeline limit Closes #401 * HDDS-2115 Add acceptance test for createPipeline CLI and datanode list CLI (#375) * HDDS-2115 Add acceptance test for createPipeline CLI and datanode list CLI. * HDDS-2772 Better management for pipeline creation limitation. (#410) * HDDS-2913 Update config names and CLI for multi-raft feature. (#462) * HDDS-2924. Fix Pipeline#nodeIdsHash collision issue. (#478) * HDDS-2923 Add fall-back protection for rack awareness in pipeline creation. (#516) * HDDS-3007 Fix CI test failure for TestSCMNodeManager. (#550) Co-authored-by: Sammi Chen <[email protected]> Co-authored-by: Xiaoyu Yao <[email protected]>

timmylicheng force-pushed the HDDS-2115 branch from dfa21af to 083c5d4 Compare December 19, 2019 02:27

adoroszlai reviewed Dec 19, 2019

View reviewed changes

timmylicheng force-pushed the HDDS-2115 branch from 083c5d4 to bdf6078 Compare December 19, 2019 09:54

adoroszlai approved these changes Dec 19, 2019

View reviewed changes

ChenSammi force-pushed the HDDS-1564 branch from 40da915 to 3e6f02d Compare December 23, 2019 08:08

timmylicheng force-pushed the HDDS-2115 branch 2 times, most recently from 1e7f167 to 8b0785d Compare December 23, 2019 08:16

ChenSammi force-pushed the HDDS-1564 branch from 3e6f02d to 53afa84 Compare December 23, 2019 08:16

timmylicheng force-pushed the HDDS-2115 branch from 8b0785d to 2833470 Compare December 30, 2019 03:35

timmylicheng force-pushed the HDDS-2115 branch from b1109aa to 9fd2c86 Compare January 7, 2020 03:28

ChenSammi force-pushed the HDDS-1564 branch from 53afa84 to cc0060c Compare January 8, 2020 03:57

HDDS-2115 Add acceptance test for createPipeline CLI and datanode lis…

4d57617

…t CLI.

timmylicheng force-pushed the HDDS-2115 branch from 9fd2c86 to 4d57617 Compare January 8, 2020 03:59

retrigger build with empty commit

779206b

hanishakoneru merged commit 129d464 into apache:HDDS-1564 Jan 8, 2020

timmylicheng deleted the HDDS-2115 branch January 9, 2020 02:50

timmylicheng added a commit to timmylicheng/hadoop-ozone that referenced this pull request Feb 10, 2020

HDDS-2115 Add acceptance test for createPipeline CLI and datanode lis…

cb5486b

…t CLI (apache#375) * HDDS-2115 Add acceptance test for createPipeline CLI and datanode list CLI.

timmylicheng added a commit to timmylicheng/hadoop-ozone that referenced this pull request Feb 12, 2020

HDDS-2115 Add acceptance test for createPipeline CLI and datanode lis…

6613b05

…t CLI (apache#375) * HDDS-2115 Add acceptance test for createPipeline CLI and datanode list CLI.

timmylicheng mentioned this pull request Sep 1, 2020

HDDS-4115. CLI command to show current SCM leader and follower status. #1346

Merged

HDDS-2115 Add acceptance test for createPipeline CLI and datanode list CLI #375

HDDS-2115 Add acceptance test for createPipeline CLI and datanode list CLI #375

Uh oh!

Conversation

timmylicheng commented Dec 18, 2019

What changes were proposed in this pull request?

What is the link to the Apache JIRA

How was this patch tested?

Uh oh!

timmylicheng commented Dec 19, 2019 • edited by elek Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

timmylicheng commented Dec 19, 2019 • edited by elek Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

adoroszlai left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

timmylicheng commented Dec 19, 2019

Uh oh!

adoroszlai left a comment

Choose a reason for hiding this comment

Uh oh!

hanishakoneru commented Dec 21, 2019

Uh oh!

hanishakoneru commented Dec 21, 2019

Uh oh!

timmylicheng commented Dec 23, 2019

Uh oh!

timmylicheng commented Dec 23, 2019

Uh oh!

elek commented Jan 6, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

hanishakoneru commented Jan 6, 2020

Uh oh!

timmylicheng commented Jan 7, 2020

Uh oh!

adoroszlai commented Jan 7, 2020

Uh oh!

hanishakoneru commented Jan 7, 2020

Uh oh!

timmylicheng commented Jan 8, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

elek commented Jan 8, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

hanishakoneru commented Jan 8, 2020

Uh oh!

hanishakoneru commented Jan 8, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

timmylicheng commented Dec 19, 2019 •

edited by elek

Loading

timmylicheng commented Dec 19, 2019 •

edited by elek

Loading

elek commented Jan 6, 2020 •

edited

Loading

timmylicheng commented Jan 8, 2020 •

edited

Loading

elek commented Jan 8, 2020 •

edited

Loading