Skip to content

HDDS-2646. Start acceptance tests only if at least one THREE pipeline is available #282

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
anuengineer merged 6 commits into from
Dec 4, 2019
Merged

HDDS-2646. Start acceptance tests only if at least one THREE pipeline is available #282

Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
30effc2
HDDS-2646. Start acceptance tests only if at least one THREE pipeline…
elek Nov 28, 2019
41733fc
retrigger build
elek Nov 28, 2019
758c7d6
Merge remote-tracking branch 'origin/master' into HDDS-2646
elek Nov 29, 2019
bd99167
remove configs where we don't need them
elek Dec 3, 2019
1300a0a
switch to use ENV based variable with default
elek Dec 3, 2019
0312cdf
Merge remote-tracking branch 'origin/master' into HDDS-2646
elek Dec 3, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions hadoop-ozone/dist/src/main/compose/ozone-hdfs/docker-config
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,7 @@ OZONE-SITE.XML_ozone.metadata.dirs=/data/metadata
OZONE-SITE.XML_ozone.scm.client.address=scm
OZONE-SITE.XML_ozone.replication=1
OZONE-SITE.XML_hdds.datanode.dir=/data/hdds
OZONE-SITE.XML_hdds.scm.safemode.min.datanode=3
Comment thread
elek marked this conversation as resolved.
Outdated

HDFS-SITE.XML_dfs.datanode.plugins=org.apache.hadoop.ozone.HddsDatanodeService
HDFS-SITE.XML_dfs.namenode.rpc-address=namenode:9000
Expand Down
1 change: 1 addition & 0 deletions hadoop-ozone/dist/src/main/compose/ozone-mr/common-config
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@ OZONE-SITE.XML_ozone.scm.block.client.address=scm
OZONE-SITE.XML_ozone.metadata.dirs=/data/metadata
OZONE-SITE.XML_ozone.scm.client.address=scm
OZONE-SITE.XML_ozone.replication=3
OZONE-SITE.XML_hdds.scm.safemode.min.datanode=3

@ChenSammi ChenSammi Nov 29, 2019

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not very faimilar with docker-compose. Where do we tell docker-compose to start three datanodes with all these configurations?

@elek elek Dec 3, 2019

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

test.sh calls start_docker_env from the testlib.sh which calls docker-compose scale datanode=3.

Unfortunately there is no easy way to define the expected number of the containers in the docker-compose.yaml. ( There is a deploy / replicas but it's available only for docker swarm and not for docker-compose)

@ChenSammi ChenSammi Dec 5, 2019

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see. Thanks for the explanation.


OZONE-SITE.XML_hdds.datanode.dir=/data/hdds
HDFS-SITE.XML_dfs.datanode.address=0.0.0.0:1019
Expand Down
1 change: 1 addition & 0 deletions hadoop-ozone/dist/src/main/compose/ozone-om-ha-s3/docker-config
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,5 +29,6 @@ OZONE-SITE.XML_ozone.metadata.dirs=/data/metadata
OZONE-SITE.XML_ozone.handler.type=distributed
OZONE-SITE.XML_ozone.scm.client.address=scm
OZONE-SITE.XML_ozone.replication=3
OZONE-SITE.XML_hdds.scm.safemode.min.datanode=3
OZONE-SITE.XML_hdds.datanode.dir=/data/hdds
OZONE-SITE.XML_hdds.profiler.endpoint.enabled=true
1 change: 1 addition & 0 deletions hadoop-ozone/dist/src/main/compose/ozone-recon/docker-config
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,7 @@ OZONE-SITE.XML_ozone.scm.client.address=scm
OZONE-SITE.XML_ozone.replication=3
OZONE-SITE.XML_hdds.datanode.dir=/data/hdds
OZONE-SITE.XML_hdds.profiler.endpoint.enabled=true
OZONE-SITE.XML_hdds.scm.safemode.min.datanode=3
HDFS-SITE.XML_rpc.metrics.quantile.enable=true
HDFS-SITE.XML_rpc.metrics.percentiles.intervals=60,300
ASYNC_PROFILER_HOME=/opt/profiler
Expand Down
2 changes: 2 additions & 0 deletions hadoop-ozone/dist/src/main/compose/ozone-topology/docker-config
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,8 @@ OZONE-SITE.XML_ozone.scm.container.placement.impl=org.apache.hadoop.hdds.scm.con
OZONE-SITE.XML_net.topology.node.switch.mapping.impl=org.apache.hadoop.net.TableMapping
OZONE-SITE.XML_net.topology.table.file.name=/opt/hadoop/compose/ozone-topology/network-config
OZONE-SITE.XML_dfs.network.topology.aware.read.enable=true
OZONE-SITE.XML_hdds.scm.safemode.min.datanode=3

HDFS-SITE.XML_rpc.metrics.quantile.enable=true
HDFS-SITE.XML_rpc.metrics.percentiles.intervals=60,300
ASYNC_PROFILER_HOME=/opt/profiler
Expand Down
1 change: 1 addition & 0 deletions hadoop-ozone/dist/src/main/compose/ozone/docker-config
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,7 @@ OZONE-SITE.XML_ozone.scm.client.address=scm
OZONE-SITE.XML_ozone.replication=3
OZONE-SITE.XML_hdds.datanode.dir=/data/hdds
OZONE-SITE.XML_hdds.profiler.endpoint.enabled=true
OZONE-SITE.XML_hdds.scm.safemode.min.datanode=3

@adoroszlai adoroszlai Nov 29, 2019

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This will make it impossible to use these environments with a single datanode without modifying the config locally.

I would like to propose an alternative solution:

  1. define this config in the environment section in docker-compose.yaml using a variable that defaults to 1:
    - "OZONE-SITE.XML_hdds.scm.safemode.min.datanode=${SAFEMODE_MIN_DATANODES:-1}"
  2. set the variable to 3 in testlib.sh:
    export SAFEMODE_MIN_DATANODES=3

@elek elek Nov 29, 2019

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, this is the problem what I tried to describe in #238 I am fine with the suggested approach but it makes more complex the definition.

What I am thinking is to create a simple compose definition which can work with one datanode (and we can adjust there the replication factor and the s3 storage type as well).

Almost all the tested functionality requires datanode=3, it seems to be enough to have one cluster which can work with one datanode...

@adoroszlai adoroszlai Nov 30, 2019

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK, I'm fine with the hard-coded values in order to get acceptance tests in a good shape. We can refine it later. Until then, config can be edited locally if needed.

@elek elek Dec 3, 2019

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

During an offline conversation I understood your use case: at some cases it can be useful to make the compose folder usable just with one datanode (eg. when ui / recon or shell scripts is tested).

I pushed a new commit to experiment with your proposal.

HDFS-SITE.XML_rpc.metrics.quantile.enable=true
HDFS-SITE.XML_rpc.metrics.percentiles.intervals=60,300
ASYNC_PROFILER_HOME=/opt/profiler
Expand Down
2 changes: 2 additions & 0 deletions hadoop-ozone/dist/src/main/compose/ozoneblockade/docker-config
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,8 @@ OZONE-SITE.XML_hdds.scm.wait.time.after.safemode.exit=30s
OZONE-SITE.XML_hdds.scm.replication.thread.interval=6s
OZONE-SITE.XML_hdds.scm.replication.event.timeout=10s
OZONE-SITE.XML_dfs.ratis.server.failure.duration=35s
OZONE-SITE.XML_hdds.scm.safemode.min.datanode=3
Comment thread
elek marked this conversation as resolved.

HDFS-SITE.XML_rpc.metrics.quantile.enable=true
HDFS-SITE.XML_rpc.metrics.percentiles.intervals=60,300

Expand Down
1 change: 1 addition & 0 deletions hadoop-ozone/dist/src/main/compose/ozoneperf/docker-config
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,7 @@ OZONE-SITE.XML_hdds.datanode.dir=/data/hdds
OZONE-SITE.XML_ozone.replication=3
OZONE-SITE.XML_hdds.prometheus.endpoint.enabled=true
OZONE-SITE.XML_hdds.profiler.endpoint.enabled=true
OZONE-SITE.XML_hdds.scm.safemode.min.datanode=3

ASYNC_PROFILER_HOME=/opt/profiler

Expand Down
1 change: 1 addition & 0 deletions hadoop-ozone/dist/src/main/compose/ozones3-haproxy/docker-config
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@ OZONE-SITE.XML_ozone.metadata.dirs=/data/metadata
OZONE-SITE.XML_ozone.scm.client.address=scm
OZONE-SITE.XML_ozone.replication=3
OZONE-SITE.XML_hdds.datanode.dir=/data/hdds
OZONE-SITE.XML_hdds.scm.safemode.min.datanode=3

HDFS-SITE.XML_rpc.metrics.quantile.enable=true
HDFS-SITE.XML_rpc.metrics.percentiles.intervals=60,300
Expand Down
1 change: 1 addition & 0 deletions hadoop-ozone/dist/src/main/compose/ozones3/docker-config
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@ OZONE-SITE.XML_ozone.metadata.dirs=/data/metadata
OZONE-SITE.XML_ozone.scm.client.address=scm
OZONE-SITE.XML_ozone.replication=3
OZONE-SITE.XML_hdds.datanode.dir=/data/hdds
OZONE-SITE.XML_hdds.scm.safemode.min.datanode=3

HDFS-SITE.XML_rpc.metrics.quantile.enable=true
HDFS-SITE.XML_rpc.metrics.percentiles.intervals=60,300
Expand Down
2 changes: 2 additions & 0 deletions hadoop-ozone/dist/src/main/compose/ozonescripts/docker-config
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,8 @@ OZONE-SITE.XML_ozone.scm.client.address=scm
OZONE-SITE.XML_ozone.replication=1
OZONE-SITE.XML_hdds.datanode.dir=/data/hdds
OZONE-SITE.XML_hdds.datanode.plugins=org.apache.hadoop.ozone.web.OzoneHddsDatanodeService
OZONE-SITE.XML_hdds.scm.safemode.min.datanode=3
Comment thread
elek marked this conversation as resolved.
Outdated

HDFS-SITE.XML_dfs.namenode.rpc-address=namenode:9000
HDFS-SITE.XML_dfs.namenode.name.dir=/data/namenode
HDFS-SITE.XML_rpc.metrics.quantile.enable=true
Expand Down
1 change: 1 addition & 0 deletions hadoop-ozone/dist/src/main/compose/ozonesecure-mr/docker-config
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,7 @@ OZONE-SITE.XML_ozone.om.kerberos.keytab.file=/etc/security/keytabs/om.keytab
OZONE-SITE.XML_ozone.s3g.keytab.file=/etc/security/keytabs/HTTP.keytab
OZONE-SITE.XML_ozone.s3g.authentication.kerberos.principal=HTTP/s3g@EXAMPLE.COM
OZONE-SITE.XML_ozone.administrators=*
OZONE-SITE.XML_hdds.scm.safemode.min.datanode=3

OZONE-SITE.XML_ozone.security.enabled=true
OZONE-SITE.XML_hdds.scm.http.kerberos.principal=HTTP/scm@EXAMPLE.COM
Expand Down
1 change: 1 addition & 0 deletions hadoop-ozone/dist/src/main/compose/ozonesecure/docker-config
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,7 @@ OZONE-SITE.XML_ozone.recon.authentication.kerberos.principal=HTTP/recon@EXAMPLE.
OZONE-SITE.XML_ozone.recon.keytab.file=/etc/security/keytabs/HTTP.keytab
OZONE-SITE.XML_ozone.recon.db.dir=/data/metadata/recon
OZONE-SITE.XML_recon.om.snapshot.task.initial.delay=20s
OZONE-SITE.XML_hdds.scm.safemode.min.datanode=3

OZONE-SITE.XML_ozone.security.enabled=true
OZONE-SITE.XML_ozone.acl.enabled=true
Expand Down
49 changes: 0 additions & 49 deletions hadoop-ozone/dist/src/main/compose/testlib.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,54 +31,6 @@ create_results_dir() {
chmod ogu+w "$RESULT_DIR"
}

## @description print the number of datanodes up
## @param the docker-compose file
count_datanodes() {
local compose_file=$1

local jmx_url='http://scm:9876/jmx?qry=Hadoop:service=SCMNodeManager,name=SCMNodeManagerInfo'
if [[ "${SECURITY_ENABLED}" == 'true' ]]; then
docker-compose -f "${compose_file}" exec -T scm bash -c "kinit -k HTTP/scm@EXAMPLE.COM -t /etc/security/keytabs/HTTP.keytab && curl --negotiate -u : -s '${jmx_url}'"
else
docker-compose -f "${compose_file}" exec -T scm curl -s "${jmx_url}"
fi \
| jq -r '.beans[0].NodeCount[] | select(.key=="HEALTHY") | .value' || true
}

## @description wait until datanodes are up (or 30 seconds)
## @param the docker-compose file
## @param number of datanodes to wait for (default: 3)
wait_for_datanodes(){
local compose_file=$1
local -i datanode_count=${2:-3}

#Reset the timer
SECONDS=0

#Don't give it up until 30 seconds
while [[ $SECONDS -lt 90 ]]; do

#This line checks the number of HEALTHY datanodes registered in scm over the
# jmx HTTP servlet
datanodes=$(count_datanodes "${compose_file}")
if [[ "$datanodes" ]]; then
if [[ ${datanodes} -ge ${datanode_count} ]]; then

#It's up and running. Let's return from the function.
echo "$datanodes datanodes are up and registered to the scm"
return
else

#Print it only if a number. Could be not a number if scm is not yet started
echo "$datanodes datanode is up and healthy (until now)"
fi
fi

sleep 2
done
echo "WARNING! Datanodes are not started successfully. Please check the docker-compose files"
return 1
}

## @description wait until safemode exit (or 30 seconds)
## @param the docker-compose file
Expand Down Expand Up @@ -123,7 +75,6 @@ start_docker_env(){

docker-compose -f "$COMPOSE_FILE" --no-ansi down
docker-compose -f "$COMPOSE_FILE" --no-ansi up -d --scale datanode="${datanode_count}" \
&& wait_for_datanodes "$COMPOSE_FILE" "${datanode_count}" \
&& wait_for_safemode_exit "$COMPOSE_FILE" \
&& sleep 10

Expand Down