Skip to content

HDDS-2588. Consolidate compose environments #238

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
adoroszlai wants to merge 8 commits into from
Closed

HDDS-2588. Consolidate compose environments #238

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
800cda2
HDDS-2588. Consolidate compose environments
adoroszlai Nov 20, 2019
b4b5866
HDDS-2588. Rename ozoneperf compose env to ozone
adoroszlai Nov 20, 2019
1677175
HDDS-2588. Extract monitoring and profiling into composable parts
adoroszlai Nov 21, 2019
2fe6bb8
Merge remote-tracking branch 'origin/master' into HDDS-2588
adoroszlai Nov 30, 2019
246e35b
HDDS-2588. Update README, fix Freon
adoroszlai Nov 30, 2019
382e2a5
HDDS-2588. Add run.sh for convenience; control replication factor via…
adoroszlai Dec 1, 2019
ae2bc55
HDDS-2588. No need for one-liner config files
adoroszlai Dec 4, 2019
537da78
Merge remote-tracking branch 'origin/master' into HDDS-2588
adoroszlai Dec 5, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
30 changes: 25 additions & 5 deletions hadoop-ozone/dist/src/main/compose/ozone/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,10 +23,27 @@ There are two optional add-ons:

## How to start

Start the cluster with `docker-compose`
TL;DR:

* one datanode: `docker-compose up -d`
* three datanodes: `docker-compose up -d --scale datanode=3`
1. single datanode:
```
./run.sh -d
```
2. three datanodes for replication:
```
export OZONE_REPLICATION_FACTOR=3
./run.sh -d
```

### Basics

The cluster can be started with regular `docker-compose up` command. Use `-d` to start the cluster in the background.

You can change the number of datanodes to start using the `--scale` option. Eg. to start 3 datanodes: `docker-compose up -d --scale datanode=3`.

The cluster's replication factor (1 or 3) can be controlled by setting the `OZONE_REPLICATION_FACTOR` environment variable. It defaults to 1 to match the number of datanodes started by default, without the `--scale` option.

For convenience the `run.sh` script can be used to make sure the replication factor and the number of datanodes match. It also passes any additional arguments provided on the command-line (eg. `-d`) to `docker-compose`.

### Add-ons

Expand All @@ -52,11 +69,14 @@ docker-compose exec scm bash
ozone freon ockg -n1000
```

You can also start two flavors of Freon as separate services, which allows scaling it up. Once all the datanodes are started, start Freon by adding its definition to `COMPOSE_FILE` and re-running the `docker-compose up` command:
You can also start two flavors of Freon as separate services, which allows scaling them up. Once all the datanodes are started, start Freon by adding its definition to `COMPOSE_FILE` and re-running the `docker-compose up` or `run.sh` command:

```
export COMPOSE_FILE="${COMPOSE_FILE}:freon-ockg.yaml"
docker-compose up -d --scale datanode=3

docker-compose up -d --no-recreate --scale datanode=3
# OR
./run.sh -d
```

## How to use
Expand Down
18 changes: 12 additions & 6 deletions hadoop-ozone/dist/src/main/compose/ozone/docker-compose.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,9 @@ x-common-config:
- ../..:/opt/hadoop
env_file:
- docker-config
environment:
- "OZONE-SITE.XML_hdds.scm.safemode.min.datanode=${OZONE_REPLICATION_FACTOR:-1}"
- "OZONE-SITE.XML_ozone.replication=${OZONE_REPLICATION_FACTOR:-1}"

services:
datanode:
Expand All @@ -34,17 +37,19 @@ services:
command: ["ozone","datanode"]
om:
<<: *common-config
env_file:
- docker-config
- om.conf
ports:
- 9874:9874
environment:
ENSURE_OM_INITIALIZED: /data/metadata/om/current/VERSION
command: ["ozone","om"]
scm:
<<: *common-config
ports:
- 9876:9876
environment:
ENSURE_SCM_INITIALIZED: /data/metadata/scm/current/VERSION
Copy link
Member

@elek elek Dec 3, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is it better to move out this to a separated file? I think it's easier to overview if it's in this file. Do we really need a separated file to store this one line?

Copy link
Contributor Author

@adoroszlai adoroszlai Dec 3, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I hated moving it out, but the following considerations together made me do it:

  1. Variable substitution only works in the yaml files, not in configs passed via env_file. So safemode.min.datanode and ozone.replication need to be in environment.
  2. When merging common-config, one complete dict overrides the other: depending on the order either common-config or the specific service gets to define environment.

So I moved out these one-liners to avoid duplicating the "two-liners". Plus it seems unlikely that anyone ever wants to change these infrastructure related variables.

Now another alternative occurred to me: we might define a separate dict for the configs to be merged into environment. Let me experiment a bit with this. If it works, we could avoid the separate config files.

Copy link
Member

@elek elek Dec 6, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would prefer the separated common configs. Especially as we have only a few lines of settings they can be included in the common configs all together. But it's not a blocker for now, we can commit it (thanks to explain the reason behind the small files...)

Copy link
Contributor Author

@adoroszlai adoroszlai Dec 6, 2019 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ae2bc55 moves these one-liners back to docker-compose.yaml and the separate files are no longer needed.

env_file:
- docker-config
- scm.conf
command: ["ozone","scm"]
s3g:
<<: *common-config
Expand All @@ -55,6 +60,7 @@ services:
<<: *common-config
ports:
- 9888:9888
environment:
WAITFOR: om:9874
env_file:
- docker-config
- recon.conf
command: ["ozone","recon"]
1 change: 0 additions & 1 deletion hadoop-ozone/dist/src/main/compose/ozone/docker-config
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,4 +24,3 @@ OZONE-SITE.XML_ozone.recon.db.dir=/data/metadata/recon
OZONE-SITE.XML_ozone.recon.om.db.dir=/data/metadata/recon
Copy link
Contributor

@bharatviswa504 bharatviswa504 Nov 21, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/data/metadata/reon -> /data/metadata/om

Copy link
Contributor Author

@adoroszlai adoroszlai Nov 21, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @bharatviswa504 for spotting this. These config values from ozone-recon env as they were.

@avijayanhwx @swagle can Recon use the same directory for both ozone.recon.db.dir and ozone.recon.om.db.dir?

Copy link
Contributor

@avijayanhwx avijayanhwx Nov 21, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@adoroszlai Yes the same directory can be used.

Copy link
Contributor Author

@adoroszlai adoroszlai Nov 21, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @avijayanhwx. Then I think we can keep it as is.

OZONE-SITE.XML_ozone.scm.client.address=scm
OZONE-SITE.XML_hdds.datanode.dir=/data/hdds
OZONE-SITE.XML_ozone.replication=3
3 changes: 3 additions & 0 deletions hadoop-ozone/dist/src/main/compose/ozone/freon-ockg.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,4 +23,7 @@ services:
env_file:
- docker-config
- monitoring.conf
environment:
- "OZONE-SITE.XML_hdds.scm.safemode.min.datanode=${OZONE_REPLICATION_FACTOR:-1}"
- "OZONE-SITE.XML_ozone.replication=${OZONE_REPLICATION_FACTOR:-1}"
command: ["ozone","freon","ockg","-n100000"]
3 changes: 3 additions & 0 deletions hadoop-ozone/dist/src/main/compose/ozone/freon-rk.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,4 +23,7 @@ services:
env_file:
- docker-config
- monitoring.conf
environment:
- "OZONE-SITE.XML_hdds.scm.safemode.min.datanode=${OZONE_REPLICATION_FACTOR:-1}"
- "OZONE-SITE.XML_ozone.replication=${OZONE_REPLICATION_FACTOR:-1}"
command: ["ozone","freon","rk"]
17 changes: 17 additions & 0 deletions hadoop-ozone/dist/src/main/compose/ozone/om.conf
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

ENSURE_OM_INITIALIZED=/data/metadata/om/current/VERSION
17 changes: 17 additions & 0 deletions hadoop-ozone/dist/src/main/compose/ozone/recon.conf
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

WAITFOR=om:9874
20 changes: 20 additions & 0 deletions hadoop-ozone/dist/src/main/compose/ozone/run.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
#!/usr/bin/env bash
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

declare -ix OZONE_REPLICATION_FACTOR
: ${OZONE_REPLICATION_FACTOR:=1}
docker-compose up --scale datanode=${OZONE_REPLICATION_FACTOR} --no-recreate "$@"
Copy link
Member

@elek elek Dec 3, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you please help me to understand why do we need --no-recreate?

Copy link
Contributor Author

@adoroszlai adoroszlai Dec 3, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Without --no-recreate, if I run docker-compose up a second time to start Freon after cluster is up, Docker Compose may recreate all existing containers. I noticed that this happens when running the second command from another terminal, while following logs in the original terminal. Not sure if this is really the cause, or what other conditions could trigger it. Using the flag helps avoid this situation.

Do you think it may cause problems in other cases?

Copy link
Contributor

@anuengineer anuengineer Dec 4, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Well, while I agree the call for re-create is correct here. @adoroszlai You did ask the question. The issue can happen if we have a cluster that was running that has error-ed out. Then re-running this command will not reset the system. But it is probably something that we can live with, or fix much later. I predict for a long time, when someone reports an issue, we will say " make sure you kill all running docker containers". But then, traditionally that is our first debugging step whenever someone tells us that docker based stuff is not stable for them.

Just to make sure, I am +1 and ok with this change. Just responding to your question; that is all.

Copy link
Member

@elek elek Dec 6, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In my experience the docker-compose up worked well even from other terminal if nothing has been changed and the docker-compose file set was the same.

Can we start the scm first with docker-compose up -d scm and after everything else with docker-compose up -d with this no-recreate?

Copy link
Contributor Author

@adoroszlai adoroszlai Dec 6, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if nothing has been changed and the docker-compose file set was the same.

But the readme says freon compose file should be added only when datanodes are up, so the set is not the same.

https://github.com/apache/hadoop-ozone/blob/76ad638b47232761a1732281188162e5c31308d8/hadoop-ozone/dist/src/main/compose/ozoneperf/README.md#L47-L51

17 changes: 17 additions & 0 deletions hadoop-ozone/dist/src/main/compose/ozone/scm.conf
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

ENSURE_SCM_INITIALIZED=/data/metadata/scm/current/VERSION
1 change: 1 addition & 0 deletions hadoop-ozone/dist/src/main/compose/ozone/test.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@ COMPOSE_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"
export COMPOSE_DIR

export SECURITY_ENABLED=false
export OZONE_REPLICATION_FACTOR=3

# shellcheck source=/dev/null
source "$COMPOSE_DIR/../testlib.sh"
Expand Down