Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion 01_install_requirements.sh
Original file line number Diff line number Diff line change
Expand Up @@ -45,7 +45,8 @@ fi
sudo yum -y update

# make sure additional requirments are installed
sudo yum install -y bind-utils ansible python-netaddr python-virtualbmc libvirt libvirt-devel libvirt-daemon-kvm qemu-kvm virt-install jq python-ironicclient python-ironic-inspector-client python-openstackclient
sudo yum install -y bind-utils ansible python-netaddr python-virtualbmc libvirt libvirt-devel libvirt-daemon-kvm qemu-kvm virt-install jq \
python-ironicclient python-ironic-inspector-client python-openstackclient podman

if [ ! -f $HOME/.ssh/id_rsa.pub ]; then
ssh-keygen -f ~/.ssh/id_rsa -P ""
Expand Down
4 changes: 3 additions & 1 deletion 02_configure_host.sh
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,9 @@ EOF
fi

# Allow ipmi to the virtual bmc processes that we just started
sudo iptables -I INPUT -i baremetal -p udp -m udp --dport 6230:6235 -j ACCEPT
if ! sudo iptables -C INPUT -i baremetal -p udp -m udp --dport 6230:6235 -j ACCEPT ; then
sudo iptables -I INPUT -i baremetal -p udp -m udp --dport 6230:6235 -j ACCEPT
fi

# Need to route traffic from the provisioning host.
if [ "$EXT_IF" ]; then
Expand Down
72 changes: 72 additions & 0 deletions 04_setup_ironic.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,72 @@
#!/bin/bash

set -ex

source common.sh

# Get the various images
source get_images.sh

# ironic dnsmasq and ipxe config
cp ironic/dnsmasq.conf $IRONIC_DATA_DIR/
cp ironic/dualboot.ipxe ironic/inspector.ipxe $IRONIC_DATA_DIR/html/

# Either pull or build the ironic images
# To build the IRONIC image set
# IRONIC_IMAGE=https://github.com/metalkube/metalkube-ironic
for IMAGE_VAR in IRONIC_IMAGE IRONIC_INSPECTOR_IMAGE ; do
IMAGE=${!IMAGE_VAR}
# Is it a git repo?
if [[ "$IMAGE" =~ "://" ]] ; then
REPOPATH=~/${IMAGE##*/}
# Clone to ~ if not there already
[ -e "$REPOPATH" ] || git clone $IMAGE $REPOPATH
cd $REPOPATH
export $IMAGE_VAR=localhost/${IMAGE##*/}:latest
sudo podman build -t ${!IMAGE_VAR} .
cd -
else
sudo podman pull "$IMAGE"
fi
done

# Adding an IP address in the libvirt definition for this network results in
# dnsmasq being run, we don't want that as we have our own dnsmasq, so set
# the IP address here
if [ ! -e /etc/sysconfig/network-scripts/ifcfg-brovc ] ; then
echo -e "DEVICE=brovc\nONBOOT=yes\nNM_CONTROLLED=no\nTYPE=Ethernet\nBOOTPROTO=static\nIPADDR=172.22.0.1\nNETMASK=255.255.255.0" | sudo dd of=/etc/sysconfig/network-scripts/ifcfg-brovc
sudo ifdown brovc || true
sudo ifup brovc
fi

# Workaround so that the dracut network module does dhcp on eth0 & eth1
RHCOS_IMAGE_FILENAME_RAW="${RHCOS_IMAGE_FILENAME_OPENSTACK}.raw"
if [ ! -e "$IRONIC_DATA_DIR/html/images/$RHCOS_IMAGE_FILENAME_DUALDHCP" ] ; then
pushd $IRONIC_DATA_DIR/html/images
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: could $IRONIC_DATA_DIR/html/images directory be missing?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Its created above in get_images.sh

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
pushd $IRONIC_DATA_DIR/html/images
pushd "${IRONIC_DATA_DIR}/html/images"

qemu-img convert "$RHCOS_IMAGE_FILENAME_OPENSTACK" "${RHCOS_IMAGE_FILENAME_RAW}"
LOOPBACK=$(sudo losetup --show -f "${RHCOS_IMAGE_FILENAME_RAW}" | cut -f 3 -d /)
mkdir -p /tmp/mnt
sudo kpartx -a /dev/$LOOPBACK
sudo mount /dev/mapper/${LOOPBACK}p1 /tmp/mnt
sudo sed -i -e 's/ip=eth0:dhcp/ip=eth0:dhcp ip=eth1:dhcp/g' /tmp/mnt/grub2/grub.cfg
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: is it possible that ip=eth1:dhcp is already present in the config? Or ip=eth0:dhcp is absent?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The safest thing would probably be to use augeas cli for this purpose

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We could make this a little more robust in a later patch, I'm trying as much as possible to just move around the existing code to keep the patch as simple as possible.

sudo umount /tmp/mnt
sudo kpartx -d /dev/${LOOPBACK}
sudo losetup -d /dev/${LOOPBACK}
qemu-img convert -O qcow2 -c "$RHCOS_IMAGE_FILENAME_RAW" "$RHCOS_IMAGE_FILENAME_DUALDHCP"
rm "$RHCOS_IMAGE_FILENAME_RAW"
popd
fi


for name in ironic ironic-inspector ; do
sudo podman ps | grep -w "$name$" && sudo podman kill $name
sudo podman ps --all | grep -w "$name$" && sudo podman rm $name
done

# Start Ironic and inspector
sudo podman run -d --net host --privileged --name ironic \
-v $IRONIC_DATA_DIR/dnsmasq.conf:/etc/dnsmasq.conf \
-v $IRONIC_DATA_DIR/html/images:/var/www/html/images \
-v $IRONIC_DATA_DIR/html/dualboot.ipxe:/var/www/html/dualboot.ipxe \
-v $IRONIC_DATA_DIR/html/inspector.ipxe:/var/www/html/inspector.ipxe ${IRONIC_IMAGE}
sudo podman run -d --net host --privileged --name ironic-inspector "${IRONIC_INSPECTOR_IMAGE}"
File renamed without changes.
41 changes: 0 additions & 41 deletions 05_deploy_bootstrap_vm.sh → 06_deploy_bootstrap_vm.sh
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,6 @@ set -e

source ocp_install_env.sh
source common.sh
source get_images.sh
source utils.sh

# FIXME this is configuring for the libvirt backend which is dev-only ref
Expand Down Expand Up @@ -98,46 +97,6 @@ sudo systemctl reload NetworkManager
# Wait for ssh to start
while ! ssh -o "StrictHostKeyChecking=no" core@$IP id ; do sleep 5 ; done

# ironic dnsmasq and ipxe config
cat ironic/dnsmasq.conf | ssh -o "StrictHostKeyChecking=no" core@$IP sudo dd of=dnsmasq.conf
cat ironic/dualboot.ipxe | ssh -o "StrictHostKeyChecking=no" core@$IP sudo dd of=dualboot.ipxe
cat ironic/inspector.ipxe | ssh -o "StrictHostKeyChecking=no" core@$IP sudo dd of=inspector.ipxe

# Workaround so that the dracut network module does dhcp on eth0 & eth1
if [ ! -e images/redhat-coreos-maipo-47.284-openstack_dualdhcp.qcow2 ] ; then
qemu-img convert images/redhat-coreos-maipo-47.284-openstack.qcow2 images/redhat-coreos-maipo-47.284-openstack.raw
LOOPBACK=$(sudo losetup --show -f images/redhat-coreos-maipo-47.284-openstack.raw | cut -f 3 -d /)
mkdir -p /tmp/mnt
sudo kpartx -a /dev/$LOOPBACK
sudo mount /dev/mapper/${LOOPBACK}p1 /tmp/mnt
sudo sed -i -e 's/ip=eth0:dhcp/ip=eth0:dhcp ip=eth1:dhcp/g' /tmp/mnt/grub2/grub.cfg
sudo umount /tmp/mnt
sudo kpartx -d /dev/${LOOPBACK}
sudo losetup -d /dev/${LOOPBACK}
qemu-img convert -O qcow2 -c images/redhat-coreos-maipo-47.284-openstack.raw images/redhat-coreos-maipo-47.284-openstack_dualdhcp.qcow2
rm images/redhat-coreos-maipo-47.284-openstack.raw
fi

# Copy images the bootstrap node
tar -cf - images | ssh -o "StrictHostKeyChecking=no" "core@$IP" tar -xf -

# Retrieve and start the ironic container
IRONIC_IMAGE=${IRONIC_IMAGE:-"quay.io/metalkube/metalkube-ironic"}
echo -e "RHCOS_IMAGE_FILENAME_OPENSTACK=${RHCOS_IMAGE_FILENAME_OPENSTACK}\nIRONIC_IMAGE=${IRONIC_IMAGE}" \
| ssh -o "StrictHostKeyChecking=no" core@$IP sudo dd of=/etc/ironicservice

# Now that we have the Environment and the image, we can pull the image and start the ironic service
ssh -o "StrictHostKeyChecking=no" core@$IP sudo podman pull "$IRONIC_IMAGE"
ssh -o "StrictHostKeyChecking=no" core@$IP sudo systemctl start ironic.service

# Retrieve and start the inspector container
IRONIC_INSPECTOR_IMAGE=${IRONIC_INSPECTOR_IMAGE:-"quay.io/metalkube/metalkube-ironic-inspector"}
ssh -o "StrictHostKeyChecking=no" "core@$IP" sudo podman pull "${IRONIC_INSPECTOR_IMAGE}"

ssh -o "StrictHostKeyChecking=no" core@$IP sudo podman run \
-d --net host --privileged --name ironic-inspector \
"${IRONIC_INSPECTOR_IMAGE}"

# Create a master_nodes.json file
jq '.nodes[0:3] | {nodes: .}' "${NODES_FILE}" | tee "${MASTER_NODES_FILE}"

Expand Down
4 changes: 2 additions & 2 deletions 06_deploy_masters.sh → 07_deploy_masters.sh
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ source common.sh
# Note This logic will likely run in a container (on the bootstrap VM)
# for the final solution, but for now we'll prototype the workflow here
export OS_TOKEN=fake-token
export OS_URL=http://api.ostest.test.metalkube.org:6385/
export OS_URL=http://localhost:6385/

wait_for_json ironic \
"${OS_URL}/v1/nodes" \
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not specific to this patch but I noticed the wait_for_ironic thing doesn't work.

I noticed this because the default makefile target is still missing ironic, so the containers weren't ruinning:

[shardy@tripleodev2 dev-scripts]$ curl -g -X GET http://localhost:6385//v1/nodes '-H Accept: application/json -H Content-Type: application/json -H User-Agent: wait-for-json -H X-Auth-Token: fake-token'
curl: (7) Failed connect to localhost:6385; Connection refused
[shardy@tripleodev2 dev-scripts]$ curl -g -X GET http://localhost:6385//v1/nodes '-H Accept: application/json -H Content-Type: application/json -H User-Agent: wait-for-json -H X-Auth-Token: fake-token' | jq .
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0curl: (7) Failed connect to localhost:6385; Connection refused
[shardy@tripleodev2 dev-scripts]$ echo $?
0
Waiting for ironic to respond++ date +%s
+ start_time=1550658918
+ curl -g -X GET http://localhost:6385//v1/nodes '-H Accept: application/json -H Content-Type: application/json -H User-Agent: wait-for-json -H X-Auth-Token: fake-token'
+ jq .
+ echo ' Success!'
 Success!
+ return 0

Copy link
Collaborator Author

@derekhiggins derekhiggins Feb 20, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not specific to this patch but I noticed the wait_for_ironic thing doesn't work.

Ya, I also noticed this wasn't working, I havn't look into why,

I noticed this because the default makefile target is still missing ironic, so the containers weren't ruinning:

Opps I thought I added it, will fix now.

Expand All @@ -29,7 +29,7 @@ cp ocp/master.ign configdrive/openstack/latest/user_data
for node in $(jq -r .nodes[].name $MASTER_NODES_FILE); do

# FIXME(shardy) we should parameterize the image
openstack baremetal node set $node --instance-info image_source=http://172.22.0.1/images/redhat-coreos-maipo-47.284-openstack_dualdhcp.qcow2 --instance-info image_checksum=$(md5sum images/redhat-coreos-maipo-47.284-openstack_dualdhcp.qcow2 | awk '{print $1}') --instance-info root_gb=25 --property root_device="{\"name\": \"$ROOT_DISK\"}"
openstack baremetal node set $node --instance-info "image_source=http://172.22.0.1/images/$RHCOS_IMAGE_FILENAME_DUALDHCP" --instance-info image_checksum=$(md5sum "$IRONIC_DATA_DIR/html/images/$RHCOS_IMAGE_FILENAME_DUALDHCP" | awk '{print $1}') --instance-info root_gb=25 --property root_device="{\"name\": \"$ROOT_DISK\"}"
openstack baremetal node manage $node --wait
openstack baremetal node provide $node --wait
done
Expand Down
19 changes: 11 additions & 8 deletions Makefile
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
.PHONY: default requirements configure repo_sync build ocp_run clean ocp_cleanup libvirt_cleanup
default: requirements configure repo_sync build ocp_run
.PHONY: default requirements configure repo_sync ironic build ocp_run clean ocp_cleanup host_cleanup
default: requirements configure repo_sync ironic build ocp_run

requirements:
./01_install_requirements.sh
Expand All @@ -10,18 +10,21 @@ configure:
repo_sync:
./03_ocp_repo_sync.sh

ironic:
./04_setup_ironic.sh

build:
./04_build_ocp_installer.sh
./05_build_ocp_installer.sh

ocp_run:
./05_deploy_bootstrap_vm.sh
./06_deploy_masters.sh
./06_deploy_bootstrap_vm.sh
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think either here or in the default target we need to call 04_setup_ironic.sh?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

opps, will put it into default

./07_deploy_masters.sh

clean: ocp_cleanup libvirt_cleanup
clean: ocp_cleanup host_cleanup

ocp_cleanup:
./ocp_cleanup.sh

libvirt_cleanup:
./libvirt_cleanup.sh
host_cleanup:
./host_cleanup.sh

18 changes: 8 additions & 10 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -49,12 +49,16 @@ server with:
$ go run "${GOPATH}/src/github.com/metalkube/facet/main.go" server
```

- `./04_build_ocp_installer.sh`
- `./04_setup_ironic.sh`

This will setup Ironic on the host server and download the resources it requires

- `./05_build_ocp_installer.sh`

These will pull and build the openshift-install and some other things from
source.

- `./05_deploy_bootstrap_vm.sh`
- `./06_deploy_bootstrap_vm.sh`

This will run the openshift-install to generate ignition configs and boot the
bootstrap VM, including a bootstrap ironic all in one container.
Expand All @@ -70,7 +74,7 @@ Then you can interact with the k8s API on the bootstrap VM e.g
You can also see the status of the bootkube.sh script which is running via
`journalctl -b -f -u bootkube.service`.

- `./06_deploy_masters.sh`
- `./07_deploy_masters.sh`

This will deploy the master nodes via ironic, using the Ignition config
generated in the previous step.
Expand All @@ -84,17 +88,11 @@ openstack baremetal node list
...
```

To ssh to the master nodes, you can route trafic through the bootstrap node
```
sudo ip route add 172.22.0.0/24 via $(getent hosts api.ostest.test.metalkube.org | grep 192 | awk '{ print $1 }')
ssh core@etcd-<n>.ostest.test.metalkube.org
```

## Cleanup

- To clean up the ocp deployment run `./ocp_cleanup.sh`

- To clean up the dummy baremetal VMs and associated libvirt resources run `./libvirt_cleanup.sh`
- To clean up the dummy baremetal VMs and associated libvirt resources run `./host_cleanup.sh`

e.g. to clean and re-install ocp run:

Expand Down
6 changes: 6 additions & 0 deletions common.sh
Original file line number Diff line number Diff line change
Expand Up @@ -57,6 +57,12 @@ export RHCOS_IMAGE_NAME="redhat-coreos-maipo-${RHCOS_IMAGE_VERSION}"
# config drive support, or perhaps a completely new image?
export RHCOS_IMAGE_FILENAME="${RHCOS_IMAGE_NAME}-qemu.qcow2"
export RHCOS_IMAGE_FILENAME_OPENSTACK="${RHCOS_IMAGE_NAME}-openstack.qcow2"
export RHCOS_IMAGE_FILENAME_DUALDHCP="${RHCOS_IMAGE_NAME}-dualdhcp.qcow2"

# Ironic vars
export IRONIC_IMAGE=${IRONIC_IMAGE:-"quay.io/metalkube/metalkube-ironic"}
export IRONIC_INSPECTOR_IMAGE=${IRONIC_INSPECTOR_IMAGE:-"quay.io/metalkube/metalkube-ironic-inspector"}
export IRONIC_DATA_DIR="$WORKING_DIR/ironic"

# Log output automatically
LOGDIR="$(dirname $0)/logs"
Expand Down
4 changes: 4 additions & 0 deletions config_example.sh
Original file line number Diff line number Diff line change
Expand Up @@ -3,3 +3,7 @@
# Get a valid pull secret (json string) from
# You can get this secret from https://cloud.openshift.com/clusters/install#pull-secret
export PULL_SECRET=''

# Uncomment to build a copy of ironic or inspector locally
#export IRONIC_INSPECTOR_IMAGE=https://github.com/metalkube/metalkube-ironic-inspector
#export IRONIC_IMAGE=https://github.com/metalkube/metalkube-ironic
10 changes: 8 additions & 2 deletions get_images.sh
Original file line number Diff line number Diff line change
Expand Up @@ -7,9 +7,15 @@ if [ ! -f "$RHCOS_IMAGE_FILENAME" ]; then
curl --insecure --compressed -L -o "${RHCOS_IMAGE_FILENAME}" "${RHCOS_IMAGE_URL}/${RHCOS_IMAGE_VERSION}/${RHCOS_IMAGE_FILENAME}".gz
fi

mkdir -p images
pushd images
mkdir -p "$IRONIC_DATA_DIR/html/images"
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I had to chown my $WORKING_DIR for this to work, but that could be because I've still got it pointed to /home/stack which was world readable but not writable by my local shardy user..

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One thing which would be nice is to move any existing images directory to $IRONIC_DATA_DIR, then any existing users can avoid downloading all-the-things again? :)

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok

# Move images from the old directory to the new one if we have already downloaded them
# TODO: delete this in a week or so
if [ -d images ] ; then
find images -type f -exec mv {} "$IRONIC_DATA_DIR/html/images/" \;
rmdir images
fi

pushd "$IRONIC_DATA_DIR/html/images"
if [ ! -f "${RHCOS_IMAGE_FILENAME_OPENSTACK}" ]; then
curl --insecure --compressed -L -o "${RHCOS_IMAGE_FILENAME_OPENSTACK}" "${RHCOS_IMAGE_URL}/${RHCOS_IMAGE_VERSION}/${RHCOS_IMAGE_FILENAME_OPENSTACK}".gz
fi
Expand Down
6 changes: 6 additions & 0 deletions libvirt_cleanup.sh → host_cleanup.sh
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,12 @@ set -xe

source common.sh

# Kill and remove the running ironic
for name in ironic ironic-inspector ; do
sudo podman ps | grep -w "$name$" && sudo podman kill $name
sudo podman ps --all | grep -w "$name$" && sudo podman rm $name
done

ANSIBLE_FORCE_COLOR=true ansible-playbook \
-e "working_dir=$WORKING_DIR" \
-e "local_working_dir=$HOME/.quickstart" \
Expand Down
1 change: 0 additions & 1 deletion ignition_patches/bootstrap/03_ironic.json

This file was deleted.

4 changes: 3 additions & 1 deletion ironic/dnsmasq.conf
Original file line number Diff line number Diff line change
@@ -1,4 +1,6 @@
interface=eth1
interface=brovc
except-interface=lo
bind-dynamic
dhcp-range=172.22.0.10,172.22.0.100
enable-tftp
tftp-root=/tftpboot
Expand Down