Move ironic to the virt host #75

derekhiggins · 2019-02-15T12:40:00Z

No description provided.

04_setup_ironic.sh

etingof

I have a handful of nits, some may not be relevant to this change because it moves code from one place to the other.

etingof · 2019-02-15T13:56:28Z

04_setup_ironic.sh

+
+# Workaround so that the dracut network module does dhcp on eth0 & eth1
+if [ ! -e $IRONIC_DATA_DIR/html/images/redhat-coreos-maipo-47.284-openstack_dualdhcp.qcow2 ] ; then
+    pushd $IRONIC_DATA_DIR/html/images


nit: could $IRONIC_DATA_DIR/html/images directory be missing?

Its created above in get_images.sh

etingof · 2019-02-15T14:09:17Z

04_setup_ironic.sh

+# Workaround so that the dracut network module does dhcp on eth0 & eth1
+if [ ! -e $IRONIC_DATA_DIR/html/images/redhat-coreos-maipo-47.284-openstack_dualdhcp.qcow2 ] ; then
+    pushd $IRONIC_DATA_DIR/html/images
+    qemu-img convert redhat-coreos-maipo-47.284-openstack.qcow2 redhat-coreos-maipo-47.284-openstack.raw


nit: will qemu-img fail if redhat-coreos-maipo-47.284-openstack.raw accidentally exists from some unsuccessful previous run?

Nope, its ok if the file already exists

etingof · 2019-02-15T14:10:46Z

04_setup_ironic.sh

+    mkdir -p /tmp/mnt
+    sudo kpartx -a /dev/$LOOPBACK
+    sudo mount /dev/mapper/${LOOPBACK}p1 /tmp/mnt
+    sudo sed -i -e 's/ip=eth0:dhcp/ip=eth0:dhcp ip=eth1:dhcp/g' /tmp/mnt/grub2/grub.cfg 


nit: is it possible that ip=eth1:dhcp is already present in the config? Or ip=eth0:dhcp is absent?

The safest thing would probably be to use augeas cli for this purpose

We could make this a little more robust in a later patch, I'm trying as much as possible to just move around the existing code to keep the patch as simple as possible.

etingof · 2019-02-15T14:11:48Z

04_setup_ironic.sh

+#fi
+
+# Workaround so that the dracut network module does dhcp on eth0 & eth1
+if [ ! -e $IRONIC_DATA_DIR/html/images/redhat-coreos-maipo-47.284-openstack_dualdhcp.qcow2 ] ; then


nit: would it make sense to move redhat-coreos-maipo-47.284 name to a variable to ease further maintenance?

Yup, I'll add another patch to do this, the current patch is just moving code around

etingof · 2019-02-15T14:14:57Z

04_setup_ironic.sh

+
+
+for name in ironic ironic-inspector ; do 
+    sudo podman ps | grep "$name$" && sudo podman kill $name


nit: would grep -w harden the matching a bit?

will do, thanks

etingof · 2019-02-15T14:16:04Z

README.md

- `./04_build_ocp_installer.sh`
+- `./04_setup_ironic.sh`
+
+This we setup Ironic on the host server and download the resources it requires


nit: "will" perhaps

celebdor · 2019-02-15T14:29:30Z

04_setup_ironic.sh

+#fi
+
+# Workaround so that the dracut network module does dhcp on eth0 & eth1
+if [ ! -e $IRONIC_DATA_DIR/html/images/redhat-coreos-maipo-47.284-openstack_dualdhcp.qcow2 ] ; then


Suggested change

if [ ! -e $IRONIC_DATA_DIR/html/images/redhat-coreos-maipo-47.284-openstack_dualdhcp.qcow2 ] ; then

if [ ! -e "${IRONIC_DATA_DIR}/html/images/redhat-coreos-maipo-47.284-openstack_dualdhcp.qcow2" ] ; then

celebdor · 2019-02-15T14:30:01Z

04_setup_ironic.sh

+
+# Workaround so that the dracut network module does dhcp on eth0 & eth1
+if [ ! -e $IRONIC_DATA_DIR/html/images/redhat-coreos-maipo-47.284-openstack_dualdhcp.qcow2 ] ; then
+    pushd $IRONIC_DATA_DIR/html/images


Suggested change

pushd $IRONIC_DATA_DIR/html/images

pushd "${IRONIC_DATA_DIR}/html/images"

celebdor · 2019-02-15T14:33:50Z

04_setup_ironic.sh

+    mkdir -p /tmp/mnt
+    sudo kpartx -a /dev/$LOOPBACK
+    sudo mount /dev/mapper/${LOOPBACK}p1 /tmp/mnt
+    sudo sed -i -e 's/ip=eth0:dhcp/ip=eth0:dhcp ip=eth1:dhcp/g' /tmp/mnt/grub2/grub.cfg 


The safest thing would probably be to use augeas cli for this purpose

celebdor · 2019-02-15T14:36:15Z

04_setup_ironic.sh

+done
+
+# Start Ironic and inspector
+sudo podman run -d --net host --privileged --name ironic -v $IRONIC_DATA_DIR/dnsmasq.conf:/etc/dnsmasq.conf -v $IRONIC_DATA_DIR/html/images:/var/www/html/images -v $IRONIC_DATA_DIR/html/dualboot.ipxe:/var/www/html/dualboot.ipxe -v $IRONIC_DATA_DIR/html/inspector.ipxe:/var/www/html/inspector.ipxe ${IRONIC_IMAGE}


Suggested change

sudo podman run -d --net host --privileged --name ironic -v $IRONIC_DATA_DIR/dnsmasq.conf:/etc/dnsmasq.conf -v $IRONIC_DATA_DIR/html/images:/var/www/html/images -v $IRONIC_DATA_DIR/html/dualboot.ipxe:/var/www/html/dualboot.ipxe -v $IRONIC_DATA_DIR/html/inspector.ipxe:/var/www/html/inspector.ipxe ${IRONIC_IMAGE}

sudo podman run -d --net host --privileged --name ironic \

-v "${IRONIC_DATA_DIR}/dnsmasq.conf:/etc/dnsmasq.conf" \

-v "${IRONIC_DATA_DIR}/html/images:/var/www/html/images" \

-v "${IRONIC_DATA_DIR}/html/dualboot.ipxe:/var/www/html/dualboot.ipxe" \

-v "${IRONIC_DATA_DIR}/html/inspector.ipxe:/var/www/html/inspector.ipxe" \

"$IRONIC_IMAGE"

Ideally we'd use the bridge cni plugin to run it with an interface on the brovc bridge so it doesn't need net host. That would probably even let it run non-privileged.

I'm not familiar with the bridge cni plugin, is it something we could swap out later?

celebdor · 2019-02-15T14:38:01Z

07_deploy_masters.sh


  # FIXME(shardy) we should parameterize the image
-  openstack baremetal node set $node --instance-info image_source=http://172.22.0.1/images/redhat-coreos-maipo-47.284-openstack_dualdhcp.qcow2 --instance-info image_checksum=$(md5sum images/redhat-coreos-maipo-47.284-openstack_dualdhcp.qcow2 | awk '{print $1}') --instance-info root_gb=25 --property root_device="{\"name\": \"$ROOT_DISK\"}"
+  openstack baremetal node set $node --instance-info image_source=http://172.22.0.1/images/redhat-coreos-maipo-47.284-openstack_dualdhcp.qcow2 --instance-info image_checksum=$(md5sum $IRONIC_DATA_DIR/html/images/redhat-coreos-maipo-47.284-openstack_dualdhcp.qcow2 | awk '{print $1}') --instance-info root_gb=25 --property root_device="{\"name\": \"$ROOT_DISK\"}"


Suggested change

openstack baremetal node set $node --instance-info image_source=http://172.22.0.1/images/redhat-coreos-maipo-47.284-openstack_dualdhcp.qcow2 --instance-info image_checksum=$(md5sum $IRONIC_DATA_DIR/html/images/redhat-coreos-maipo-47.284-openstack_dualdhcp.qcow2 | awk '{print $1}') --instance-info root_gb=25 --property root_device="{\"name\": \"$ROOT_DISK\"}"

openstack baremetal node set $node \

--instance-info image_source=http://172.22.0.1/images/redhat-coreos-maipo-47.284-openstack_dualdhcp.qcow2 \

--instance-info image_checksum=$(md5sum "${IRONIC_DATA_DIR}/html/images/redhat-coreos-maipo-47.284-openstack_dualdhcp.qcow2" | awk '{print $1}') \

--instance-info root_gb=25 --property root_device="{\"name\": \"$ROOT_DISK\"}"

Gonna leave this as is for the moment as I havn't touched the file, just renamed it

celebdor · 2019-02-15T14:39:14Z

common.sh

+# Ironic vars
+export IRONIC_IMAGE=${IRONIC_IMAGE:-"quay.io/metalkube/metalkube-ironic"}
+export IRONIC_INSPECTOR_IMAGE=${IRONIC_INSPECTOR_IMAGE:-"quay.io/metalkube/metalkube-ironic-inspector"}
+export IRONIC_DATA_DIR=$WORKING_DIR/ironic


Suggested change

export IRONIC_DATA_DIR=$WORKING_DIR/ironic

export IRONIC_DATA_DIR="${WORKING_DIR}/ironic"

celebdor · 2019-02-15T14:39:36Z

get_images.sh


-mkdir -p images
-pushd images
+mkdir -p $IRONIC_DATA_DIR/html/images


Suggested change

mkdir -p $IRONIC_DATA_DIR/html/images

mkdir -p "${IRONIC_DATA_DIR}/html/images"

celebdor · 2019-02-15T14:39:51Z

get_images.sh

-mkdir -p images
-pushd images
+mkdir -p $IRONIC_DATA_DIR/html/images
+pushd $IRONIC_DATA_DIR/html/images


Suggested change

pushd $IRONIC_DATA_DIR/html/images

pushd "${IRONIC_DATA_DIR}/html/images"

hardys · 2019-02-19T14:58:20Z

Makefile

 ocp_run:
-	./05_deploy_bootstrap_vm.sh
-	./06_deploy_masters.sh
+	./06_deploy_bootstrap_vm.sh


I think either here or in the default target we need to call 04_setup_ironic.sh?

opps, will put it into default

hardys · 2019-02-19T15:02:18Z

get_images.sh


-mkdir -p images
-pushd images
+mkdir -p "$IRONIC_DATA_DIR/html/images"


I had to chown my $WORKING_DIR for this to work, but that could be because I've still got it pointed to /home/stack which was world readable but not writable by my local shardy user..

One thing which would be nice is to move any existing images directory to $IRONIC_DATA_DIR, then any existing users can avoid downloading all-the-things again? :)

hardys · 2019-02-19T15:51:01Z

This looks really close modulo nits already mentioned, but is anyone else having issues accessing the new image location via http?

[shardy@tripleodev2 html]$ wget http://172.22.0.1/
--2019-02-19 15:50:08--  http://172.22.0.1/
Connecting to 172.22.0.1:80... connected.
HTTP request sent, awaiting response... 403 Forbidden
2019-02-19 15:50:08 ERROR 403: Forbidden.

[shardy@tripleodev2 html]$ wget http://172.22.0.1/images
--2019-02-19 15:50:09--  http://172.22.0.1/images
Connecting to 172.22.0.1:80... connected.
HTTP request sent, awaiting response... 404 Not Found
2019-02-19 15:50:09 ERROR 404: Not Found.

This causes the ironic deploy to fail, but may be specific to my env (permissions look OK but I am using a non-default WORKING_DIR pointing at /home/stack)

derekhiggins · 2019-02-19T16:11:08Z

This causes the ironic deploy to fail, but may be specific to my env (permissions look OK but I am using a non-default WORKING_DIR pointing at /home/stack)

hmm, can you check if the images directory got mounted ok?

# podman exec -it ironic ls /var/www/html/images/
ironic-python-agent.initramfs  ironic-python-agent.kernel  redhat-coreos-maipo-47.284-dualdhcp.qcow2  redhat-coreos-maipo-47.284-openstack.qcow2  redhat-coreos-maipo-47.284-openstack_dualdhcp.qcow2

Also check the logs for the podman run command to make sure the correct directories were selected for the volumes.

hardys · 2019-02-20T10:32:51Z

Looks good, sorry I think I caused another conflict though by merging #85 (rebased cleanly for me locally though).

One question - where should we wire in cleanup for the ironic containers?

I guess we could create a new script and ensure it's called on make clean, or add it to the existing libvirt_cleanup (renamed to host_cleanup.sh perhaps?)

derekhiggins · 2019-02-20T10:39:52Z

Looks good, sorry I think I caused another conflict though by merging #85 (rebased cleanly for me locally though).

No prob, I'll rebase now

One question - where should we wire in cleanup for the ironic containers?

I guess we could create a new script and ensure it's called on make clean, or add it to the existing libvirt_cleanup (renamed to host_cleanup.sh perhaps?)

I'll add it to libvirt_cleanup and rename

hardys · 2019-02-20T10:43:45Z

07_deploy_masters.sh

+export OS_URL=http://localhost:6385/

 wait_for_json ironic \
    "${OS_URL}/v1/nodes" \


Not specific to this patch but I noticed the wait_for_ironic thing doesn't work.

I noticed this because the default makefile target is still missing ironic, so the containers weren't ruinning:

[shardy@tripleodev2 dev-scripts]$ curl -g -X GET http://localhost:6385//v1/nodes '-H Accept: application/json -H Content-Type: application/json -H User-Agent: wait-for-json -H X-Auth-Token: fake-token' curl: (7) Failed connect to localhost:6385; Connection refused [shardy@tripleodev2 dev-scripts]$ curl -g -X GET http://localhost:6385//v1/nodes '-H Accept: application/json -H Content-Type: application/json -H User-Agent: wait-for-json -H X-Auth-Token: fake-token' | jq . % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed connect to localhost:6385; Connection refused [shardy@tripleodev2 dev-scripts]$ echo $? 0

Waiting for ironic to respond++ date +%s + start_time=1550658918 + curl -g -X GET http://localhost:6385//v1/nodes '-H Accept: application/json -H Content-Type: application/json -H User-Agent: wait-for-json -H X-Auth-Token: fake-token' + jq . + echo ' Success!' Success! + return 0

Not specific to this patch but I noticed the wait_for_ironic thing doesn't work.

Ya, I also noticed this wasn't working, I havn't look into why,

I noticed this because the default makefile target is still missing ironic, so the containers weren't ruinning:

Opps I thought I added it, will fix now.

Just a placeholder script for the moment, so the next patches are easier to follow.

hardys · 2019-02-20T10:49:48Z

Also check the logs for the podman run command to make sure the correct directories were selected for the volumes.

Yeah this looks Ok so not yet sure why it's 404ing:

+ openstack baremetal node deploy --config-drive configdrive openshift-master-0
Failed to validate deploy or power info for node b683113a-7ce1-41d6-88c1-dd626de1c124. Error: Validation of image href http://172.22.0.1/images/redhat-coreos-maipo-47.284-dualdhcp.qcow2 failed, reason: Got HTTP code 404 instead of 200 in response to HEAD request. (HTTP 400)

[shardy@tripleodev2 dev-scripts]$ sudo podman exec -it ironic ls /var/www/html/images
ironic-python-agent.initramfs  redhat-coreos-maipo-47.284-dualdhcp.qcow2   redhat-coreos-maipo-47.284-openstack_dualdhcp.qcow2
ironic-python-agent.kernel     redhat-coreos-maipo-47.284-openstack.qcow2

[shardy@tripleodev2 dev-scripts]$ curl http://172.22.0.1/images/redhat-coreos-maipo-47.284-dualdhcp.qcow2
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>404 Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL /images/redhat-coreos-maipo-47.284-dualdhcp.qcow2 was not found on this server.</p>
</body></html>

Move the ironic and inspector containers from the bootstrap node to the virt host. This will reduce delays between booting the bootstrap node and having it available to do work.

These shouldn't have been hardcoded in the first place.

We don't need this at the moment as ironic is now on the host. We will need it soon for ironic on the openshift masters but one rule will do.

Add a hook to build the ironic containers while they are being beveloped locally. Also move some vars common.sh so they can be set in config_$USER.sh if needed.

hardys · 2019-02-20T11:18:06Z

Edit the 404 was because I had httpd running on the host, and the container starts even though there's a port conflict. I can push a follow-up to check for that.

hardys · 2019-02-20T13:12:59Z

I'm still having issues in that http://172.22.0.1/dualboot.ipxe is accessible via the host, but the openshift_master_* nodes can't reach it on boot. It seems to be using the right device, getting a DHCP lease and the next-server, but then failing with Connnection Timed out attempting to download the ipxe config.

Anyone else seeing similar? Possibly something specific to my env as this isn't a freshly installed box.

derekhiggins · 2019-02-20T15:02:40Z

I'm still having issues in that http://172.22.0.1/dualboot.ipxe is accessible via the host, but the openshift_master_* nodes can't reach it on boot. It seems to be using the right device, getting a DHCP lease and the next-server, but then failing with Connnection Timed out attempting to download the ipxe config.

Anyone else seeing similar? Possibly something specific to my env as this isn't a freshly installed box.

I havn't seen this but 2 things worth checking that spring to mind

Make sure you have only one interface with 172.22.0.1 , (also check the bootstrap node)
If you have previously added a route on your host to 172.22.0.1 , you should probably delete it (I didn't have one set so mightn't have seen the potential issue)

hardys · 2019-02-20T16:36:18Z

2. If you have previously added a route on your host to 172.22.0.1 , you should probably delete it (I didn't have one set so mightn't have seen the potential issue)

Yeah thanks this was it, I had a stale route sending traffic via virbr0 (doh!), everything now works fine for me, so I'm good to merge if nobody else has issues :)

hardys · 2019-02-20T17:32:49Z

Ok for the benefit of anyone else who may hit similar issues, I had two problems when testing this which were related to my environment - httpd was running on the host which slilently conflicts with the webserver in the ironic container, and there was a stale route for 172.22.0.0/24 which needed to be removed.

Other than that everything seems to work well so lets merge and iterate on remaining issues.

derekhiggins force-pushed the move-ironic branch from 58a9086 to a9fb092 Compare February 15, 2019 12:42

etingof reviewed Feb 15, 2019

View reviewed changes

04_setup_ironic.sh Outdated Show resolved Hide resolved

etingof reviewed Feb 15, 2019

View reviewed changes

celebdor reviewed Feb 15, 2019

View reviewed changes

derekhiggins force-pushed the move-ironic branch 4 times, most recently from 1ad9162 to d3f1048 Compare February 19, 2019 12:56

derekhiggins changed the title ~~WIP: Move ironic to the host~~ Move ironic to the virt host Feb 19, 2019

hardys reviewed Feb 19, 2019

View reviewed changes

derekhiggins force-pushed the move-ironic branch 2 times, most recently from eed56bc to b5ccabf Compare February 20, 2019 09:47

hardys reviewed Feb 20, 2019

View reviewed changes

Add additional ironic script into the setup process

9d273dc

Just a placeholder script for the moment, so the next patches are easier to follow.

derekhiggins force-pushed the move-ironic branch from b5ccabf to fa4ee66 Compare February 20, 2019 10:47

derekhiggins added 4 commits February 20, 2019 05:51

Move ironic onto the virt host

3b4d360

Move the ironic and inspector containers from the bootstrap node to the virt host. This will reduce delays between booting the bootstrap node and having it available to do work.

Remove hardcoded image names

5f01bc4

These shouldn't have been hardcoded in the first place.

Only add ipmi iptables rule once

75e5425

We don't need this at the moment as ironic is now on the host. We will need it soon for ironic on the openshift masters but one rule will do.

Allow ironic containers to be built

8b8ee80

Add a hook to build the ironic containers while they are being beveloped locally. Also move some vars common.sh so they can be set in config_$USER.sh if needed.

derekhiggins force-pushed the move-ironic branch from fa4ee66 to 8b8ee80 Compare February 20, 2019 10:51

hardys self-requested a review February 20, 2019 16:37

hardys assigned hardys and derekhiggins Feb 20, 2019

hardys approved these changes Feb 20, 2019

View reviewed changes

hardys removed their assignment Feb 20, 2019

hardys merged commit 6613c13 into openshift-metal3:master Feb 20, 2019

hardys mentioned this pull request Feb 20, 2019

Update install-config #86

Merged

derekhiggins deleted the move-ironic branch April 24, 2019 14:37



		for name in ironic ironic-inspector ; do
		sudo podman ps \| grep "$name$" && sudo podman kill $name

	if [ ! -e $IRONIC_DATA_DIR/html/images/redhat-coreos-maipo-47.284-openstack_dualdhcp.qcow2 ] ; then
	if [ ! -e "${IRONIC_DATA_DIR}/html/images/redhat-coreos-maipo-47.284-openstack_dualdhcp.qcow2" ] ; then

	pushd $IRONIC_DATA_DIR/html/images
	pushd "${IRONIC_DATA_DIR}/html/images"

-sudo podman run -d --net host --privileged --name ironic -v $IRONIC_DATA_DIR/dnsmasq.conf:/etc/dnsmasq.conf -v $IRONIC_DATA_DIR/html/images:/var/www/html/images -v $IRONIC_DATA_DIR/html/dualboot.ipxe:/var/www/html/dualboot.ipxe -v $IRONIC_DATA_DIR/html/inspector.ipxe:/var/www/html/inspector.ipxe ${IRONIC_IMAGE}
+sudo podman run -d --net host --privileged --name ironic \
+    -v "${IRONIC_DATA_DIR}/dnsmasq.conf:/etc/dnsmasq.conf" \
+    -v "${IRONIC_DATA_DIR}/html/images:/var/www/html/images" \
+    -v "${IRONIC_DATA_DIR}/html/dualboot.ipxe:/var/www/html/dualboot.ipxe" \
+    -v "${IRONIC_DATA_DIR}/html/inspector.ipxe:/var/www/html/inspector.ipxe" \
+    "$IRONIC_IMAGE"

-  openstack baremetal node set $node --instance-info image_source=http://172.22.0.1/images/redhat-coreos-maipo-47.284-openstack_dualdhcp.qcow2 --instance-info image_checksum=$(md5sum $IRONIC_DATA_DIR/html/images/redhat-coreos-maipo-47.284-openstack_dualdhcp.qcow2 | awk '{print $1}') --instance-info root_gb=25 --property root_device="{\"name\": \"$ROOT_DISK\"}"
+  openstack baremetal node set $node \
+      --instance-info image_source=http://172.22.0.1/images/redhat-coreos-maipo-47.284-openstack_dualdhcp.qcow2 \
+      --instance-info image_checksum=$(md5sum "${IRONIC_DATA_DIR}/html/images/redhat-coreos-maipo-47.284-openstack_dualdhcp.qcow2" | awk '{print $1}') \
+      --instance-info root_gb=25 --property root_device="{\"name\": \"$ROOT_DISK\"}"

	export IRONIC_DATA_DIR=$WORKING_DIR/ironic
	export IRONIC_DATA_DIR="${WORKING_DIR}/ironic"

	mkdir -p $IRONIC_DATA_DIR/html/images
	mkdir -p "${IRONIC_DATA_DIR}/html/images"

Move ironic to the virt host #75

Move ironic to the virt host #75

Uh oh!

Conversation

derekhiggins commented Feb 15, 2019

Uh oh!

Uh oh!

etingof left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

hardys commented Feb 19, 2019

Uh oh!

derekhiggins commented Feb 19, 2019

Uh oh!

hardys commented Feb 20, 2019

Uh oh!

derekhiggins commented Feb 20, 2019

Uh oh!

Choose a reason for hiding this comment

Uh oh!

derekhiggins Feb 20, 2019 •

edited

Loading