Kuryr var generation in OSt dynamic inventory #6319

celebdor · 2017-11-30T10:19:23Z

This PR adds the necessary network resources for Kuryr in the heat
stack, including putting a l4 load balancer in front of the API.

It then automatically uses the resources to generate the OSEv3 variables
needed for the already existing kuryr role.

Change-Id: I6fc4c6bc9835217334db1289987daf358dcf287b
Signed-off-by: Antoni Segura Puimedon antonisp@celebdor.com

bogdando · 2017-12-04T08:45:55Z

playbooks/openstack/sample-inventory/inventory.py

+        return None
+
+    data = {}
+    for output in stack['outputs']:


could we instead use metadata properties, like we do above, and omit the stack outputs parsing, and hardocded stack names as well?

We're working on not having to hardcode the stack name (the idea is adding a tag to the stack which we'll filter for here).

But this is exactly what Heat outputs are for? Why shouldn't we use them? Also, where should we attach this metadata other than the stack? Some of the data here (the network and SG ids) are not known before the stack is created.

This is what heat outputs are for

bogdando · 2017-12-04T08:46:37Z

playbooks/openstack/sample-inventory/inventory.py

+    # TODO: Filter the cluster stack with tags once it is supported in shade
+    hardcoded_cluster_name = 'openshift.example.com'
+    stack = cloud_client.get_stack(hardcoded_cluster_name)
+    if stack is None or stack['stack_status'] != 'CREATE_COMPLETE':


I'm not sure we want a dynamic inventory to be stack-state-aware. Let's just note instead one shall use it after the stack is created perhaps?

IIUC how the dynamic inventory works, everytime that ansible needs inventory data, the script gets run. I am not sure there is a way to prevent the dynamic inventory from running before the stacking.

Toni is correct, the inventory script is run at the before any tasks are executed as well (where no stack exists yet). So the check needs to happen. But Toni: we could also just ignore the stack state and only set the vars if the right outputs exist.

If you want to keep checking the stack status (which is fine by me), it must also include UPDATE_COMPLETE though.

Folks, could we please please do not introduce the finite-state machine for heat stacks into a dynamic inventory? Let's keep this simple please.

bogdando · 2017-12-04T08:47:45Z

playbooks/openstack/sample-inventory/inventory.py

+        cloud_client.auth['user_domain_id'])
+    settings['kuryr_openstack_project_id'] = cloud_client.current_project_id
+    settings['kuryr_openstack_project_domain_name'] = (
+        cloud_client.auth['project_domain_id'])


it seems we already have inventory vars for this. Let's use them as a source of creds instead.

The Kuryr role is pre-existing and does not currently depend on the openshift_openstack playbook. That's why we adapt those vars here.

Do you mean this inventory.py has to maintain compatibility with external use cases? Please put fixme then, I hope we'll have that kuryr role merged into this repo eventually. So this can be simplified to the existing inventory vars instead.

I'll put the fixme

celebdor · 2017-12-04T09:07:10Z

playbooks/openstack/README.md

+#os_sdn_network_plugin_name: cni
+#openshift_node_proxy_mode: userspace
+#openshift_hosted_manage_registry: false
+#openshift_hosted_manage_router: false


We should add here also:
#use_trunk_ports: true

We should also mention that this requires the Trunk ports extension (which I didn't have enabled). This guide describes how to detect and validate it's enable:

https://docs.openstack.org/neutron/pike/admin/config-trunking.html

I wonder if we can check for this in our prerequisites (as a separate PR).

Done. I'll update the patch

celebdor · 2017-12-04T09:08:02Z

playbooks/openstack/sample-inventory/group_vars/all.yml

+#
+## You should set the following if you want to use Kuryr/Neutron as your SDN
+#openshift_use_kuryr: True
+#openshift_use_openshift_sdn: False


use_trunk_ports: True

tomassedovic

Looks good! I've managed to create a Heat stack, but the full deployment didn't finish for me yet.

That may be an unrelated issue, will check later. In the meantime, I've left a few comments inline.

tomassedovic · 2017-12-04T11:55:57Z

playbooks/openstack/sample-inventory/inventory.py

+    # TODO: Filter the cluster stack with tags once it is supported in shade
+    hardcoded_cluster_name = 'openshift.example.com'
+    stack = cloud_client.get_stack(hardcoded_cluster_name)
+    if stack is None or stack['stack_status'] != 'CREATE_COMPLETE':


Toni is correct, the inventory script is run at the before any tasks are executed as well (where no stack exists yet). So the check needs to happen. But Toni: we could also just ignore the stack state and only set the vars if the right outputs exist.

If you want to keep checking the stack status (which is fine by me), it must also include UPDATE_COMPLETE though.

tomassedovic · 2017-12-04T12:31:51Z

playbooks/openstack/sample-inventory/group_vars/all.yml

+
+# # If you VM images will name the ethernet device different than 'eth0',
+# # override this
+#kuryr_cni_link_interface: eth0


The Kuryr role doesn't actually have a default for this, so it must always be set. I'd prefer adding the default, but if not, the description here (and in the README) must change.

Better set the default. Done

tomassedovic · 2017-12-04T13:24:34Z

playbooks/openstack/sample-inventory/inventory.py

+    settings['kuryr_openstack_username'] = cloud_client.auth['username']
+    settings['kuryr_openstack_password'] = cloud_client.auth['password']
+    settings['kuryr_openstack_user_domain_name'] = (
+        cloud_client.auth['user_domain_id'])


Do we require Keystone V3? If so, we should mention that in the readme.

Also, this must fall back to user_domain_name if user_domain_id isn't there. And same below with the project_domain_id.

That's why I proposed to take data from inventory vars and omit that complexity of v2 vs v3 keystone. Let's please follow up on this item if I just can't get the case right.

I'll document and put a FIXME for allowing v2

tomassedovic · 2017-12-04T13:28:05Z

playbooks/openstack/README.md

+#os_sdn_network_plugin_name: cni
+#openshift_node_proxy_mode: userspace
+#openshift_hosted_manage_registry: false
+#openshift_hosted_manage_router: false


We should also mention that this requires the Trunk ports extension (which I didn't have enabled). This guide describes how to detect and validate it's enable:

https://docs.openstack.org/neutron/pike/admin/config-trunking.html

I wonder if we can check for this in our prerequisites (as a separate PR).

tomassedovic · 2017-12-04T13:30:28Z

Here's the issue I'm hitting in task [openshift_service_catalog : wait for api server to be ready]:

fatal: [master-0.openshift.example.com]: FAILED! => {"attempts": 120, "changed": false, "cmd": ["curl", "-k", "https://apiserver.kube-service-catalog.svc/healthz"], "d
elta": "0:00:10.523017", "end": "2017-12-04 13:07:43.308304", "failed": true, "msg": "non-zero return code", "rc": 6, "start": "2017-12-04 13:07:32.785287", "stderr":
"  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current\n                                 Dload  Upload   Total   Spent    Left  Speed\n\r  0
    0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0\r  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0\r  0     0
  0     0    0     0      0      0 --:--:--  0:00:01 --:--:--     0\r  0     0    0     0    0     0      0      0 --:--:--  0:00:02 --:--:--     0\r  0     0    0
 0    0     0      0      0 --:--:--  0:00:03 --:--:--     0\r  0     0    0     0    0     0      0      0 --:--:--  0:00:04 --:--:--     0\r  0     0    0     0    0
     0      0      0 --:--:--  0:00:05 --:--:--     0\r  0     0    0     0    0     0      0      0 --:--:--  0:00:06 --:--:--     0\r  0     0    0     0    0     0
     0      0 --:--:--  0:00:07 --:--:--     0\r  0     0    0     0    0     0      0      0 --:--:--  0:00:08 --:--:--     0\r  0     0    0     0    0     0      0
     0 --:--:--  0:00:09 --:--:--     0curl: (6) Could not resolve host: apiserver.kube-service-catalog.svc; Name or service not known", "stderr_lines": ["  % Total
 % Received % Xferd  Average Speed   Time    Time     Time  Current", "                                 Dload  Upload   Total   Spent    Left  Speed", "", "  0     0
  0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0", "  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0", "  0     0    0
     0    0     0      0      0 --:--:--  0:00:01 --:--:--     0", "  0     0    0     0    0     0      0      0 --:--:--  0:00:02 --:--:--     0", "  0     0    0
  0    0     0      0      0 --:--:--  0:00:03 --:--:--     0", "  0     0    0     0    0     0      0      0 --:--:--  0:00:04 --:--:--     0", "  0     0    0     0
    0     0      0      0 --:--:--  0:00:05 --:--:--     0", "  0     0    0     0    0     0      0      0 --:--:--  0:00:06 --:--:--     0", "  0     0    0     0
 0     0      0      0 --:--:--  0:00:07 --:--:--     0", "  0     0    0     0    0     0      0      0 --:--:--  0:00:08 --:--:--     0", "  0     0    0     0    0
    0      0      0 --:--:--  0:00:09 --:--:--     0curl: (6) Could not resolve host: apiserver.kube-service-catalog.svc; Name or service not known"], "stdout": "", "s
tdout_lines": []}

tomassedovic · 2017-12-04T15:49:44Z

Okay, so I'm seeing the service catalog issue above even on master without Kuryr.

bogdando · 2017-12-04T16:09:59Z

@tomassedovic perhaps we want this #6086 (comment)

tomassedovic · 2017-12-04T16:54:58Z

@bogdando good find! That worked for me on master, I'll try it with this PR. For anyone else wanting to try it, try putting these lines in your inventory/group_vars/OSEv3.yml:

openshift_enable_service_catalog: false
template_service_broker_install: false

(but as the original commenter pointted out, that means we're not deploying OpenShift 3.7 fully)

tomassedovic · 2017-12-04T17:19:55Z

@celebdor okay, with the fixes above, I got to a the deployment finished successfully.

So, how do I test this? It's not running the router and registry pods. You mentioned using oadm to create them. Can you paste the exact command to run? Going forward, can we have them created by default as with the default SDN?

How can I verify that it's all working?

bogdando

Please document everything needed to test this and have e2e passing

tomassedovic · 2017-12-06T09:13:41Z

@bogdando if I understand it correctly, the openshift router is not implemented in kuryr yet, so the traditional end-end we've been doing -- where we spin up a pod, create a route and verify its connectivity won't work with this patch.

But we should document what is testable and how to do it.

bogdando · 2018-01-02T10:12:50Z

roles/openshift_openstack/library/os_network_extensions.py

+# Copyright 2016 Red Hat, Inc. and/or its affiliates
+#
+# Ansible is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by


is it ok to add GNU / GPL here (for this repo)?

bogdando

LGTM, hoping to see how-to-test instructions please

tomassedovic · 2018-02-13T10:50:14Z

/ok-to-test

tomassedovic

I haven't tested this yet (will do shortly) but it looks solid. Needs a few small changes though.

tomassedovic · 2018-02-13T14:59:01Z

playbooks/openstack/inventory.py

+def _get_kuryr_vars(cloud_client):
+    """Returns a dictionary of Kuryr variables resulting of heat stacking"""
+    # TODO: Filter the cluster stack with tags once it is supported in shade
+    cluster_name = os.getenv('OPENSHIFT_CLUSTER', 'openshift.example.com')


This default should match the one in roles/openshift_openstack/defaults/main.yml, otherwise we get failures when OPENSHIFT_CLUSTER is unset.

I would also prefer the default to be something like openshift-cluster or just openshift. If the deployer wants to tie it to the cluster domain, let them be explicit about it.

Done. Went with 'openshift-cluster'

tomassedovic · 2018-02-13T15:00:02Z

roles/openshift_openstack/library/os_network_extensions.py

+    import shade
+    HAS_SHADE = True
+except ImportError:
+    HAS_SHADE = False


Tiny nit: I'd rather see shade = None here and test for that instead of introducing another global. But it's not a big deal.

I know it is a bit odd. I did it to match the upstream ansible/ansible code in hopes of submitting it there at some point:
https://github.com/ansible/ansible/blob/devel/lib/ansible/modules/cloud/openstack/os_keypair.py#L91-L95

All the modules there do it like this (at least the ones I checked)

Ah, fair enough. Thanks!

tomassedovic · 2018-02-13T15:03:54Z

roles/openshift_openstack/tasks/generate-templates.yml


+- name: set stack name
+  set_fact:
+    openshift_openstack_stack_name: "{{ lookup('env', 'OPENSHIFT_CLUSTER') | ternary (lookup('env', 'OPENSHIFT_CLUSTER'), omit) }}"


Set fact is kind of hard to track and we don't generally allow env lookup in roles as that can be confusing to the role users.

Please set the default in roles/openshift_openstack/defaults/main.yml to the same value as in the inventory (e.g. openshift-cluster) and put the env lookup to the import_role call in playbooks/openstack/openshift-cluster/provision.yml.

That way the standalone users of this role will have full control over the value from Ansible.

tomassedovic · 2018-02-13T15:10:02Z

roles/openshift_openstack/templates/heat_stack.yaml.j2


 {% if not openshift_openstack_provider_network_name %}
+{% if openshift_use_kuryr|default(false)|bool %}
+  api_lb:


Just checking: is the API load balancer actually necessary or is it possible to run without it?

If the latter, I'd like to make it optional, but we can deal with that later (I'll update my LB patch on top of this).

It is necessary for the pod workloads that talk to the openshift API.

tomassedovic · 2018-02-13T15:32:00Z

playbooks/openstack/README.md

+#os_sdn_network_plugin_name: cni
+#openshift_node_proxy_mode: userspace
+#openshift_hosted_manage_registry: false
+#openshift_hosted_manage_router: false


The all.yml sample file doesn't require router to be disabled but this does. Which is it? (we should be consistent)

Since we are not networking the routers (they use host networking) they should be fine. Registry still needs to wait.

tomassedovic · 2018-02-28T15:15:26Z

/lgtm

The controller only needs the segmentation driver config, which changed its name to default_driver. The cni part only cares about the link device. Change-Id: I757425c47bba976b0e5f526d98d7b396e861ec48 Signed-off-by: Antoni Segura Puimedon <antonisp@celebdor.com>

This patch adds the necessary network resources for Kuryr in the heat stack, including putting a l4 load balancer in front of the API. It then automatically uses the resources to generate the OSEv3 variables needed for the already existing kuryr role. Change-Id: I6fc4c6bc9835217334db1289987daf358dcf287b Signed-off-by: Antoni Segura Puimedon <antonisp@celebdor.com>

In order to fail gracefully we must detect when Neutron is not configured in the way Kuryr needs. Change-Id: I2c07ddc864b8622b958f18b4808f9d96f3906532 Signed-off-by: Antoni Segura Puimedon <antonisp@celebdor.com>

It is important to allow users to point to their own image builds as well as making life easier for developers that are developing some feature and want to point to their own registry. Change-Id: I4e2d69af2adf106f38167e9f55aa8e519cb8d0e4 Signed-off-by: Antoni Segura Puimedon <antonisp@celebdor.com>

Lacking this entry made the sanity check tasks fail when deploying openshift on openstack with kuryr. Change-Id: Ia139d0474f0de9584c814b6dc0e2bc8f4ea1eb84 Signed-off-by: Antoni Segura Puimedon <antonisp@celebdor.com>

This option was missing and was preventing the ConfigMap rendering from working. Change-Id: I4d8b1440f3fe2d20b9637a1fa8e4628d72694f70 Signed-off-by: Antoni Segura Puimedon <antonisp@celebdor.com>

Upstream kuryr uses kube-system. However, in OpenShift it comes with a default nodeSelector as if it were a namespace for apps. It seems that metrics and other openshift system Deployments and DaemonSets are expected to use 'openshift-infra' instead. Change-Id: Iebe09c0016320ccde0664545cefbba954d1389e0 Signed-off-by: Antoni Segura Puimedon <antonisp@celebdor.com>

The legacy neutron-lbaasv2 implementation based on haproxy is not aware of security groupts at VIP creation and sets the default SG. The default SG may not provide access from the cluster ingress, so we make sure in this patch that the right SG is used. This was not an issue with OpenStack's hybrid firewall because the haproxy running on namespaces were bypassing the security groups. The problem popped up when newer firewall implementations in Neutron fixed that hole and exposed the wrong SG setting. Finally, it is not possible to set the SG directly in Heat and that is a wontfix bug there. Signed-off-by: Antoni Segura Puimedon <antonisp@celebdor.com> Change-Id: Ia5100609655fef2ad7906a2f9df63aa8e8d2f33f

Change-Id: Ic48bbdc69e1d905f2bee0c1de914307e526e6356

Change-Id: Ie90428e79d1ca97719300c64226d12c5a762244f Signed-off-by: Antoni Segura Puimedon <antonisp@celebdor.com>

The previous patches in the branch left kuryr working but made things break when no LB nor kuryr were in the deployment. Change-Id: I3a40d5934b901ba50e098c8c3bcef033cb1c931f Signed-off-by: Antoni Segura Puimedon <antonisp@celebdor.com>

Change-Id: Ic41853de7957eb7f603cc307565207241a232fec Signed-off-by: Antoni Segura Puimedon <antonisp@celebdor.com>

tomassedovic · 2018-03-02T16:32:53Z

/lgtm

openshift-merge-robot · 2018-03-02T16:38:51Z

/test all [submit-queue is verifying that this PR is safe to merge]

tomassedovic · 2018-03-02T20:21:54Z

/retest

tomassedovic · 2018-03-03T00:20:35Z

/retest

openshift-merge-robot · 2018-03-03T04:24:40Z

/test all [submit-queue is verifying that this PR is safe to merge]

openshift-merge-robot · 2018-03-03T05:52:56Z

Automatic merge from submit-queue.

openshift-ci-robot · 2018-03-03T09:01:26Z

@celebdor: The following tests failed, say /retest to rerun them all:

Test name	Commit	Details	Rerun command
ci/openshift-jenkins/system-containers	`f5c83ef`	link	`/test system-containers`
ci/openshift-jenkins/containerized	`f5c83ef`	link	`/test containerized`
ci/openshift-jenkins/extended_conformance_install_crio	`f5c83ef`	link	`/test crio`
ci/openshift-jenkins/logging	`72a1515`	link	`/test logging`

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

celebdor changed the title ~~[wip]Kuryr var generation in OSt dynamic inventory~~ Kuryr var generation in OSt dynamic inventory Dec 3, 2017

openshift-ci-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Dec 3, 2017

bogdando reviewed Dec 4, 2017

View reviewed changes

celebdor commented Dec 4, 2017

View reviewed changes

tomassedovic suggested changes Dec 4, 2017

View reviewed changes

bogdando suggested changes Dec 5, 2017

View reviewed changes

openshift-merge-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Dec 7, 2017

openshift-merge-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Dec 21, 2017

bogdando reviewed Jan 2, 2018

View reviewed changes

bogdando approved these changes Jan 10, 2018

View reviewed changes

luis5tb added a commit to luis5tb/openshift-ansible that referenced this pull request Jan 10, 2018

Added last (needed) modifications from openshift#6319

d4b94df

openshift-merge-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jan 12, 2018

openshift-bot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Feb 13, 2018

openshift-ci-robot removed the needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. label Feb 13, 2018

tomassedovic suggested changes Feb 13, 2018

View reviewed changes

tomassedovic self-assigned this Feb 13, 2018

tomassedovic reviewed Feb 13, 2018

View reviewed changes

openshift-ci-robot added lgtm Indicates that a PR is ready to be merged. size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. and removed size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Feb 28, 2018

openshift-merge-robot removed the lgtm Indicates that a PR is ready to be merged. label Feb 28, 2018

tomassedovic approved these changes Feb 28, 2018

View reviewed changes

openshift-ci-robot added the lgtm Indicates that a PR is ready to be merged. label Feb 28, 2018

openshift-merge-robot removed the lgtm Indicates that a PR is ready to be merged. label Feb 28, 2018

celebdor and others added 12 commits March 2, 2018 10:39

Check openstack kuryr prerequisites

6c1794a

In order to fail gracefully we must detect when Neutron is not configured in the way Kuryr needs. Change-Id: I2c07ddc864b8622b958f18b4808f9d96f3906532 Signed-off-by: Antoni Segura Puimedon <antonisp@celebdor.com>

sanity_checks: add missing kuryr net_plugin

492112a

Lacking this entry made the sanity check tasks fail when deploying openshift on openstack with kuryr. Change-Id: Ia139d0474f0de9584c814b6dc0e2bc8f4ea1eb84 Signed-off-by: Antoni Segura Puimedon <antonisp@celebdor.com>

kuryr: required pub subnet configuration option

1a3a0a8

This option was missing and was preventing the ConfigMap rendering from working. Change-Id: I4d8b1440f3fe2d20b9637a1fa8e4628d72694f70 Signed-off-by: Antoni Segura Puimedon <antonisp@celebdor.com>

kuryr: fix API LB and DNS access

a64fd19

Change-Id: Ic48bbdc69e1d905f2bee0c1de914307e526e6356

kuryr: fix linting tests

a80f5e7

Change-Id: Ie90428e79d1ca97719300c64226d12c5a762244f Signed-off-by: Antoni Segura Puimedon <antonisp@celebdor.com>

openshift on openstack: fix non kuryr non API LB

195b60b

The previous patches in the branch left kuryr working but made things break when no LB nor kuryr were in the deployment. Change-Id: I3a40d5934b901ba50e098c8c3bcef033cb1c931f Signed-off-by: Antoni Segura Puimedon <antonisp@celebdor.com>

openstack: set a default when no API LB is needed

72a1515

Change-Id: Ic41853de7957eb7f603cc307565207241a232fec Signed-off-by: Antoni Segura Puimedon <antonisp@celebdor.com>

openshift-ci-robot added the lgtm Indicates that a PR is ready to be merged. label Mar 2, 2018

openshift-merge-robot merged commit 74ef4d9 into openshift:master Mar 3, 2018

luis5tb mentioned this pull request Mar 6, 2018

Adding wrong VM names to DNS #7404

Closed

Kuryr var generation in OSt dynamic inventory #6319

Kuryr var generation in OSt dynamic inventory #6319

Uh oh!

Conversation

celebdor commented Nov 30, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bogdando Dec 4, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tomassedovic left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tomassedovic commented Dec 4, 2017

Uh oh!

tomassedovic commented Dec 4, 2017

Uh oh!

bogdando commented Dec 4, 2017

Uh oh!

tomassedovic commented Dec 4, 2017

Uh oh!

tomassedovic commented Dec 4, 2017

Uh oh!

bogdando left a comment

Choose a reason for hiding this comment

Uh oh!

tomassedovic commented Dec 6, 2017

Uh oh!

bogdando Jan 2, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bogdando left a comment

Choose a reason for hiding this comment

Uh oh!

celebdor commented Nov 30, 2017 •

edited

Loading

bogdando Dec 4, 2017 •

edited

Loading

bogdando Jan 2, 2018 •

edited

Loading