[WIP] Allow different public and private DNS hostnames by bogdando · Pull Request #845 · openshift/openshift-ansible-contrib

bogdando · 2017-11-09T11:57:30Z

What does this PR do?

Allow different public and private DNS hostnames.
Do not manage hostnames with openstack provider any more.
Instead, allow the cloud-init to do its job.
Ignore the hostnames for provisioned VMs. Rely on openshift-ansible
inventory variables to configure openshift clusters.

When a private FQDN differs from the public FQDN,
define 'private_fqdn' metadata for provisioned with Heat Nova servers.
Place 'openshift_(public)_ip/hostname' vars into a static and
dynamic inventory depending on the public/private IP and FQDNs setup.

How should this be manually tested?

e2e with defaults should pass
e2e with arbitrary hostnames of provisioned VMs should pass.
Provision a cluster with -e openshift_openstack_private_hostname_suffix: openshift-priv and -e openshift_openstack_private_dns_domain: example.local.
- The stack name, nova servers names, ansible inventory hostnames should keep using the public name prefix (defaults to cluster id, 'openshift') and belong to the public DNS domain
- nslookup by short name executed on nodes should resolve with the public FQDN "autocompleted"
  (that depends on the hosts' search domain setup in /etc/resolv.conf, so this one is allowed to fail)
- nslookup by public FQDNs should resolve (with a floating IP provided) via external queries against the public DNS server
- nslookup by private FQDNs should NOT resolve via external queries against the public DNS server
- nslookup by private FQDNs executed on nodes should resolve via internal DNS server
- nslookup by public FQDNs executed on nodes should resolve via internal DNS server, yet with a private IP, not the floating IP
Deploy a cluster with -e openshift_openstack_private_hostname_suffix: openshift-priv and -e openshift_openstack_private_dns_domain: example.local. E2e should pass with default and arbitrary hostnames of provisioned VMs.
Provision using the same custom vars as above, but with an external DNS server deployed by osp 10 ref arch and multi-master setup. Expectations are the same as above.

Maintain the change compatible with
openshift/openshift-ansible#6039

Is there a relevant Issue open for this?

Provide a link to any open issues that describe the problem you are solving.

Who would you like to review this?

cc: @tomassedovic @oybed PTAL

openshift-bot · 2017-11-09T11:57:57Z

Can one of the admins verify this patch?
I understand the following commands:

bot, add author to whitelist
bot, test pull request
bot, test pull request once

openshift-bot · 2017-11-09T11:59:57Z

Can one of the admins verify this patch?
I understand the following commands:

bot, add author to whitelist
bot, test pull request
bot, test pull request once

openshift-bot · 2017-11-09T12:03:57Z

Can one of the admins verify this patch?
I understand the following commands:

bot, add author to whitelist
bot, test pull request
bot, test pull request once

bogdando · 2017-11-10T14:41:50Z

It works for me now! PTAL folks

bogdando · 2017-11-10T16:03:27Z

@tomassedovic reworked to allow arbitrary hostnames managed by cloud-init, deployment now uses openshift_(public)_hostname, f.e. like in this comment openshift/openshift-ansible#5883 (comment) described for AWS case.

tomassedovic

Thanks! I really like the hostname role removal, but it needs a few fixes before that can really work.

The way I see it though, the primary motivation for this feature is when you only have a single DNS with one zone.

See e.g. the red hat reference architecture:

https://access.redhat.com/documentation/en-us/reference_architectures/2017/html-single/deploying_and_managing_red_hat_openshift_container_platform_3.6_on_red_hat_openstack_platform_10/

It uses a single DNS server with one zone and one view. That scenario needs to be supported.

I like that this supports a private and public zone as well, but we need to be able to do a deployment on a single zone+view.

tomassedovic · 2017-11-15T12:08:20Z

playbooks/provisioning/openstack/sample-inventory/inventory.py


+        vars['openshift_public_hostname'] = server.name
+        if 'private_fqdn' in server.metadata:
+            vars['openshift_hostname'] = server.metadata.private_fqdn


This must always be set. Even when the public and private hostnames are equal.

If openshift_hostname is not set, OpenShift will try to look up servers via their hostnames and on openstack, these can have a suffix. E.g. my master hostname with this patch is master-0.openshift.example.com.rdocloud, but OpenShift is looking for master-0.openshift.example.com.

The hostname role handled that so if we want to remove it, we have to set this correctly ourselves.

Adding these two lines should fix it:

else: vars['openshift_hostname'] = server.name

The deployment works for me with arbitrary hostnames. I removed any dependencies on the host names. As I understood from some folks' comments from related patches, only the openshift_public_hostname needs to be set, as you noted. And it is set. The openshift_hostname is only needed to be set if internal and public naming differs. So this addresses exactly that case. Although I may be wrong and didn't test all the possible cases.

tomassedovic · 2017-11-15T12:11:53Z

roles/static_inventory/templates/inventory.j2

 {{ host }}{% if 'ansible_host' in hostvars[host]
 %} ansible_host={{ hostvars[host]['ansible_host'] }}{% endif %}
+{% if 'openshift_hostname' in hostvars[host]
+%} openshift_hostname={{ hostvars[host]['openshift_hostname'] }}{% endif %}


Same thing as with the dynamic inventory script here: openshift_hostname must always be set. If there is no private_fqdn, this must have the same value as openshift_public_hostname.

tomassedovic · 2017-11-15T12:34:43Z

roles/openstack-stack/templates/heat_stack.yaml.j2

              params:
                cluster_id: {{ stack_name }}
                k8s_type: {{ etcd_hostname | default('etcd') }}
+{% if openshift_openstack_private_dns_domain != openshift_openstack_public_dns_domain %}


Same in all these: the check should be for openshift_openstack_full_private_dns_domain and openshift_openstack_full_public_dns_domain.

Also, can we just skip the if altogether? And assume that if private is not set, it will default to public?

That should make the templates simpler and not harm anything.

I'll try to rework this, good idea

tomassedovic · 2017-11-15T12:41:20Z

roles/openstack-stack/templates/heat_stack.yaml.j2

                k8s_type: {{ etcd_hostname | default('etcd') }}
+{% if openshift_openstack_private_dns_domain != openshift_openstack_public_dns_domain %}
+          private_fqdn: {{ etcd_hostname | default('etcd') }}-%index%.{{ openshift_openstack_private_dns_domain }}
+{% endif %}


This should say openshift_openstack_full_public_dns_domain as well, otherwise the value here is without the prefix.

tomassedovic · 2017-11-15T12:48:44Z

roles/openstack-stack/templates/heat_stack_server.yaml.j2

-    description: Name
+    description: Public (FQDN) Name
+
+{% if openshift_openstack_private_dns_domain != openshift_openstack_public_dns_domain %}


Shouldn't this compare openshift_openstack_full_private_dns_domain and openshift_openstack_full_public_dns_domain?

Otherwise private_fqdn won't get set even when the openshift_openstack_public_hostname_suffix and openshift_openstack_private_hostname_suffix values differ.

tomassedovic · 2017-11-15T12:49:25Z

roles/openstack-stack/templates/heat_stack_server.yaml.j2

        host-type: { get_param: type }
        sub-host-type:    { get_param: subtype }
        node_labels: { get_param: node_labels }
+{% if openshift_openstack_private_dns_domain != openshift_openstack_public_dns_domain %}


And the same here, it should compare the _full_ domains here otherwise the prefix gets ignored.

tomassedovic · 2017-11-15T13:07:52Z

roles/dns-records/tasks/main.yml

+  set_fact:
+    private_named_records:
+      - view: "private"
+        zone: "{{ openshift_openstack_full_public_dns_domain }}"


Since this is using the public zone, the DNS servers never receive the entries with the private prefixes.

I think this task should say "private" everywhere here.

FWIW, that is the main point for multiple names resolvable via internal view. Public names should be as well resolved, yet with priv IPs. It doesn't work another way. PTAL the test case in the commit msg, it describes the behavior that matches this code. Firstly, let's adjust the test cases if I understood the thing wrong

tomassedovic · 2017-11-15T13:26:11Z

playbooks/provisioning/openstack/sample-inventory/group_vars/all.yml

+# and the private DNS domain.
+openshift_openstack_public_hostname_suffix: "{{ openshift_openstack_clusterid }}"
+openshift_openstack_private_hostname_suffix: "{{ openshift_openstack_clusterid }}"
+openshift_openstack_private_dns_domain: "{{ openshift_openstack_public_dns_domain }}"


Can we set a default value so this doesn't break existing users?

it has it at the line 3, transitioned via the line 10

tomassedovic · 2017-11-15T14:40:26Z

Also, I wonder whether setting the openshift hostname and IP options like we do here means we could do the deployment without requiring the internal DNS whatsoever.

I'll go & try to test that. If we can, that'll be fantastic news for the move to openshift-ansible, because they don't want to ship DNS there.

Signed-off-by: Bogdan Dobrelya <bdobreli@redhat.com>

Keep using public FQDN for hostnames. Make DNS server to resolve both priv/pub FQDNs via internal network and only external FQDNs via external access. Signed-off-by: Bogdan Dobrelya <bdobreli@redhat.com>

When private FQDNs differ from public ones, add the public domain into the private multi-zoned view (or private DNS server) to contain public FQDNs as well, yet resolving to private IPs. So it can resolve both private and public FQDNs as internal IPs for the cluster control network. Signed-off-by: Bogdan Dobrelya <bdobreli@redhat.com>

Instead, allow the cloud-init to do its job. Ignore the hostnames for provisioned VMs. Rely on openshift-ansible inventory variables to configure openshift clusters. When a private FQDN differs from the public FQDN, define 'private_fqdn' metadata for provisioned with Heat Nova servers. Place 'openshift_(public)_ip/hostname' vars into a static and dynamic inventory depending on the public/private IP and FQDNs setup. Signed-off-by: Bogdan Dobrelya <bdobreli@redhat.com>

Signed-off-by: Bogdan Dobrelya <bdobreli@redhat.com>

bogdando · 2017-11-17T10:20:44Z

It's time to resubmit on top of openshift/openshift-ansible#6039

bogdando · 2017-11-17T13:31:30Z

because they don't want to ship DNS there

;-(

setting the openshift hostname and IP options like we do here means we could do the deployment
without requiring the internal DNS whatsoever.

It seems like exactly the case, right. See this comment as an example for AWS It looks like they do not need DNS resolve at all to deploy. Not sure how things work for workloads (e2e) relying on SkyDNS or the like:

Generally the only things that need to resolve externally are the
API and any domains, which ought to be going through a LB

bogdando · 2017-11-17T13:38:04Z

@tomassedovic btw, I hope we could merge it here, once I submit those fixes for _full_ != ..., and hope to see it either addressed in the openshift-ansible as well, or not.

bogdando · 2017-11-28T10:42:01Z

DNS management is deprecated for the provider. And the provider had been moved into openshift-ansible w/o DNS setup bits.

bogdando · 2017-12-06T16:51:50Z

Reimplemented as openshift/openshift-ansible#6349

bogdando added the osp label Nov 9, 2017

bogdando changed the title ~~Allow different public and private DNS hostnames~~ [WIP] Allow different public and private DNS hostnames Nov 9, 2017

bogdando changed the title ~~[WIP] Allow different public and private DNS hostnames~~ Allow different public and private DNS hostnames Nov 10, 2017

bogdando mentioned this pull request Nov 10, 2017

[WIP] Hostlevel DNS/connectivity pre-flight checks openshift/openshift-ansible#5883

Closed

tomassedovic suggested changes Nov 15, 2017

View reviewed changes

bogdando added 5 commits November 17, 2017 10:20

Allow different public and private DNS hostnames

a24256d

Signed-off-by: Bogdan Dobrelya <bdobreli@redhat.com>

Do not change hostnames if private FQDNs added

4f40511

Keep using public FQDN for hostnames. Make DNS server to resolve both priv/pub FQDNs via internal network and only external FQDNs via external access. Signed-off-by: Bogdan Dobrelya <bdobreli@redhat.com>

Update documentaion related to hostnames options

afe8791

Signed-off-by: Bogdan Dobrelya <bdobreli@redhat.com>

bogdando changed the title ~~Allow different public and private DNS hostnames~~ [WIP] Allow different public and private DNS hostnames Nov 17, 2017

bogdando closed this Nov 28, 2017

bogdando deleted the custom_dns_prefixes branch November 28, 2017 11:28

Conversation

bogdando commented Nov 9, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

How should this be manually tested?

Is there a relevant Issue open for this?

Who would you like to review this?

Uh oh!

openshift-bot commented Nov 9, 2017

Uh oh!

openshift-bot commented Nov 9, 2017

Uh oh!

openshift-bot commented Nov 9, 2017

Uh oh!

bogdando commented Nov 10, 2017

Uh oh!

bogdando commented Nov 10, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tomassedovic left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bogdando Nov 17, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bogdando Nov 17, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bogdando Nov 17, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tomassedovic commented Nov 15, 2017

Uh oh!

bogdando commented Nov 17, 2017

Uh oh!

bogdando commented Nov 17, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

bogdando commented Nov 17, 2017

Uh oh!

bogdando commented Nov 28, 2017

Uh oh!

bogdando commented Dec 6, 2017

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

bogdando commented Nov 9, 2017 •

edited

Loading

bogdando commented Nov 10, 2017 •

edited

Loading

bogdando Nov 17, 2017 •

edited

Loading

bogdando Nov 17, 2017 •

edited

Loading

bogdando Nov 17, 2017 •

edited

Loading

bogdando commented Nov 17, 2017 •

edited

Loading