Fix broken update handling of path-based routes by ironcladlou · Pull Request #21877 · openshift/origin

ironcladlou · 2019-01-28T14:43:24Z

When an existing route's path is updated to a new value not previously observed,
the new route state is correctly activated, but the previous route is not
replaced in the hostRules internal active list. Instead, the updated route
state (with a different path but duplicate UID) is appended to the active list.
A hostRules active list with more than one route instance sharing a UID and
host is an invalid internal state. The net effect is some unintended confusing
behavior which manifests when updating the route's path, including:

Updating from path A->B->A preventing future transitions to B
Updating from path A->B->A->B leaves an orphaned claim on the host in the
namespace, preventing the recreation of the route with either of the previously
observed paths until the router is restarted (causing the internal state to be
rebuilt)

Routes updates which are affected by this bug will be rejected with a status
like:

message: route foo already exposes foo-ns.os.example.com and is older
reason: HostAlreadyClaimed

Where foo refers to itself. This indicates the route update was rejected due
to a claim made by the same route being updated.

To fix the problem, ensure that when a path change is detected on the same route
the prior existing state is removed from the index, eliminating the possibility
of keeping an orphaned claim. Then add the new route as usual, which handles
acceptance/rejection normally.

Fixes https://bugzilla.redhat.com/show_bug.cgi?id=1660598

When an existing route's path is updated to a new value not previously observed, the new route state is correctly activated, but the previous route is not replaced in the `hostRules` internal active list. Instead, the updated route state (with a different path but duplicate UID) is appended to the active list. A `hostRules` active list with more than one route instance sharing a UID and host is an invalid internal state. The net effect is some unintended confusing behavior which manifests when updating the route's path, including: * Updating from path A->B->A preventing future transitions to B * Updating from path A->B->A->B leaves an orphaned claim on the host in the namespace, preventing the recreation of the route with either of the previously observed paths until the router is restarted (causing the internal state to be rebuilt) Routes updates which are affected by this bug will be rejected with a status like: message: route foo already exposes foo-ns.os.example.com and is older reason: HostAlreadyClaimed Where `foo` refers to itself. This indicates the route update was rejected due to a claim made by the same route being updated. To fix the problem, ensure that when a path change is detected on the same route the prior existing state is removed from the index, eliminating the possibility of keeping an orphaned claim. Then add the new route as usual, which handles acceptance/rejection normally.

ironcladlou · 2019-01-28T14:44:06Z

/cc @openshift/sig-network-edge @smarterclayton

knobunc · 2019-01-28T14:47:27Z

/lgtm
/approve

openshift-ci-robot · 2019-01-28T14:47:52Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: ironcladlou, knobunc

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~pkg/router/OWNERS~~ [ironcladlou,knobunc]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

ironcladlou · 2019-01-28T14:50:50Z

/cherry-pick 3.10

ironcladlou · 2019-01-28T14:53:19Z

/cherry-pick release-3.10

ironcladlou · 2019-01-28T15:19:32Z

/test extended_conformance_install

ironcladlou · 2019-01-28T15:51:55Z

/retest

ironcladlou · 2019-01-28T16:21:25Z

TASK [Run variable sanity checks] **********************************************
task path: /usr/share/ansible/openshift-ansible/playbooks/init/sanity_checks.yml:14
fatal: [localhost]: FAILED! => {
    "generated_timestamp": "2019-01-28 16:16:08.680131", 
    "msg": "last_checked_host: localhost, last_checked_var: openshift_master_identity_providers;Found removed variables: openshift_node_labels is replaced by openshift_node_groups[<item>].labels; "
}
 [WARNING]: Could not create retry file '/usr/share/ansible/openshift-
ansible/playbooks/prerequisites.retry'.         [Errno 13] Permission denied:
u'/usr/share/ansible/openshift-ansible/playbooks/prerequisites.retry'

No idea on this one...

/retest

openshift-bot · 2019-01-28T16:23:02Z

/retest

Please review the full test history for this PR and help us cut down flakes.

ironcladlou · 2019-01-28T16:30:54Z

/retest

ironcladlou · 2019-01-28T18:11:56Z

/retest

openshift-bot · 2019-01-28T18:24:44Z

/retest

Please review the full test history for this PR and help us cut down flakes.

ironcladlou · 2019-01-28T18:50:55Z

/retest

ironcladlou · 2019-01-28T19:48:30Z

/retest

ironcladlou · 2019-01-28T20:35:01Z

/retest

ironcladlou · 2019-01-28T21:02:22Z

/retest

openshift-bot · 2019-01-28T22:26:06Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2019-01-29T00:27:00Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2019-01-29T02:29:52Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2019-01-29T04:28:52Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2019-01-29T06:32:32Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2019-01-29T08:31:52Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2019-01-29T12:37:19Z

/retest

Please review the full test history for this PR and help us cut down flakes.

ironcladlou · 2019-01-29T12:39:05Z

CI errors possibly fixed by openshift-eng/aos-cd-jobs#1688

sdodson · 2019-01-29T13:43:08Z

/retest

stevekuznetsov · 2019-01-29T16:09:12Z

/cherrypick release-3.10

openshift-cherrypick-robot · 2019-01-29T16:11:54Z

@stevekuznetsov: new pull request created: #21901

Details

In response to this:

/cherrypick release-3.10

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-ci-robot added the size/M Denotes a PR that changes 30-99 lines, ignoring generated files. label Jan 28, 2019

openshift-ci-robot requested a review from smarterclayton January 28, 2019 14:44

openshift-ci-robot added the sig/network-edge label Jan 28, 2019

openshift-ci-robot assigned knobunc Jan 28, 2019

openshift-ci-robot added the lgtm Indicates that a PR is ready to be merged. label Jan 28, 2019

openshift-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jan 28, 2019

openshift-ci-robot requested review from knobunc and pecameron January 28, 2019 15:10

openshift-merge-robot merged commit b750162 into openshift:release-3.11 Jan 29, 2019

openshift-cherrypick-robot mentioned this pull request Jan 29, 2019

[release-3.10] Fix broken update handling of path-based routes #21901

Closed

ironcladlou mentioned this pull request Jan 29, 2019

[release-3.10] Fix broken update handling of path-based routes #21903

Merged

Conversation

ironcladlou commented Jan 28, 2019

Uh oh!

ironcladlou commented Jan 28, 2019

Uh oh!

knobunc commented Jan 28, 2019

Uh oh!

openshift-ci-robot commented Jan 28, 2019

Uh oh!

ironcladlou commented Jan 28, 2019

Uh oh!

ironcladlou commented Jan 28, 2019

Uh oh!

ironcladlou commented Jan 28, 2019

Uh oh!

ironcladlou commented Jan 28, 2019

Uh oh!

ironcladlou commented Jan 28, 2019

Uh oh!

openshift-bot commented Jan 28, 2019

Uh oh!

ironcladlou commented Jan 28, 2019

Uh oh!

ironcladlou commented Jan 28, 2019

Uh oh!

openshift-bot commented Jan 28, 2019

Uh oh!

ironcladlou commented Jan 28, 2019

Uh oh!

ironcladlou commented Jan 28, 2019

Uh oh!

ironcladlou commented Jan 28, 2019

Uh oh!

ironcladlou commented Jan 28, 2019

Uh oh!

openshift-bot commented Jan 28, 2019

Uh oh!

openshift-bot commented Jan 29, 2019

Uh oh!

openshift-bot commented Jan 29, 2019

Uh oh!

openshift-bot commented Jan 29, 2019

Uh oh!

openshift-bot commented Jan 29, 2019

Uh oh!

openshift-bot commented Jan 29, 2019

Uh oh!

openshift-bot commented Jan 29, 2019

Uh oh!

ironcladlou commented Jan 29, 2019

Uh oh!

sdodson commented Jan 29, 2019

Uh oh!

stevekuznetsov commented Jan 29, 2019

Uh oh!

openshift-cherrypick-robot commented Jan 29, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants