Single broken entity breaks the whole batch

[linki]

If a single dns record cannot be created, e.g. when a desired DNS name doesn't match the managed zone it will fail and the whole batch will be rolled back. We should tolerate individual records failing.

[joshrosso]

Hi @linki

Some folks I work with (or myself) may have interest in solving this problem as it's something we run into.

I'm curious on a few points, if you'd know off hand.

• Does this issue exist for all providers? Or just AWS?
• Is our preferred approach to attempt to use batch still? But fall back to single records if batch fails? Or perhaps reattempt batches with the 'tainted' record removed?

[linki]

Ho @joshrosso,

It depends whether the provider implementation in ExternalDNS uses batch logic (e.g. AWS and Google do and DigitalOcean doesn't, afaik) and then how individual failure is handled (e.g. ignore the failure, stop any further processissing, rollback everything).

If we could identify the broken record from the error message we should - like you say - ignore it and optionally taint it so it doesn't get retried in the next iteration.

If we cannot identify the record we could use a more generic strategy - like you say as well - stop using batch and apply records individually and ignore the failing ones or - a bit more efficient - split the batches in half recursively and apply them until only the smallest failing ones are left.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

[k8s-ci-robot]

@fejta-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[linki]

/reopen

[k8s-ci-robot]

@linki: Reopened this issue.

In response to this:

/reopen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[linki]

Collecting the related issues:

• bug report with first batch size feature: Fix for #368 #374
• another bug report: external-dns stops updating route53 on error #421
• another bug report and fix: AWS Route53 - InvalidChangeBatch: RDATA character limit of 32000 exceeded. #515 Add aws max change count flag #596
• another bug report: create/upsert records fail if too many AWS records to apply #731

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

[k8s-ci-robot]

@fejta-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.