fix(txt-registry): skip creation of already-existing TXT records (#4914)

[u-kai]

What does it do ?

Adds a cache-based filter to TXTRegistry so that External-DNS no longer
tries to recreate TXT records that already exist in the provider.
Fixes #4914

Motivation

When only the data record (A/CNAME) is removed, External-DNS next sync loop
recreates that record and also tries to create the unchanged TXT record,
which providers like Route53 reject with “record already exists”.
Caching existing TXT entries and skipping them in ApplyChanges() prevents
this failure.

More

• Yes, this PR title follows Conventional Commits
• Yes, I added unit tests
• Yes, I updated end user documentation accordingly

[k8s-ci-robot]

Welcome @u-kai!

It looks like this is your first PR to kubernetes-sigs/external-dns 🎉. Please refer to our pull request process documentation to help your PR have a smooth ride to approval.

You will be prompted by a bot to use commands during the review process. Do not be afraid to follow the prompts! It is okay to experiment. Here is the bot commands documentation.

You can also check if kubernetes-sigs/external-dns has its own contribution guidelines.

You may want to refer to our testing guide if you run into trouble with your tests not passing.

If you are having difficulty getting your pull request seen, please follow the recommended escalation practices. Also, for tips and tricks in the contribution process you may want to read the Kubernetes contributor cheat sheet. We want to make sure your contribution gets all the attention it needs!

Thank you, and welcome to Kubernetes. 😃

[k8s-ci-robot]

Hi @u-kai. Thanks for your PR.

I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[mloiseleur]

Thanks.
I'm not sure of this approach 🤔 It may increase memory usage on big users.
/ok-to-test
cc @szuecs

[mloiseleur]

What happens the txt record has also been deleted ?
It will still be in this cache, so not re-created ?

[u-kai]

@mloiseleur
Thanks for pointing that out — you’re right, the current patch keeps the TXT snapshot alive across the entire reconcile loop.
Apologies — I overlooked this in the first draft.

Proposed update

func (r *TXTRegistry) ApplyChanges(ctx context.Context, c *plan.Changes) error {
    ...

    if err := r.provider.ApplyChanges(ctx, c); err != nil {
        return err
    }

    // ✨ NEW: drop the snapshot so the next Records() call rebuilds it
    r.existingTXT = txtSnapshot{}  // formerly existingTXT

    return nil
}

• Memory – The snapshot now lives for just one reconciliation pass.
  In my understanding, GC might run a little later than before, but the peak heap usage right after r.provider.Records() when the full slice of endpoints is allocated — remains exactly the same.
  If I’m mistaken, please let me know. 🙇

• Correctness – If a TXT record is deleted between loops, the next Records() call rebuilds the snapshot without it, so the data record will be recreated.

Let me know if this per-loop cleanup addresses your concerns; I can push the change right away. 🙇‍♂️

[ivankatliarchuk]

There are concerns Regarding TXT Caching as a Solution

I'm unclear how caching will resolve the underlying issue. From my perspective, this looks like a bug in either the CRD source, the TXT registry, or how records and plans are being handled. Instead of identifying and fixing that root cause, the current proposal is to introduce a new feature – TXT caching. The issue missing how-to reproduce, what the arguments are and version for reported bug is ~0.13, when latest version is 0.17, which means, the issue may no longer even exist.

The approach could actually worsen the situation for TXT records. Implementing a cache on a stateless service means the problem is likely to reappear during restarts or new service startups.

This proposed caching mechanism isn't a fix. We need to find the root cause of the problem. The expectation should be to reproduce the issue in a controlled environment and then try to pinpoint where the bug lies.

Relevant issue #4998

It appears there's a bug within either the CRD source, the TXT registry, or the mechanism for pulling records for a specific provider and handling the plan. Instead of addressing this underlying bug, the decision has been made to introduce a new feature: TXT caching.

[ivankatliarchuk]

TXT registry already have a cache

external-dns/registry/txt.go

Line 46 in 36bc7d6

recordsCache []*endpoint.Endpoint

and local endpoints and TXT records cache https://github.com/kubernetes-sigs/external-dns/blob/master/registry/txt.go#L138-L141

This is a high-impact issue, and without being able to reproduce it, it's tough to pinpoint the exact cause. It's even possible that upsert-only failed, but everything works fine when the sync flag is set.

I propose to start with:

• Adding a unit test that specifically shows the issue where the record is deleted but the registry record persists.
• Reproducing the actual bug on a local cluster using the Route53 provider and the latest external-dns.
• Implementing the fix once the root cause is identified.

First two could be done in any order, but with focus on reproducing the problem and providing a unit test, that reveals the bug.

This is currently just an assumption, and without reproducing it, it's hard to be certain. We need to confirm that if a TXT registry record exists without a corresponding managed record (like A, AAAA, or CNAME), we either:

• Mark the TXT registry record for deletion (though this won't work with upsert-only) and then apply the new changes, OR
• Leave the TXT registry record untouched and only update the actual record.

[ivankatliarchuk]

Fixes #5340

[ivankatliarchuk]

Fixes #5003

^ this issue is related, but not sure where scope of the fix will cover it

[u-kai]

@ivankatliarchuk

Thank you for your feedback.

Initially, I approached this by introducing a caching mechanism.
However, based on mloiseleur suggestion, I revised the implementation to create a temporary data structure that exists only between the Records and ApplyChanges phases.
This structure functions similarly to a filter: it identifies and excludes any TXT records that already exist, preventing their re-creation. Once ApplyChanges is executed, this temporary structure is cleaned up.

The primary issue I'm addressing is ensuring that ExternalDNS can recreate an A record if it was accidentally deleted. Currently, when attempting to create an A record, ExternalDNS also tries to create a corresponding TXT record via the TXTRegistry.
If the TXT record already exists, providers like Route53 return an error, causing the entire operation to fail.

To resolve this, I implemented a mechanism to store existing TXT records during the Records phase. Before ApplyChanges attempts to create new records, it checks against this stored data to ensure it doesn't attempt to recreate existing TXT records, thereby avoiding errors.

Regarding recordsCache, I understand now that it's intended for non-TXT records and is scoped differently than my current change.
In contrast, my approach is limited to TXT records only, and the snapshot is used strictly within a single reconciliation loop to avoid stale state.
So while the naming may appear similar, the intention and scope are different.

I’ve pushed the updated implementation based on this approach.

Please let me know if I misunderstood any part of your suggestion — happy to revise if needed!

[u-kai]

Hi @mloiseleur, I mentioned earlier that I would wait for your confirmation before pushing the changes, but I received a detailed comment from @ivankatliarchuk and pushed the revised implementation to help clarify the current behavior.

The latest code uses a temporary structure between Records and ApplyChanges instead of caching across reconciliation loops, as previously discussed.

Let me know if this direction still looks good to you — I’m happy to adjust if needed!

[ivankatliarchuk]

Have you managed to validate that issue is still present or it's related to specific flags

• upsert-only
• sync
  ?

[ivankatliarchuk]

relates: #3977

[u-kai]

@ivankatliarchuk

Yes, I was able to reproduce the issue with both --policy=sync and --policy=upsert-only.
For clarity, I'm using hoge.com here as a placeholder — the actual domain has been replaced.

Environment

• branch: master
• command:

   external-dns \
  --provider aws \
  --registry txt \
  --txt-owner-id demo \
  --policy sync \ # and upsert-only
  --log-level debug \
  --source ingress
  

• ingress resource

  apiVersion: networking.k8s.io/v1
   kind: Ingress
  metadata:
    name: demo-wild
    annotations:
      external-dns.alpha.kubernetes.io/aws-weight: "100"
      external-dns.alpha.kubernetes.io/hostname: api.hoge.com
      external-dns.alpha.kubernetes.io/set-identifier: demo
  spec:
    rules:
      - host: "api.hoge.com"
        http:
          paths:
            - path: /
              pathType: Prefix
              backend:
                service:
                  name: demo-service
                  port:
                    number: 80

Steps to Reproduce

1. Deploy ExternalDNS.
2. Deploy the above Ingress and let ExternalDNS create the records.
3. Manually delete the A record for api.hoge.com from Route53.

Wait for the next reconciliation loop.

Result

On the next loop, ExternalDNS tries to recreate the A record and the associated TXT record.
However, since the TXT record still exists, Route 53 throws an error due to the attempted re-creation of an already-existing record.

DEBU[0001] Adding api.hoge.com. to zone hoge.com. [Id: /hostedzone/ZONEIDXXXXXXXXX]
DEBU[0001] Adding api.hoge.com. to zone hoge.com. [Id: /hostedzone/ZONEIDXXXXXXXXX]
DEBU[0001] Adding a-api.hoge.com. to zone hoge.com. [Id: /hostedzone/ZONEIDXXXXXXXXX]
INFO[0001] Desired change: CREATE a-api.hoge.com TXT  profile=default zoneID=/hostedzone/ZONEIDXXXXXXXXX zoneName=hoge.com.
INFO[0001] Desired change: CREATE api.hoge.com A   profile=default zoneID=/hostedzone/ZONEIDXXXXXXXXX zoneName=hoge.com.
INFO[0001] Desired change: CREATE api.hoge.com TXT  profile=default zoneID=/hostedzone/ZONEIDXXXXXXXXX zoneName=hoge.com.
...
DEBU[0061] Adding api.hoge.com. to zone hoge.com. [Id: /hostedzone/ZONEIDXXXXXXXXX]
DEBU[0061] Adding api.hoge.com. to zone hoge.com. [Id: /hostedzone/ZONEIDXXXXXXXXX]
DEBU[0061] Adding a-api.hoge.com. to zone hoge.com. [Id: /hostedzone/ZONEIDXXXXXXXXX]
INFO[0061] Desired change: CREATE a-api.hoge.com TXT  profile=default zoneID=/hostedzone/ZONEIDXXXXXXXXX zoneName=hoge.com.
INFO[0061] Desired change: CREATE api.hoge.com A   profile=default zoneID=/hostedzone/ZONEIDXXXXXXXXX zoneName=hoge.com.
INFO[0061] Desired change: CREATE api.hoge.com TXT  profile=default zoneID=/hostedzone/ZONEIDXXXXXXXXX zoneName=hoge.com.
ERRO[0061] Failure in zone hoge.com. when submitting change batch: operation error Route 53: ChangeResourceRecordSets, https response error StatusCode: 400, RequestID: 08d2eafe-c886-46d3-bd49-c5f905a7d3ee, InvalidChangeBatch: [Tried to create resource record set [name='a-api.hoge.com.', type='TXT', set-identifier='demo'] but it already exists, Tried to create resource record set [name='api.hoge.com.', type='TXT', set-identifier='demo'] but it already exists]  profile=default zoneID=/hostedzone/ZONEIDXXXXXXXXX zoneName=hoge.com.
ERRO[0062] Failed to do run once: soft error
failed to submit all changes for the following zones: [/hostedzone/ZONEIDXXXXXXXXX]

[u-kai]

@ivankatliarchuk
Thanks a lot for the review and helpful suggestion!
I've reflected the changes accordingly.

I'm currently reviewing the test cases and will follow up once everything is ready.

[ivankatliarchuk]

I do not understand this logic behind the solution.
Why we could not have something simple like?

for _, r := range filteredChanges.Create {
		if im.existingTXTs.is(not)Managed(r) {
			continue
		}
		if r.Labels == nil {
			r.Labels = make(map[string]string)
		}
		r.Labels[endpoint.OwnerLabelKey] = im.ownerID

		if im.cacheInterval > 0 {
			im.addToCache(r)
		}
	}

[u-kai]

Thanks for the suggestion!

The reason the logic is structured this way is because filteredChanges.Create does not initially contain TXT records.
The TXT records are generated on the fly using im.generateTXTRecord(r), and those are what I intend to filter using existingTXTs.

Checking r directly (as in your example) wouldn’t work in this case, since it’s typically an A or CNAME record, not the TXT itself.
Let me know if I misunderstood your suggestion!

[ivankatliarchuk]

Current implementation seems quite expensive in terms for computation.

It should be possible to pass any record to im.existingTXTs.is(not)Managed and validate it somehow.

[u-kai]

Just pushed a refactor — the generateTXTRecordWithFilter part is a bit more involved, but the main loop is now cleaner and should be more efficient overall.
Open to feedback, let me know what you think!

[ivankatliarchuk]

rest seems legit to me

We just w8 for @szuecs view

[ivankatliarchuk]

function not fully covered

[ivankatliarchuk]

But agree, we need to be super careful, as pretty much every deployment rely on txt registry

[szuecs]

However, if the goal is to stop managing certain records, wouldn't it be more appropriate to configure domainFilter or similar mechanisms? What do you think about that approach?

No that shouldn't be the tool.
Domainfilter should be used if you have zone .foo.example and suppose you have 2 automations that each manage one subdomain:

• .bar.foo.example
• .qux.foo.example

Txt registry is the fundamental ownership record such that you can have multiple clusters with multiple external-dns and whatever for example OpenTofu to manage a single zone.
If we rewrite txt records we should do this only in a script or run-once in order to hand over records from one external-dns to another one or rewrite prefix/suffix patterns to be able to change the design of your infrastructure.

[u-kai]

@szuecs
Thanks for the clarification — I now understand the role of the TXT record and that it shouldn't be recreated.
I've made the changes as you suggested. I'd appreciate it if you could take another look.

[szuecs]

Some non critical changes I would like to see.

Thanks for your work!

[szuecs]

ok

[szuecs]

I see but I think it's more confusing to have this comment.
I also see that how it's integrated we store only TXT records. This filters other records before *existingTXTs.add() is called https://github.com/kubernetes-sigs/external-dns/pull/5459/files#diff-353145d31754c56c5c08d53f4a5b1ee4cf2116cd605bbb9a7bc572c9072b2ab8R192-R194

I think in general the function name should be isAbsent or isNotKnown or isUnknown or isNotCached. We can think of existingTXTs.entries as an in-memory cache/store.
It's used as im.generateTXTRecordWithFilter(r, im.existingTXTs.isNotManaged)

im.generateTXTRecordWithFilter(r, im.existingTXTs.isAbsent)
im.generateTXTRecordWithFilter(r, im.existingTXTs.isNotKnown)
im.generateTXTRecordWithFilter(r, im.existingTXTs.isUnknown)
im.generateTXTRecordWithFilter(r, im.existingTXTs.isNotCached)

For me im.generateTXTRecordWithFilter(r, im.existingTXTs.isUnknown) reads best.
wdyt?

[u-kai]

@szuecs
Thank you for the review. I've applied the requested changes. If there are no problems, I'd appreciate your approval. Please take a look when you have a moment.

[szuecs]

/approve

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: szuecs

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• OWNERS [szuecs]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[ivankatliarchuk]

/lgtm

[ivankatliarchuk]

/hold cancel