Automation of PRIVATE section removals

[dnsguru]

This project has evolved a fairly strong ingress process for adding entries.

Once folks get their entry/ies added, it is typically a set and forget activity until they want to amend or modify them.

Few if any submitters come back to remove their entries once set or functioning.

We have put in place a requirement on new submissions to the PRIVATE section that 2+ years of service before expiry are required, in anticipation of some form of culling automation using the expiry date being a data point used to determine the domain is still 'on the air'. Another data point might be the ongoing presence of a _psl txt record where possible.

Without being prescriptive of the sensors and triggers for removals, there seems a need to have some automated deletion process that can scan and use some agreed logic for parsing and removing entries.

Long version:
The PSL gets adds and updates, but typically few or no deletions. One exception is the ICANN contracting json update automation for adding and removing TLDs in the ICANN section of the file, but an area that is growing in size which would benefit from attention is the PRIVATE section.

The file size has grown, and continues to grow while the PSL is leveraged as a fast-fail on inclusion or other derivative use cases that seem to require a PSL entry to get some special handling or status.

The volunteers suspect that there are numerous domains that are in the list which have not been renewed or have expired and been re-registered/drop-caught or have been acquired in the secondary market prior to their expiry by third parties.

This project is staffed by volunteers, all of whom are not looking to have more volunteer time extracted to deal with more than the current process of adds/modifies, which has been greatly aided by automation with the test scripts and the root zone deltas.

Would like to discuss.

Is there consensus that we need deletion automation within the PRIVATE section, and:

• frequency
• sensors/trigger conditions
• overrides
• phases

[dnsguru]

So far, there are a number of ccTLDs that do not present the expiration date in automated lookups.

Whatever automation might have to look to those domains differently for validation of existence and frequency of flagging/purges

[dnsguru]

I am going to run a manual sweep on the private section to determine expired domains for removal using RDAP response expiry dates on gTLDs, and will report the results in aggregate here.

[fsodic]

@dnsguru Is this a discussion? if so, there may be better solutions for forming a healthier "private section". Like looking for tools that can "dig" _psl automatically for domains in this private section.

[dnsguru]

Fajar it is absolutely a discussion about how to get this done - and appreciate your input! Lookups for the presence of a _psl txt record that persists could be a strong way to do this review faster... Because the existence or non-existence of the domain in the registry does not take into consideration that it is still registered by the same submitting party or even registrant, or is even connected in DNS in the same manner or intent that was part of the PR or rationale for inclusion into the PSL. Doing this in DNS would work well, actually. Most folks seem to move on to whatever else they were doing after requesting inclusion, and the PSL entry becomes 'set and forget'. Because of the validation step where a _psl txt record gets added in order to qualify an entry, one could reckon that just leaving that in place is simpler, as it takes another step to remove the entry from DNS. Then, we leave the entry while the _psl txt record remains. Interesting to consider.

…

[fsodic]

It might be a problem when the domain does not have a psl for reasons that a dns error coincides with it and dig might detect the absence of the dns as a sign the domain is inactive.
So a better solution is to add 2 or 3 files that hold dns data from an existing domain separated into 3 different levels.
Orange or Yellow: for a light warning flag
Red: for intermediate deletion warnings
Black: for the last warning
and the last crawl is permanent deletion.

and an additional solution is with the psl which is scheduled for 1 week so that there is a 1 month chance to fix their dns psl.

Another part is when the domain falls into each category there may be a reminder email to the sender of the psl that they include in the registration of the person in charge.

[fsodic]

It might be a problem when the domain does not have a psl for reasons that a dns error coincides with it and dig might detect the absence of the dns as a sign the domain is inactive.
So a better solution is to add 2 or 3 files that hold dns data from an existing domain separated into 3 different levels.
Orange or Yellow: for a light warning flag
Red: for intermediate deletion warnings
Black: for the last warning
and the last crawl is permanent deletion.

and an additional solution is with the psl which is scheduled for 1 week so that there is a 1 month chance to fix their dns psl.

Another part is when the domain falls into each category there may be a reminder email to the sender of the psl that they include in the registration of the person in charge.

And the auto removal <1 year isn't need anymore when the PSL still active.
Every approve merger, make sure the submitter must keep the psl in their dns domain.

i'm sure our PSL be better with this system model.

[frknakk]

We have put in place a requirement on new submissions to the PRIVATE section that 2+ years of service before expiry are required

I ran into a few problems with this requirement that I'd like to mention here:

• There are TLD's where the expiry date is not publicly retrievable (e.g. german .de domains).
• Providers that are paid by invoice often don't have an option to specify a domain expiration date. The domain is simply extended every year by 1 year until the contract gets cancelled by the customer.

Of course you can avoid both problems by switching to other TLD's and other providers. I just wanted to mention that it could restrict many people :)

[dnsguru]

We have seen a number of TLDs that do not provide the expiration date in the public whois/rdap. In addition to .de, I have seen this to be the case with .je/.gg, .ru, .nl, and .pl as well, and I am certain there are others we will find. The invoice payment scenario, as well as others that we have not heard of, whereby the 2 year + term may not be possible will certainly exist. We can make a catch-all, but it doesn't allow for automation to really work... The solution seems to be that the submitting party just acknowledges that their entries can be subject to removal if not maintained through committing to keep the domain name(s) submitted renewed in their PR.

…

On Wed, Dec 16, 2020 at 6:21 PM Furkan A. ***@***.***> wrote: We have put in place a requirement on new submissions to the PRIVATE section that 2+ years of service before expiry are required I ran into a few problems with this requirement that I'd like to mention here: - There are TLD's where the expiry date is not publicly retrievable (e.g. german .de domains). - Providers that are paid by invoice often don't have an option to specify a domain expiration date. The domain is simply extended every year by 1 year until the contract gets cancelled by the customer. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1119 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AACQTJK4SVPNCHP4X5WGNBTSVFTL5ANCNFSM4SJM3HNA> .

[pereceh]

I think the requirements on the wiki are pretty good.

just use the dns _psl records as a benchmark for mass deletion... then give time if the user's DNS server changes to re-create those _psl DNS records (using their old values)...

[wdhdev]

@simon-friedberger I think we can close this, most of this stuff will have to be done manually (it will likely be quite difficult to automate, which we don't have the resources for) and most clean ups are already being done by @groundcat and I.

[simon-friedberger]

I generally agree. The reason I left this open is that I have a concrete idea of something I would like us to do:

Take the psltool output - which can already check the entire list - and turn it into a diff/changelog.

Something that shows us: "last week these domains dropped their _psl entry". I think that would be useful information to have for the manual work and it is transient information that is not captured anywhere. And it is not particularly hard to do.

1. Make psltool output something that can easily be diffed and parsed.
2. Make a github action that runs once a day and stores the output somewhere (probably in the website repository)
3. Make a github action that runs once a day and takes the last two outputs and turns them into something like

Domain	_psl present	_psl entry changes
example.com	Yes	Removed 2024-11-27, Added 2024-11-28
...

Because if we ever want to implement a policy like "if the _psl record is missing for a month we remove your entry" we will need this.