Ordering of UTIL autopull logic needs a tweak to let IANA list be dominant #1325

dnsguru · 2021-05-19T03:17:34Z

@cpu can we revisit the AUTOPULL Bot?

There is a modification needed - I was investimagating the issue in #1176 and think that we need a review and change in the logic for what makes the autopull so that if there is something in the IANA TLD list (which is basically a list of the root TLDs) but not in the autopull JSON that is sourced from the contracting / cancels NOR in the greater PSL, that the TLD gets added/included.

Example: .WED - in IANA list and active TLD, but not in PSL
.WED was a TLD that was picked up and now operates under EBERO, which is a program at ICANN that has an assigned provider continue operations for TLDs - helping keep TLDs operational when they close due to whatever circumstance.

The TLD appears programatically in the JSON as the other cancelled TLDs, but the TLD still remains in the root zone.

By letting the IANA list be the override, it would catch intances of new IDN TLDs for ccTLDs and include them, plus any fresh ccTLD Delegations if they occur.

These may or may not have the typical meta information that nTLDs do, but inclusion vs not having them is preferred. These could be included in-line with the alphabetized nTLDs, or aggregated at the end of the nTLDs in a small section of their own inside the #ICANN section.

cpu · 2021-05-19T12:54:29Z

@cpu can we revisit the AUTOPULL Bot?

👋 @dnsguru Hey there - of course.

What you're proposing sounds reasonable to me. My memory on the specifics is a little vague but I believe the original version of the bot tooling that I made for github.com/zmap/zlint already looks at the tlds-alpha-by-domain.txt file you linked and uses it to augment the gTLD JSON. It shouldn't be too bad to adopt more of this logic tailored for the PSL.

Is there a timeline you'd like to see these adjustments made by? It will take me a little bit of time to page all of these details back into memory and I'm not sure I'll have a chance before the weekend at the earliest.

dnsguru · 2021-05-19T15:28:06Z

@cpu this is important but not urgent - I can invent a deadline if it helps motivation or bias for action.

Separate area of attention, while you'd be under the hood: Testing autopull in presence of "Stubs"
During the time when you were updating the UTIL to autopull, There had been a request #1159 for some stub zones (akin to what we see inside of ccTLDs) from another request that I'd deferred the requestor to place in the PRIVATE section which I would have liked to have seen the impact of if those were placed near their entry in the automation included section.

Typically we would treat these like .AERO or .PRO's registry-associated-managed stubs akin to ccTLDs as mentioned, but due to the timing of the request, I'd asked them to put those in the PRIVATE section, but really because .РУС was a 2012 round nTLD, it is in the section that is generated by the autopull.

Can we, as part of testing the iana txt in the current issue you are reading, test registry associated stubs?

cpu · 2021-05-20T21:42:52Z

@cpu this is important but not urgent - I can invent a deadline if it helps motivation or bias for action.

I will see what I can do 👍

Separate area of attention, while you'd be under the hood: Testing autopull in presence of "Stubs"
...
Can we, as part of testing the iana txt in the current issue you are reading, test registry associated stubs?

Would you mind filing a separate issue for that and tagging me (or assigning if possible)? I'm open to the idea but I don't know that I'll have time to do both at once this week.

dnsguru · 2021-05-21T02:41:11Z

Will do - lets focus on this one for now, given constraints. The help is appreciated!

…

On Thu, May 20, 2021 at 2:43 PM Daniel McCarney ***@***.***> wrote: @cpu <https://github.com/cpu> this is important but not urgent - I can invent a deadline if it helps motivation or bias for action. I will see what I can do 👍 Separate area of attention, while you'd be under the hood: Testing autopull in presence of "Stubs" ... Can we, as part of testing the iana txt in the current issue you are reading, test registry associated stubs? Would you mind filing a separate issue for that and tagging me (or assigning if possible)? I'm open to the idea but I don't know that I'll have time to do both at once this week. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1325 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AACQTJOEV2VO6IQFFQDL2VLTOV665ANCNFSM45DXEKCA> .

cpu · 2021-06-02T12:57:21Z

Haven't forgotten about this, just overestimated my ability to be at a computer after hours in the summer :-) Hoping to get going soon but also happy to hand this off if someone else is ready to take it on immediately, just shout so we don't duplicate work.

dnsguru · 2021-06-02T16:51:35Z

No worries - that was more housekeeping activity than intended as a nudge your help is very much appreciated with the project!

…

On Wed, Jun 2, 2021 at 5:57 AM Daniel McCarney ***@***.***> wrote: Haven't forgotten about this, just overestimated my ability to be at a computer after hours in the summer :-) Hoping to get going soon but also happy to hand this off if someone else is ready to take it on immediately, just shout so we don't duplicate work. — You are receiving this because you were assigned. Reply to this email directly, view it on GitHub <#1325 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AACQTJIMW7I7JSDHDLYGJ23TQYTD7ANCNFSM45DXEKCA> .

Only change is to add a URL to the IANA TLD txt file and some comment lines to add context. Doesn't solve #1325 but helps identify the resource to use.

dnsguru · 2021-06-06T04:58:23Z

@cpu I made PR #1347 for the IANA txt file - I think I oversimplified the matter ...

Looks like there would be some extra complexity for the following reasons:

IANA TXT is all upper case for TLDs (solveable by normalizing case)

IANA TXT WOULD include, but ICANN JSON WOULD NOT include:

EBERO TLDs still in the root but listed in the JSON as cancelled
New ccTLD entries to the root
New IDN ccTLD entries to the root

Some of the challenges that come with this are:

The IANA TXT names will not have the other information that would typically accompany the nTLDs.
The IANA TXT IDNs are contained as the A-Label (Punycode) vs using the U-Label (raw UTF8) - the code will need to be smart about testing these against the commented A-Label for matches or perform the IDNA conversions or there will be false positives.

Without this going full "Matryoshka" and expanding the scope too widely on the request to add the IANA TLDs which might be missing or impacting the ICANN section beyond the nTLD items that this autopull process was designed for (re: ccTLDs/IDNccTLDs).

ICANN have recently announced that they will auction .WED (the EBERO TLD that triggered this request). That might update it in the JSON (or not, this is new ground). It appears between that and the ccTLDs that might be missing, we might cross streams with the existing legacy stuff, which has all kinds of tech debt on reviewing the non-2012 round stuff that this update would change.

At the core of the request here, we want to list names that:

ARE NOT in the last public_suffix_list.dat; and,
ARE in the IANA TXT (Root zone, simplified); and,
ARE not present, or are listed as cancelled in the ICANN JSON (gTLDs under contract, may be superset of IANA TXT)

For these,
I propose we add a small section. This could be either just above the newGTLDs header or just after the above the end of the ICANN section, which ever is easier to output based upon the process)

(so just above either // newGTLDs or // ===END ICANN DOMAINS=== line)

I'll make up a called the Jothan Frakes Islands (.JF) and let's for example imagine it has an IDN also of XN--EXAMPLEJF, and include the .WED in the example below. (On the "IDNccTLD" let's set aside the A-Label vs U-Label challenge and just list it as the A-Label)

New Section:

// Additional IANA TLDs
// The following were included by automated review of the IANA TLDs LIST
// while pricessing the ICANN JSON File of contracted gTLDs.
// Recently added ccTLD or IDN ccTLDs, EBERO TLDs or other TLDs from
// IANA's PSL https://data.iana.org/TLD/tlds-alpha-by-domain.txt
// will appear here for inclusion until relocated within the dat file. 

// jf
jf

// xn--examplejf
xn--examplejf

// wed
wed

This expanded the scope a little but also hopefully shrunk it while increasing the coverage.

I am also not opposed to this being a separate automation, but thought it best to keep this in the same tool because it represents a minimal element of additional processing.

Hope this is helpful, and @cpu thank you ... and hit me up with any questions or clarifications.

cpu · 2021-06-06T18:15:25Z

@dnsguru Thanks, your last comment is very helpful. I've started working on this. I think it'll take a little bit of restructuring but should be achievable.

One immediate question, when you called out the core of the request you said:

At the core of the request here, we want to list names that:

ARE NOT in the last public_suffix_list.dat; and,
ARE in the IANA TXT (Root zone, simplified); and,
ARE not present, or are listed as cancelled in the ICANN JSON (gTLDs under contract, may be superset of IANA TXT)

For that last point when you say "cancelled in the ICANN JSON", do you mean entries with "contractTerminated" : true, or something else?

dnsguru · 2021-06-06T18:39:04Z

Made this to illstrate the various cases here, hopefully it adds clarity vs being more confusing.

	Delegated TLD (g or cc)	Contracted, PreDelegation	Cancelled, Delegated	Cancelled, PreDelegation	IETF/Special	Delegated ccTLD	Delegated IDN ccTLD
Entry in last PSL	YES	EITHER, See Note	YES, See Note	YES, See Note	EITHER, See Note	NO	NO
Entry in ICANN JSON	YES	YES	YES	YES	EITHER	N/A	N/A
JSON : `"contractTerminated" : true`	No	No	YES	YES	NO	N/A	N/A
In IANA TLD TXT	YES		YES	No	YES	YES	YES

Notes	Status Quo	If missing, Add Entry	Currently: Remove from PSL; Revise to: Leave in PSL, change commented line	Remove from PSL (Status Quo)	If missing, Add Entry	If missing, Add Entry	If missing, Add Entry - Need to convert from punycode to UTF-8 and add that

Comment Line Change:			`// [tld] : [Date Cancelled]; EBERO TLD Active`		`// [tld] : [run date]`	`// [tld] : [run date]`	`// [A-tld] : [run date]` (note [A-tld] is the A-Label/Punycode)

cpu · 2021-06-06T18:53:51Z

Thanks, let me noodle with the code and see where I end up.

dnsguru · 2021-06-06T20:42:03Z

For that last point when you say "cancelled in the ICANN JSON", do you mean entries with "contractTerminated" : true, or something else?

Yes, the EBERO handler (currently only one, .wed) should catch this case where a TLD is in the IANA TLD TXT but shows in the JSON file as "contractTerminated" : true and is getting removed by that last part. I updated the grid in the comment above to help be more clear

cpu · 2021-06-06T21:00:12Z

I updated the grid in the comment above to help be more clear

Great, ty!

Here are a few updates from my side after a bit of hacking.

I think it's time to make a proper Go module for this work instead of a stand-alone .go file with a main. The primary reason is that the script now has a need for an external dependency. Specifically we need golang.org/x/net/idna for converting the a-labels to unicode. Overall I think the Go ecosystem has been trying to push people away from the more adhoc go run somefile.go type usage to modules so this feels somewhat inevitable.
On that note, I think it's worth adding another dependency to this tools module above the IDNA dep. I think it makes sense to pull in github.com/weppos/publicsuffix-go. Previously the script didn't need to parse the .dat file and I was able to get away with cutting out the gTLD section based on section delimiters and then just checking if a freshly templated rendering of that one section resulted in a diff or not. With this new logic we need to be able to determine if an entry is present in another section. Rather than invent another Go PSL DAT parser I'd rather use the LoadString interface from Weppos' lib here and then query the list that way. I don't think it will be an issue to depend on his lib for .dat updates while the lib depends on this repo for its built-in .dat.
Cancelled gTLD comments:

// [tld] : [Date Cancelled]; EBERO TLD Active

There's unfortunately no date field corresponding to the contractTerminated field in the JSON as far as I can tell. I don't think we can include that in a comment, just that it has been cancelled.

ccTLD delegation comments:

// [tld] : [run date]

As you noticed, the IANA tlds-alpha-by-domain.txt data source has no metadata (including the run date). I think the only comment we can add is with the original a-label form of the TLD if we rendered the unicode version into the .dat.

Overall this is a pretty significant lift so the diff isn't going to be a little tweak of the script as much as an overhaul. I'm happy to keep plugging at it but I wanted to make sure to set expectations in terms of the volume of code that will need to be reviewed.

dnsguru · 2021-06-07T04:09:46Z

Ok.. What could we trim back to MVP this in order to get the basic part? The pulling the iana txt file in and lowercasing it seems simple, but assessing what is easy or hard from that point is not obvious to me. I will read the code closer this eve and see if I can pivot the request.

…1347) * newgtlds.go : update Autopull - Added URL for the IANA PSL for #1325 Only change is to add a URL to the IANA TLD txt file and some comment lines to add context. Doesn't solve #1325 but helps identify the resource to use. * Update tools/newgtlds.go Co-authored-by: Daniel McCarney <[email protected]> Co-authored-by: Daniel McCarney <[email protected]>

dnsguru · 2021-06-07T08:58:45Z

Was autoclosed by #1347 ; re-opening

dnsguru · 2022-08-29T07:11:47Z

@cpu could this get a revisit

cpu · 2022-08-30T16:14:45Z

@dnsguru Sorry, I'm not available to commit to this work right now.

weppos · 2022-09-09T09:53:33Z

@dnsguru I need some time to look into the conversation and parse all the comments. As the logic is in Go, I may be able to work on it.

Perhaps it could be beneficial if you and I have a quick chat, so that you help me with a short gist of the required changes, without me having to go through the whole conversation and construct it. That will probably help me to speed up the requirement gathering phase, and jump straight to the implementation phase.

dnsguru · 2022-09-09T23:16:35Z

@weppos Thank you -this would be appreciated - can zoom or meet online at a time that works well between UTC-7 and your local time in EU. ICANN 75 is imminent as is travel and most cycles are going into preparatory work for that and responsibilities in #dayjob / #icann policy prep WIll you be in Kuala Lumpur for ICANN 75? Would be good to whiteboard this face to face. Otherwise, perhaps after I return from KL closer to the end of the month. I hope to speak with IANA and OCTO about how this issue with Israel's ccTLD IDN happened (#1595) and how we all can better avoid the matter in the future via automation and communication, collaboratively.

…

On Fri, Sep 9, 2022 at 2:53 AM Simone Carletti ***@***.***> wrote: @dnsguru <https://github.com/dnsguru> I need some time to look into the conversation and parse all the comments. As the logic is in Go, I may be able to work on it. Perhaps it could be beneficial if you and I have a quick chat, so that you help me with a short gist of the required changes, without me having to go through the whole conversation and construct it. That will probably help me to speed up the requirement gathering phase, and jump straight to the implementation phase. — Reply to this email directly, view it on GitHub <#1325 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AACQTJKAZNC3FJUCANNPRRDV5MCKRANCNFSM45DXEKCA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

wdhdev · 2024-11-29T07:49:40Z

@danderson Would you be able to do this, if it hasn't been done already?

dnsguru mentioned this issue May 19, 2021

Add .wed #1176

Closed

5 tasks

dnsguru changed the title ~~Ordering of UTL autopull logic needs a tweak to let IANA list be dominant~~ Ordering of UITL autopull logic needs a tweak to let IANA list be dominant May 19, 2021

dnsguru changed the title ~~Ordering of UITL autopull logic needs a tweak to let IANA list be dominant~~ Ordering of UTIL autopull logic needs a tweak to let IANA list be dominant May 19, 2021

dnsguru assigned dnsguru and cpu Jun 2, 2021

dnsguru added a commit that referenced this issue Jun 6, 2021

newgtlds.go : update Autopull - Added URL for the IANA PSL for #1325

01dd7d7

Only change is to add a URL to the IANA TLD txt file and some comment lines to add context. Doesn't solve #1325 but helps identify the resource to use.

dnsguru mentioned this issue Jun 6, 2021

newgtlds.go : update Autopull - Added URL for the IANA PSL for #1325 #1347

Merged

dnsguru linked a pull request Jun 6, 2021 that will close this issue

newgtlds.go : update Autopull - Added URL for the IANA PSL for #1325 #1347

Merged

dnsguru closed this as completed in #1347 Jun 7, 2021

dnsguru reopened this Jun 7, 2021

dnsguru mentioned this issue Aug 12, 2021

Missing TLDs #1394

Closed

cpu mentioned this issue Jul 25, 2023

Update Autopull from ICANN JSON to address contract renewal noise #1806

Closed

cpu mentioned this issue Nov 18, 2023

Interest in using go modules #1897

Closed

aaomidi mentioned this issue Nov 20, 2023

Introduce Go Modules to tooling #1901

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Ordering of UTIL autopull logic needs a tweak to let IANA list be dominant #1325

Ordering of UTIL autopull logic needs a tweak to let IANA list be dominant #1325

dnsguru commented May 19, 2021 •

edited

Loading

cpu commented May 19, 2021

dnsguru commented May 19, 2021

cpu commented May 20, 2021

dnsguru commented May 21, 2021 via email

cpu commented Jun 2, 2021

dnsguru commented Jun 2, 2021 via email

dnsguru commented Jun 6, 2021

cpu commented Jun 6, 2021

dnsguru commented Jun 6, 2021 •

edited

Loading

cpu commented Jun 6, 2021

dnsguru commented Jun 6, 2021 via email •

edited

Loading

cpu commented Jun 6, 2021 •

edited

Loading

dnsguru commented Jun 7, 2021 via email

dnsguru commented Jun 7, 2021

dnsguru commented Aug 29, 2022

cpu commented Aug 30, 2022

weppos commented Sep 9, 2022

dnsguru commented Sep 9, 2022 via email

wdhdev commented Nov 29, 2024

Ordering of UTIL autopull logic needs a tweak to let IANA list be dominant #1325

Ordering of UTIL autopull logic needs a tweak to let IANA list be dominant #1325

Comments

dnsguru commented May 19, 2021 • edited Loading

cpu commented May 19, 2021

dnsguru commented May 19, 2021

cpu commented May 20, 2021

dnsguru commented May 21, 2021 via email

cpu commented Jun 2, 2021

dnsguru commented Jun 2, 2021 via email

dnsguru commented Jun 6, 2021

cpu commented Jun 6, 2021

dnsguru commented Jun 6, 2021 • edited Loading

cpu commented Jun 6, 2021

dnsguru commented Jun 6, 2021 via email • edited Loading

cpu commented Jun 6, 2021 • edited Loading

dnsguru commented Jun 7, 2021 via email

dnsguru commented Jun 7, 2021

dnsguru commented Aug 29, 2022

cpu commented Aug 30, 2022

weppos commented Sep 9, 2022

dnsguru commented Sep 9, 2022 via email

wdhdev commented Nov 29, 2024

dnsguru commented May 19, 2021 •

edited

Loading

dnsguru commented Jun 6, 2021 •

edited

Loading

dnsguru commented Jun 6, 2021 via email •

edited

Loading

cpu commented Jun 6, 2021 •

edited

Loading