Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve 451 detection and www suffix addition #83

Closed
spirillen opened this issue Dec 9, 2024 · 4 comments
Closed

Improve 451 detection and www suffix addition #83

spirillen opened this issue Dec 9, 2024 · 4 comments

Comments

@spirillen
Copy link

spirillen commented Dec 9, 2024

Hi, after looking through vercel.app, I've found you have a bunch of FP's on your list,

You can see the list of active domains here: https://kb.mypdns.org/issue/MTX-1402 and a PyFunceble related issue for how to clean these up automatically here: funilrys/PyFunceble#407

PS: You should never blindly add the www. suffix to all records

@spirillen
Copy link
Author

spirillen commented Dec 9, 2024

Search result from External Hosts-Sources

@mypdns's External Hosts-Sources can be found here

data/ShadowWhispererTracking.csv:ip-geo.vercel.app
data/ShadowWhispererTracking.csv:umami-nine-olive.vercel.app
data/Ultimate.Hosts.Blacklist2.csv:tarnow7.vercel.app
data/digitalside.csv:microsoftcomdefenderupdate.vercel.app
data/kadhosts_azet12.csv:administrationprocess.vercel.app
data/kadhosts_azet12.csv:app-authorization.vercel.app
data/kadhosts_azet12.csv:auth0-phi-five.vercel.app
data/kadhosts_azet12.csv:beufrevuxoissi.vercel.app
data/kadhosts_azet12.csv:bnp-sinistre-billeterie-qzevatnq0-neat-eu.vercel.app
data/kadhosts_azet12.csv:bouyauriceufre-7037.vercel.app
data/kadhosts_azet12.csv:copyrightclaimactionreportviolation.vercel.app
data/kadhosts_azet12.csv:craparetauzei.vercel.app
data/kadhosts_azet12.csv:cruquidemeya-8567.vercel.app
data/kadhosts_azet12.csv:dappettubouya-6270.vercel.app
data/kadhosts_azet12.csv:data-management-hun4jc.vercel.app
data/kadhosts_azet12.csv:dichvuchuyentienj7ss.vercel.app
data/kadhosts_azet12.csv:disputedcontentpolicy.vercel.app
data/kadhosts_azet12.csv:donuti-1601.vercel.app
data/kadhosts_azet12.csv:download-wetransferr-access-login-token-magi-89iuj3wesd9iuj.vercel.app
data/kadhosts_azet12.csv:ebq-gules.vercel.app
data/kadhosts_azet12.csv:etiv-tcaer.vercel.app
data/kadhosts_azet12.csv:facebook-vt.vercel.app
data/kadhosts_azet12.csv:famous-banger-blow2tb.vercel.app
data/kadhosts_azet12.csv:geuquoffexadi-2234.vercel.app
data/kadhosts_azet12.csv:guidelineviolationuserreport.vercel.app
data/kadhosts_azet12.csv:hauyappasella-7452.vercel.app
data/kadhosts_azet12.csv:help-center-overview-form-10031030002333333.vercel.app
data/kadhosts_azet12.csv:help-for-business-pages-case-appeal-id-516.vercel.app
data/kadhosts_azet12.csv:helpcenter-for-business-page-caseid-535.vercel.app
data/kadhosts_azet12.csv:helpcenter-for-business-page-caseid-536.vercel.app
data/kadhosts_azet12.csv:helpcenter-for-business-page-caseid-541.vercel.app
data/kadhosts_azet12.csv:helpcenter-for-business-page-caseid-547.vercel.app
data/kadhosts_azet12.csv:helpcenter-for-business-page-caseid-549.vercel.app
data/kadhosts_azet12.csv:helpdesk-dupport-services-all.vercel.app
data/kadhosts_azet12.csv:heyegeroifoi-9006.vercel.app
data/kadhosts_azet12.csv:heyepparopre-6953.vercel.app
data/kadhosts_azet12.csv:instructions-video.vercel.app
data/kadhosts_azet12.csv:joituleffewau-2260.vercel.app
data/kadhosts_azet12.csv:kigiffomana-3601.vercel.app
data/kadhosts_azet12.csv:leadfusionquantumdata.vercel.app
data/kadhosts_azet12.csv:loctan675550.vercel.app
data/kadhosts_azet12.csv:manage-qualification-8usy78n148prm.vercel.app
data/kadhosts_azet12.csv:market-place-contact.vercel.app
data/kadhosts_azet12.csv:meta-mail-noreply.vercel.app
data/kadhosts_azet12.csv:metasupportform.vercel.app
data/kadhosts_azet12.csv:moffuraujoiffe-5082.vercel.app
data/kadhosts_azet12.csv:nhantienquocte247hs.vercel.app
data/kadhosts_azet12.csv:nottollutoibra-1997.vercel.app
data/kadhosts_azet12.csv:o2-pl.vercel.app
data/kadhosts_azet12.csv:onet-webmailx-ovhappx-panel.vercel.app
data/kadhosts_azet12.csv:protection-enable-notification.vercel.app
data/kadhosts_azet12.csv:quoijacejoilou-2223.vercel.app
data/kadhosts_azet12.csv:sataquautteura-8991.vercel.app

spirillen added a commit to mypdns/matrix that referenced this issue Dec 9, 2024
#MTX-1402 Closing as solved

Fix #1400

Rel:
- funilrys/PyFunceble#407
- funilrys/PyFunceble#408
- FiltersHeroes/KADhosts#83

---------

You can always be following My Privacy DNS at https://kb.mypdns.org/issues?u=1

Sponsor us by Donate to My Privacy DNS https://kb.mypdns.org/articles/MTX-A-3/DONATION
@krystian3w
Copy link
Contributor

krystian3w commented Dec 9, 2024

All "should be reported" to CERT Polska (based on duplicates finder from online tool):

https://hole.cert.pl/domains/v2/

Counts / Results

COUNT    | LINE
-----------------------------------------------------
       2 | administrationprocess.vercel.app
       2 | app-authorization.vercel.app
       2 | auth0-phi-five.vercel.app
       2 | beufrevuxoissi.vercel.app
       2 | bnp-sinistre-billeterie-qzevatnq0-neat-eu.vercel.app
       2 | bouyauriceufre-7037.vercel.app
       2 | copyrightclaimactionreportviolation.vercel.app
       2 | craparetauzei.vercel.app
       2 | cruquidemeya-8567.vercel.app
       2 | dappettubouya-6270.vercel.app
       2 | data-management-hun4jc.vercel.app
       2 | dichvuchuyentienj7ss.vercel.app
       2 | disputedcontentpolicy.vercel.app
       2 | donuti-1601.vercel.app
       2 | download-wetransferr-access-login-token-magi-89iuj3wesd9iuj.vercel.app
       2 | ebq-gules.vercel.app
       2 | etiv-tcaer.vercel.app
       2 | facebook-vt.vercel.app
       2 | famous-banger-blow2tb.vercel.app
       2 | geuquoffexadi-2234.vercel.app
       2 | guidelineviolationuserreport.vercel.app
       2 | hauyappasella-7452.vercel.app
       2 | help-center-overview-form-10031030002333333.vercel.app
       2 | help-for-business-pages-case-appeal-id-516.vercel.app
       2 | helpcenter-for-business-page-caseid-535.vercel.app
       2 | helpcenter-for-business-page-caseid-536.vercel.app
       2 | helpcenter-for-business-page-caseid-541.vercel.app
       2 | helpcenter-for-business-page-caseid-547.vercel.app
       2 | helpcenter-for-business-page-caseid-549.vercel.app
       2 | helpdesk-dupport-services-all.vercel.app
       2 | heyegeroifoi-9006.vercel.app
       2 | heyepparopre-6953.vercel.app
       2 | instructions-video.vercel.app
       2 | joituleffewau-2260.vercel.app
       2 | kigiffomana-3601.vercel.app
       2 | leadfusionquantumdata.vercel.app
       2 | loctan675550.vercel.app
       2 | manage-qualification-8usy78n148prm.vercel.app
       2 | market-place-contact.vercel.app
       2 | meta-mail-noreply.vercel.app
       2 | metasupportform.vercel.app
       2 | moffuraujoiffe-5082.vercel.app
       2 | nhantienquocte247hs.vercel.app
       2 | nottollutoibra-1997.vercel.app
       2 | o2-pl.vercel.app
       2 | onet-webmailx-ovhappx-panel.vercel.app
       2 | protection-enable-notification.vercel.app
       2 | quoijacejoilou-2223.vercel.app
       2 | sataquautteura-8991.vercel.app
-----------------------------------------------------
   44306 | TOTAL LINES
      49 | DISTINCT DUPLICATE LINES

without guarantine fast revert banish:

https://cert.pl/en/contact/
https://incydent.cert.pl/#!/lang=en


I myself do not associate whether I can put it in CERT_expired or rather CERT_skip (and periodically check if it has already disappeared from the CERT file):

https://github.com/FiltersHeroes/KAD/blob/master/exclusions/CERT_expired.txt
https://github.com/FiltersHeroes/KAD/blob/master/exclusions/CERT_skip.txt

it is possible that most of the results are in the file with hundreds of domains for manual verification:

https://github.com/FiltersHeroes/ScriptsPlayground/blob/master/expired-domains/KAD-unknown.txt
https://github.com/FiltersHeroes/ScriptsPlayground/blob/master/expired-domains/KADhosts-unknown.txt

of domain activity when our script and recommended PyFunceble does not satisfy us.


PS: You should never blindly add the www. suffix to all records

I'm not sure if hawkeye feels like reprogramming the www. add mechanism to see if there are any responses (redirects or some real html code not related with http error).

krystian3w added a commit to FiltersHeroes/KAD that referenced this issue Dec 9, 2024
krystian3w added a commit to FiltersHeroes/KAD that referenced this issue Dec 9, 2024
@spirillen
Copy link
Author

Hi @krystian3w Nice to hear from you 👋🏻

I didn't know you imported straight from https://hole.cert.pl/domains/v2/

But looking around on that domain, I found this line

domains are put on the List for a period of 6 months, if an entry is no longer on the List or there is information about its deletion, the domain should no longer be blocked

source: https://cert.pl/en/warning-list/

This might explain the slow removal from the source. But now we build the rule, then we'll see if @funilrys are merging it.

Thanks for your feedback

hawkeye116477 added a commit to FiltersHeroes/KAD that referenced this issue Dec 10, 2024
@krystian3w krystian3w changed the title FP: vercel.app Improve 451 detection and www suffix addition Dec 15, 2024
@hawkeye116477
Copy link
Contributor

hawkeye116477 commented Dec 16, 2024

I'm not sure if hawkeye feels like reprogramming the www. add mechanism to see if there are any responses (redirects or some real html code not related with http error).

That's right. I think that would take too much time to detect that.

You should never blindly add the www. suffix to all records

That can increase security, cuz websites can have www version and unfortunately hosts isn't blocking them unlike adblockers.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants