HTTP check gives false positives on some Cloudflare sites

[maxime-huyghe]

Hello, I was scanning a list of domains and checking every positive, and found some false positives.

Expected Behavior

Domains such as cgos.homair.com or cms.law should give negative results because they point to a live site.

Current Behavior

These domains give positive results because the HTTP body of their home page contains the string "Cloudflare", which is specified in the fingerprints.json file.

Possible Solution

Remove the "Cloudflare" fingerprint. This shouldn't impact any true positives.

Steps to Reproduce (for bugs)

1. echo "cms.law. 300 IN A 104.20.186.112" > dnszone-cms-law
2. SubdomainSleuth -resolver 1.1.1.1 -check http-fingerprint dnszone-cms-law
3. Open cms.law in your browser of choice. It should load correctly.
4. echo "cgos.homair.com. 146 IN A 104.26.1.192" > dnszone-cgos-homair-com
5. SubdomainSleuth -resolver 1.1.1.1 -check http-fingerprint dnszone-cgos-homair-com
6. Open cgos.homair.com in your browser. It should load correctly.

Should I create a pull request ?

[paulf69487623]

Hello,

Thanks for the report! Yeah, those are definitely false positives. The cgos.homair.com case would be fixed by removing the Cloudflare fingerprint. For the cms.law case, we should double check. It hits the Cloudflare fingerprint too, but it also has Ray ID which is part of another signature for CloudFlare. We should probably take a closer look at the fingerprints to see if we're detecting CloudFlare stuff effectively.

If you'd like to open a PR, that would be great!

Thanks,

Paul

[paulf69487623]

I just tested out the fingerprints a little bit, and it seems like the "error code: 1001" case covers dangling pointers to CloudFlare:

% curl http://bogus.fingerprint --connect-to bogus.fingerprint:80:cgos.homair.com:80
error code: 1001%
% curl http://bogus.fingerprint --connect-to bogus.fingerprint:80:cms.law:80
error code: 1001%

So I think we should be good to remove both the "Cloudflare" and "Cloudflare Ray ID" signatures.

Thanks,

Paul

[paulf69487623]

I was looking at https://support.cloudflare.com/hc/en-us/articles/360029779472-Troubleshooting-Cloudflare-1XXX-errors, and the "error code: 1001" and "error code: 1016" that we have look good.

We might want to also add "error code: 1014" and "error code: 1018". Thoughts?

Paul

[maxime-huyghe]

Hello and thanks for your answers,
Regarding cms.law, I can't seem to find the whole "Cloudflare Ray ID" string, but there is a "rayId" somewhere so it seems likely that the string could be found on some other Cloudflare-hosted domain.
I think "error code: 1018" is useful, but I'm not really sure that error 1014 really represents a dangling record.
It looks like Cloudflare won't let cross-user CNAMES resolve, and if their customers choose to allow it, this error won't be triggered anymore.

[paulf69487623]

Hello,

If you're still up for creating a PR to update the fingerprints, that would be great. Your conclusion on the error codes seems reasonable to me.

Thanks,

Paul