-
Notifications
You must be signed in to change notification settings - Fork 474
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Adds detection for various bots (#7661)
* Change Facebook External Hit to Facebook Crawler * Improves detection for generic bots * Adds detection for InsytfulBot * Adds detection for Statista * Adds detection for Substack Content Fetch * Adds detection for Deep SEARCH 9 * Adds detection for LiveJournal * Adds detection for Tenable.asm * Adds detection for Castopod
- Loading branch information
1 parent
861597a
commit bc62002
Showing
2 changed files
with
142 additions
and
12 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -912,27 +912,27 @@ | |
- | ||
user_agent: facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php) | ||
bot: | ||
name: Facebook External Hit | ||
name: Facebook Crawler | ||
category: Social Media Agent | ||
url: https://www.facebook.com/externalhit_uatext.php | ||
url: https://developers.facebook.com/docs/sharing/webmasters/crawler/ | ||
producer: | ||
name: Meta Platforms, Inc. | ||
url: https://www.meta.com/ | ||
- | ||
user_agent: facebookexternalua | ||
bot: | ||
name: Facebook External Hit | ||
name: Facebook Crawler | ||
category: Social Media Agent | ||
url: https://www.facebook.com/externalhit_uatext.php | ||
url: https://developers.facebook.com/docs/sharing/webmasters/crawler/ | ||
producer: | ||
name: Meta Platforms, Inc. | ||
url: https://www.meta.com/ | ||
- | ||
user_agent: facebookplatform/1.0 (+http://developers.facebook.com) | ||
bot: | ||
name: Facebook External Hit | ||
name: Facebook Crawler | ||
category: Social Media Agent | ||
url: https://www.facebook.com/externalhit_uatext.php | ||
url: https://developers.facebook.com/docs/sharing/webmasters/crawler/ | ||
producer: | ||
name: Meta Platforms, Inc. | ||
url: https://www.meta.com/ | ||
|
@@ -4568,9 +4568,9 @@ | |
- | ||
user_agent: facebookcatalog/1.0 | ||
bot: | ||
name: Facebook External Hit | ||
name: Facebook Crawler | ||
category: Social Media Agent | ||
url: https://www.facebook.com/externalhit_uatext.php | ||
url: https://developers.facebook.com/docs/sharing/webmasters/crawler/ | ||
producer: | ||
name: Meta Platforms, Inc. | ||
url: https://www.meta.com/ | ||
|
@@ -7472,3 +7472,80 @@ | |
producer: | ||
name: Google Inc. | ||
url: https://www.google.com/ | ||
- | ||
user_agent: KvshClient | ||
bot: | ||
name: Generic Bot | ||
- | ||
user_agent: Mozilla/5.0 infrawatch/0.1 | ||
bot: | ||
name: Generic Bot | ||
- | ||
user_agent: InsytfulBot/1.0; https://www.insytful.com/about-our-bot | ||
bot: | ||
name: InsytfulBot | ||
category: Crawler | ||
url: https://www.insytful.com/ | ||
producer: | ||
name: Zengenti Limited | ||
url: https://www.zengenti.com/ | ||
- | ||
user_agent: statista.com PublicationFinder-Crawler 2.0 | ||
bot: | ||
name: Statista | ||
category: Crawler | ||
url: https://www.statista.com/ | ||
producer: | ||
name: Statista, Inc. | ||
url: https://www.statista.com/ | ||
- | ||
user_agent: SubstackContentFetch/1.0; https://substack.com | ||
bot: | ||
name: Substack Content Fetch | ||
category: Crawler | ||
url: https://substack.com/ | ||
producer: | ||
name: Substack, Inc. | ||
url: https://substack.com/ | ||
- | ||
user_agent: ds9 2.000.ec2(+http://www.deepsearchnine.com/ds9.html) | ||
bot: | ||
name: Deep SEARCH 9 | ||
category: Crawler | ||
url: https://www.copyright.com/blog/ccc-expands-corporate-solutions-offering-with-new-technology/ | ||
producer: | ||
name: Copyright Clearance Center, Inc. | ||
url: https://www.copyright.com/ | ||
- | ||
user_agent: ds9 2.000.ec2 | ||
bot: | ||
name: Deep SEARCH 9 | ||
category: Crawler | ||
url: https://www.copyright.com/blog/ccc-expands-corporate-solutions-offering-with-new-technology/ | ||
producer: | ||
name: Copyright Clearance Center, Inc. | ||
url: https://www.copyright.com/ | ||
- | ||
user_agent: LiveJournal.com ([email protected]; for https://www.livejournal.com/users/example/; 1 readers) | ||
bot: | ||
name: LiveJournal | ||
url: https://www.livejournal.com/ | ||
category: Feed Fetcher | ||
producer: | ||
name: ООО "СИМ" | ||
url: https://www.livejournal.com/ | ||
- | ||
user_agent: bitdiscovery-suggestions | ||
bot: | ||
name: Tenable.asm | ||
category: Security Checker | ||
url: https://bitdiscovery.com/ | ||
producer: | ||
name: Tenable, Inc. | ||
url: https://www.tenable.com/ | ||
- | ||
user_agent: Castopod/1.0 | ||
bot: | ||
name: Castopod | ||
category: Crawler | ||
url: https://www.castopod.org/ |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters