You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
It seems that Google and Apple use "Extended" as signals in a robot.txt to exclude the scraped data from their LLMs. Since this is not sent as part of a User-Agent header, we can't block it via our bot detection. Instead, we should recommend users set up their robot.txt with these signals.
It seems that Google and Apple use "Extended" as signals in a robot.txt to exclude the scraped data from their LLMs. Since this is not sent as part of a User-Agent header, we can't block it via our bot detection. Instead, we should recommend users set up their robot.txt with these signals.
Ref arcjet/well-known-bots#13
Ref arcjet/well-known-bots#14
The text was updated successfully, but these errors were encountered: