Every request carries a user agent, the name a browser or program gives itself. Go to Traffic Guard > Rules and allowlist to add your own rules for them.
How rules work
Rules are checked before anything else, lowest priority number first; the first rule that matches decides:
- Allow: the request is let through at once, without counting it against the rate limits or scoring it. Use it for a monitoring service, a harvester you work with, or a crawler you want to welcome.
- Block: the request gets a score of at least 80, which blocks it in every mode (and bans the address when automatic bans are on).
A pattern matches as text anywhere in the user agent, ignoring case (GPTBot matches Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.2)), or, when Pattern is a regular expression is ticked, as a regular expression without delimiters, also ignoring case (^curl/).
Adding a rule
Under Add a rule, enter the pattern, choose Block or Allow, set a priority (100 by default; lower is checked first) and an optional label, and click Add rule.
Turning rules off and deleting them
Turn off keeps a rule but stops checking it, for example while you test whether it matters; Turn on brings it back. Delete removes it.
Built-in bot detection
Without any rule, user agents that contain common tool and crawler names (python-requests, Go-http-client, curl, headless browsers, Scrapy, Bytespider, SemrushBot and others; see Risk Scoring) raise a request's score. An allow rule overrides that for a tool you trust.