In the past several months many sites have seen massive traffic spikes from AI bots. This has lead to overloaded web servers, inflated web hosting costs, and negative effects for legitimate traffic.

Sites using a CDN can block some of this unwanted traffic relatively easy, once you know how.

Here's how I blocked bots using Cloudflare:

• In Security, WAF go to the "Custom Rules" section.
• Click "Create rule".
• Add a name, e.g. "Block bots from search pages".
• Add a "Field" of "URI path", "Operator" of "Equals", and then set "Value" to the path of the page which is being overloaded, e.g "/search".
• Click the "And" button to add another condition.
• Set "Field" to "Verified bot category", set "Operator" to "Is in", then in the "Value" select every option shown.
  • It might be worth excluding "Accessibility" still allow accessibility tools to scan these pages.
• Set the "Action" to "Block", and then set "With response type" to "Default Cloudflare WAF block page".
• Set "Place at" to "First", so that it runs before other rules.
• Click "Save".
• Sit back and watch as traffic your server(s) slows down in a good way.