AI-enabled (or incentivized?) web scraping and crawling is a scourge on the web. That said, some bot traffic did give me a chuckle today.
At Reclaim, we use Anubis to protect some sites targeted by bot traffic, providing an extra layer of DoS protection when needed. Anubis is an amazing tool, and the developer Xe is doing amazing work.
Today, while putting a site behind Anubis, I was monitoring…