Cloudflare Removes Perplexity from Verified Bots and Blocks Stealth Crawling Tactics

Cloudflare has officially removed Perplexity from its list of verified bots and has begun actively blocking its crawler and associated stealth bots from accessing websites protected by its services. This action follows multiple user reports accusing Perplexity of violating robots.txt protocols, leading to an investigation that uncovered the use of aggressive, deceptive crawling tactics. The Cloudflare Verified Bots program is designed to whitelist reputable crawlers that adhere to established policies, such as respecting robots.txt directives. Perplexity was found to be in breach of these standards, notably by using rotating IP addresses, changing Autonomous System Numbers (ASNs), and impersonating legitimate browsers like Chrome to evade detection. Specifically, Perplexity employed various stealth techniques, including switching to IP addresses not officially designated for their crawling activities and altering user-agent strings to mimic genuine browsers. For example, when blocked, Perplexity's bots would change their user-agent to appear as Chrome on a Mac, a tactic known as spoofing. In response, Cloudflare announced that Perplexity has been delisted as a verified bot and w...