Hacker News Digest

Тег: #no-crawl-directives

Постов: 1

Perplexity is using stealth, undeclared crawlers to evade no-crawl directives (blog.cloudflare.com) 🔥 Горячее 💬 Длинная дискуссия

by rrampage • 04 августа 2025 г. в 13:39 • 1282 points

ОригиналHN

#web-crawling#no-crawl-directives#cloudflare

Комментарии (741)

I find this problem quite difficult to solve:1. If I as a human request a website, then I should be shown the content. Everyone agrees.2. If I as the human request the software on my computer to modify the content before displaying it, for example by installing an ad-blocker into