Tags: robots.txt

New data shows AI bots are rapidly increasing their share of internet traffic, often bypassing standard safeguards like robots.txt. Publishers and website owners are confronting a sophisticated arms race as bots disguise themselves as human browsers and employ advanced scraping techniques. Companies such as TollBit, Cloudflare and others are offering tools to detect, block, or monetize bot access, while a growing market promotes services that help content appear in AI-driven search results. The shift is reshaping how the web functions and creating new revenue streams for digital publishers. Ler mais

The New York Times has filed a lawsuit in a New York federal court accusing AI startup Perplexity of copying and distributing its articles without permission. The suit alleges that Perplexity’s crawlers deliberately ignored technical safeguards, such as robots.txt, to harvest content that the Times says is reproduced verbatim or in substantially similar form. The legal action follows earlier cease‑and‑desist notices and comes amid a broader wave of publisher lawsuits targeting AI firms for alleged copyright violations. Ler mais