Robots.txt expects that the client is respecting the rules, for instance, marking that they are a scraper.

AI scrapers don’t respect this trust, and thus robots.txt is meaningless.

Well, now that y’all put it that way, I think it was pretty naive from me to think that these companies, whose business model is basically theft, would honour a lousy robots.txt file…

the scrapers ignore robots.txt. It doesn’t really ban them - it just asks them not to access things, but they are programmed by assholes.

The difference is:

• robots.txt is a promise without a door
• Anubis is a physical closed door, that opens up after some time

The problem is Ai doesn’t follow robots.txt,so Cloudflare are Anubis developed a solution.

I mean, you could have read the article before asking, it’s literally in there…