• medem@lemmy.wtf
    link
    fedilink
    arrow-up
    23
    arrow-down
    2
    ·
    18 days ago

    <Stupidquestion>

    What advantage does this software provide over simply banning bots via robots.txt?

    </Stupidquestion>

    • kcweller@feddit.nl
      link
      fedilink
      arrow-up
      73
      ·
      18 days ago

      Robots.txt expects that the client is respecting the rules, for instance, marking that they are a scraper.

      AI scrapers don’t respect this trust, and thus robots.txt is meaningless.

    • medem@lemmy.wtf
      link
      fedilink
      arrow-up
      42
      ·
      18 days ago

      Well, now that y’all put it that way, I think it was pretty naive from me to think that these companies, whose business model is basically theft, would honour a lousy robots.txt file…

    • thingsiplay@beehaw.org
      link
      fedilink
      arrow-up
      13
      ·
      17 days ago

      The difference is:

      • robots.txt is a promise without a door
      • Anubis is a physical closed door, that opens up after some time
    • Mwa@thelemmy.club
      link
      fedilink
      English
      arrow-up
      8
      ·
      18 days ago

      The problem is Ai doesn’t follow robots.txt,so Cloudflare are Anubis developed a solution.