Pro@programming.dev to Technology@lemmy.worldEnglish · 2 months agoCodeberg: army of AI crawlers are extremely slowing us; AI crawlers learned how to solve the Anubis challenges.i.imgur.comexternal-linkmessage-square225fedilinkarrow-up11.16Karrow-down111file-text
arrow-up11.15Karrow-down1external-linkCodeberg: army of AI crawlers are extremely slowing us; AI crawlers learned how to solve the Anubis challenges.i.imgur.comPro@programming.dev to Technology@lemmy.worldEnglish · 2 months agomessage-square225fedilinkfile-text
minus-squareT156@lemmy.worldlinkfedilinkEnglisharrow-up8·2 months agoThe website would also have to display to users at the end of the day. It’s a similar problem as trying to solve media piracy. Worst comes to it, the crawlers could read the page like a person would.
minus-squareacockworkorange@mander.xyzlinkfedilinkEnglisharrow-up1·2 months agoYou could have a server for open code access with very limited bandwidth and another for authenticated users with higher bandwidth.
The website would also have to display to users at the end of the day. It’s a similar problem as trying to solve media piracy. Worst comes to it, the crawlers could read the page like a person would.
You could have a server for open code access with very limited bandwidth and another for authenticated users with higher bandwidth.