Proprietor of QWeb Ltd and Bouche Chocolate. A software and game programmer since 1996, these days I mostly make #websites and #chocolate, manage #Linux servers, and drink #whisky. But #gamedev is still my big love.
Currently very slowly working on Argentauria (in #Godot ), an open-world MMORPG inspired by medieval life, tools, and architecture. Devlog at https://ko-fi.com/dev/_ric
- 0 Posts
- 1 Comment
Joined 2 years ago
Cake day: November 10th, 2022
You are not logged in. If you use a Fediverse account that is able to follow users, you can follow this user.
@remixtures@tldr.nettime.org GPTBot is the most aggressive content scraper I’ve come across in decades of server management. Totally ignores any crawl limits that you set in your robots.txt, and they operate on enough IPs to make even nginx configured rate limiting a bit futile.
You can, though, block them (and others) by their useragent string. Add this to your .htaccess to block both GPTBot and Claude, for example:
SetEnvIfNoCase ^User-Agent$ .*(ClaudeBot|GPTBot) BADBOTHAMMER
Deny from env=BADBOTHAMMER