Responsible crawling: react to status 429 by throttling back or pausing

2 votes

(Perhaps offered as an option) when a server responds with a 429 (too many requests) throttle back a bit eg. fewer requests per second, or if that's not monitored any more, add delay between requests or pause briefly.

I guess the specific reaction doesn't matter as long as it's reducing server load when it complains.

Done Suggested by: John Ferguson Upvoted: 20 Jul, '21 Comments: 0

Comments: 0