Dev site crawl

4 votes

In the crawl settings, I'd like to be able to configure a development URL as well so that I can compare the dev site with the live site. It will need password authentication, if the devs have set the thing up correctly, or an ignore Robots directive if they have left open (to the world to index...)

(DeepCrawl might do this...)

Done Suggested by: Simon Cox Upvoted: 11 Nov, '19 Comments: 0

Comments: 0