3 votes

Rather than crawl the site starting with the homepage, use most linked to pages, only crawling no external link pages after the linked pages have been crawled and internal links prioritized.

This will create a crawl of the site shaped more like Google's. Crawl depth will reflect distance from external links rather than merely distance from homepage.

Suggested by: <marquee>Russ</marquee> Upvoted: 22 Sep Comments: 1

Not planned

Comments: 1

Add a comment

0 / 1,000

* Your name will be publicly visible

* Your email will be visible only to moderators