Respect each subdomains robots.txt rules

Hi, during the analysis of websites segmented in different subdomains, I flagged the "treat subdomain as internal" function, but the robots.txt file of the individual subdomains is not taken into account but only the one of the root domain. It would be very useful to correctly analyse this type of websites to be able to enforce the rules of the individual robots.txt of each subdomain.

Under consideration Suggested by: Paolo G. Upvoted: 25 Nov, '20 Comments: 0

Add a comment

0 / 1,000

* Your name will be publicly visible

* Your email will be visible only to moderators