RECENT POSTS

P5-www-robotrules

May 26, 2018

Database of robots.txt-derived permissions

This module parses /robots.txt files which are used to forbid conforming robots from accessing parts of a web site. The parsed files are kept in a WWWRobotRules object, and this object provides methods to check if access to a given URL is prohibited.

WWW http//search.cpan.org/dist/WWW-RobotRules/