Just Parse robots.txt
WWW::RobotRules::Parser allows you to simply parse robots.txt files as described in http://www.robotstxt.org/wc/norobots.html.
$
pkg install p5-WWW-RobotRules-ParserOrigin
www/p5-WWW-RobotRules-Parser
Size
12.5KiB
License
ART10, GPLv1+
Maintainer
markun@onohara.to
Dependencies
3 packages
Required by
1 packages