p5-WWW-RobotRules
6.02_1Database of robots.txt-derived permissions
This module parses /robots.txt files which are used to forbid conforming robots from accessing parts of a web site. The parsed files are kept in a WWW::RobotRules object, and this object provides methods to check if access to a given URL is prohibited.
Origin: www/p5-WWW-RobotRules
Category: www
Size: 25.5KiB
License: ART10, GPLv1+
Maintainer: perl@FreeBSD.org
Dependencies: 2 packages
Required by: 1 packages
Website: metacpan.org/release/WWW-RobotRules
$
pkg install p5-WWW-RobotRulesDependencies (2)
Required By (1 packages)
More in www
py311-requests2.32.5
Python HTTP for Humansp5-libwww6.81
Perl5 library for WWW accessp5-HTML-Parser3.83
Perl5 module for parsing HTML documentsphp84-session8.4.16
The session shared extension for phpp5-Catalyst-Runtime5.90132_1
Elegant MVC Web Application Framework (Runtime)py311-django424.2.29
High-level Python Web Frameworkapache242.4.66
Version 2.4.x of Apache web serverp5-HTTP-Message7.01
Representation of HTTP style messagesp5-Template-Toolkit3.102
Extensible template processing systemp5-Plack1.0051
Perl extension of PSGI reference implementation and utilities