FreeBSD.software
Home/www/p5-WWW-RobotRules-Parser

p5-WWW-RobotRules-Parser

0.04001_1

Just Parse robots.txt

WWW::RobotRules::Parser allows you to simply parse robots.txt files as described in http://www.robotstxt.org/wc/norobots.html.

Origin: www/p5-WWW-RobotRules-Parser
Category: www
Size: 12.5KiB
License: ART10, GPLv1+
Maintainer: markun@onohara.to
Dependencies: 3 packages
Required by: 1 packages
Website: metacpan.org/release/WWW-RobotRules-Parser
$pkg install p5-WWW-RobotRules-Parser

Dependencies (3)

perl5p5-libwwwp5-URI

Required By (1 packages)

p5-Gungho

More in www

py311-requests2.32.5
Python HTTP for Humans
p5-libwww6.81
Perl5 library for WWW access
p5-HTML-Parser3.83
Perl5 module for parsing HTML documents
php84-session8.4.16
The session shared extension for php
p5-Catalyst-Runtime5.90132_1
Elegant MVC Web Application Framework (Runtime)
py311-django424.2.29
High-level Python Web Framework
apache242.4.66
Version 2.4.x of Apache web server
p5-HTTP-Message7.01
Representation of HTTP style messages
p5-Template-Toolkit3.102
Extensible template processing system
p5-Plack1.0051
Perl extension of PSGI reference implementation and utilities