Pure-Python robots.txt parser
Protego is a pure-Python robots.txt parser with support for modern conventions. It is mainly used by Scrapy application when crawling websites.
$
pkg install py311-protegoOrigin
www/py-protego
Size
202KiB
License
BSD3CLAUSE
Maintainer
dbaio@FreeBSD.org
Dependencies
2 packages
Required by
1 packages