Crawler for getting info about DNS domains and services
Despite the name, the crawler gets info for more services than just DNS: - DNS: - all A/AAAA records (for the 2nd level domain and www.subdomain), annotated with GeoIP - TXT records (with SPF and DMARC parsed for easier filtering) - TLSA (for the 2nd level domain and www.subdomain) - MX - DNSSEC validation - nameservers: - each server IP annotated with GeoIP - HOSTNAME.BIND, VERSION.BIND, AUTHORS.BIND and fortune (also for all IPs) - users can add custom additional RRs in the config file - E-mail (for every server from MX): - SMTP server banners (optional, ports are configurable) - TLSA records - Web: - HTTP status & headers (inc. parsed cookies) for ports 80 & 443 on each IP from A/AAAA records - certificate info for HTTPS (optionally with an entire cert chain) - webpage content (optional) - everything of the above is saved for each step in the redirect history -- the crawler follows redirects until it gets a non-redirecting status or hits a configurable limit
pkg install py311-dns-crawler