DNS crawler 1.4.2

March 24, 2020

Version 1.4.2 of DNS crawler is out!

New features:

  • added an option to save binary content if the webserver returns it for root (or redirects to it)
  • added a max_ips_per_domain limit to help with domains that take A/AAAAs to the the extreme (> 20 records) and have broken HTTPS on webservers, so they won't fit into sane job timeouts… (unlimited by default)
  • added a check to see if the IP from DNS records isn't from local range before trying to fetch web stuff from it
  • the local IP check applies to GeoIP, too – it makes no sense to throw addresses like 192.168.X.X at it