DNS crawler 1.4.2
Version 1.4.2 of DNS crawler is out!
- added an option to save binary content if the webserver returns it for root (or redirects to it)
- added a
max_ips_per_domainlimit to help with domains that take A/AAAAs to the the extreme (> 20 records) and have broken HTTPS on webservers, so they won't fit into sane job timeouts… (unlimited by default)
- added a check to see if the IP from DNS records isn't from local range before trying to fetch web stuff from it
- the local IP check applies to GeoIP, too – it makes no sense to throw addresses like