Version 1.4.2 of DNS crawler is out!

New features:

• added an option to save binary content if the webserver returns it for root (or redirects to it)
• added a max_ips_per_domain limit to help with domains that take A/AAAAs to the the extreme (> 20 records) and have broken HTTPS on webservers, so they won't fit into sane job timeouts… (unlimited by default)
• added a check to see if the IP from DNS records isn't from local range before trying to fetch web stuff from it
• the local IP check applies to GeoIP, too – it makes no sense to throw addresses like 192.168.X.X at it