Where do services like http://whoisxmlapi.com get their data?

Question

http://whoisxmlapi.com provides a high-throughput API for thick WHOIS queries. In my understanding such data can only be provided by domain registrars that put really low limits on the number of queries per minute. Had this data just been carefully scraped from the places like http://whois.godaddy.com from multiple IPs or is there a way to get such data in bulk?

Answer 1

ICANN regulations oblige registrars to make WHOIS data available in bulk, albeit for a fee. You can read the details in full here:

3.3.6.2 Registrar may charge an annual fee, not to exceed US$10,000, for such bulk access to the [WHOIS] data.

I don't know exactly how sites such as whoisxmlapi and DomainTools operate, but it's possible that they are able to perform more than the usual number of WHOIS queries because they've stumped up the cash for the privilege.

ICANN regulations still impose certain limitations on the access:

3.3.6.4 Registrar's access agreement shall require the third party to agree not to use the data to enable high-volume, automated, electronic processes [...]

Again, speculation, but perhaps large numbers of queries to registrars' databases are not frowned upon if they're not automated/scripted. Sites like whoisxmlapi and DomainTools offer a WHOIS lookup service where users can query individual domain names. It'd be very simple for them to cache and index the record before they show it to the user.

Of course, as you suggest, maybe they've just got hundreds of IP addresses at their disposal and go about harvesting the data in a crafty way.