DNS is one of those things that’s easy to take for granted. It’s been around a long time (first defined in 1983) and for the most part it just works. It’s also one of the most important protocols/services used for the Internet and networks in general. In fact, without it the Internet as we know it wouldn’t exist.
I’ve been immersed in Internet protocols for nearly twenty years now and DNS is one that I’ve worked with extensively and thought I knew a lot about. I do, but it’s also way more complex and involved than just resolving names to IP addresses, like names in a phone book to phone numbers. I’ve run my own internal / external / split-horizon name servers for years; still do in fact, but I also outsource some name services as well.
A couple weeks ago DNS came back on my radar in a big way. I manage the network for a rapidly growing online video company. We have seen about a 100 fold increase in traffic to our platform over the last year and project nearly that much growth for years to come. Each time our embedded video players load, which is hundreds of millions of times a month, several name resolutions occur. I estimate our current DNS queries to be tens of millions per day (unfortunately my current DNS provider cannot provide numbers for me).
Previously I worked for an e-commerce company whose site had maybe 10 million visitors a month and we could handle the DNS traffic in-house just fine. When I started with my current employer a couple years ago we had fairly low traffic and we didn’t have the physical infrastructure to host our own DNS so I found a cheap DNS hosting company. They have worked fine to this point, but I believe we’ve outgrown that provider in more ways than one.
Recently I had one of the top DNS hosting companies contact me and try to win my business. They spewed all kinds of fancy numbers regarding ROI and performance. I pretty much shut them down, mostly on price, but they’ve persisted trying to convince me that because of poor DNS performance from my current provider it must be costing me money – hence the ROI pitch.
While I’ve resisted the sales pitch part of this process it has caused me to take a fresh new look at our current and future needs for DNS. In the process I’ve had some pretty good realizations and even come up with some useful information & that’s what I’m going to try to provide here – some useful information on DNS. Not how DNS works or what it is (you can get that here), but whether to host my own DNS servers once again or continue outsourcing, and if I outsource, who should I use.
Outsource or Manage In-house
Today I did some research and found that 75% of the top 100 US-based Internet sites, and about 70% of the top 10,000 sites host their own DNS. In fact, all of the top 10 and 24 of the top 25 sites host their own, with Twitter being the only one that outsources their DNS (see below for top 25 sites). It makes sense if you are a large company or if your site gets so much traffic that you’d want to have your own experts to manage your DNS as a lot is riding on this service. If I had the resources I’d probably do the same, however we have a lean crew and we’re still growing so this is something I just don’t want on my plate at this time – not to mention recent issues with DNS security. So for us it definitely makes sense to outsource.
Top 25 US sites & who they use for DNS
| Rank | Site | Authoritative Name Server | DNS Provider |
|---|---|---|---|
| 1 | google.com | ns1.google.com | self |
| 2 | facebook.com | ns0.facebook.com | self |
| 3 | yahoo.com | ns1.yahoo.com | self |
| 4 | youtube.com | sjl-ins1.sjl.youtube.com | self |
| 5 | msn.com | ns1.msft.net | self |
| 6 | live.com | ns1.msft.net | self |
| 7 | amazon.com | dns-external-master.amazon.com | self |
| 8 | microsoft.com | ns1.msft.net | self |
| 9 | wikipedia.org | ns0.wikimedia.org | self |
| 10 | aol.com | dns-02.ns.aol.com | self |
| 11 | ebay.com | sjc-dns1.ebaydns.com | self |
| 12 | blogspot.com | ns1.google.com | self |
| 13 | blogger.com | ns1.google.com | self |
| 14 | ask.com | name5.ask.com | self |
| 15 | craigslist.org | ns1p.craigslist.org | self |
| 16 | bing.com | ns1.msft.net | self |
| 17 | answers.com | infra01.nj1.answers.com | self |
| 18 | myspace.com | ns1.myspace.com | self |
| 19 | about.com | nydns0.about.com | self |
| 20 | twitter.com | ns1.p26.dynect.net | Dynect (Dyn) |
| 21 | mapquest.com | dns-02.ns.aol.com | self |
| 22 | wordpress.com | ns1.wordpress.com | self |
| 23 | adobe.com | adobe-dns-3.adobe.com | self |
| 24 | photobucket.com | buster.photobucket.com | self |
| 25 | godaddy.com | cns1.secureserver.net | self |
How Should I Choose An Outsource DNS Company
I’m currently using a low-end service of one of the larger DNS hosting companies. When one of their competitors contacted me recently they claimed that 6% of the queries weren’t being answered. Of course they couldn’t provide me with detailed information about what was being tested or how; and they are probably biased against this competitor. They just said to take their word for it, and that if I signed up with their company all my worries would be over.
Not being the kind of guy to trust a sales person I setup some testing of my own. I have some versatile monitoring/alerting software that is very flexible and can test just about anything I program it for. So several days ago I setup some DNS tests to run every 5 minutes against my current authoritative name servers. The tests simply resolve about 10 of my domain names of various kinds (A, CNAME, MX) against 5 of my DNS hosting company’s web servers – the servers that are authoritative for my domain – that’s 50 tests on each of two servers, one east coast, the other west coast. Other tests that I’ll reference below were setup similarly where requests against a DNS server were to resolve names for which that server is authoritative (primary or secondary).
The aforementioned salesman told me that about 6% of the requests against my current DNS hosting provider weren’t answered. Based on my own tests, it’s actually not that far off. What I’m seeing is about a 99.5% success rate against their “#1” DNS server, the one that’s listed as the primary name server for my domain; however, the secondary, tertiary, four-whatever and five-whatever servers range from about 95 – 98% success rate.
I’m also testing against the servers of the guy trying to sell me his services and a couple others for comparison. While I’ve found the failure or non-answer rate to be lower with this other company, others I’m testing actually perform a little better. I brought this up to the sales guy and one of his “engineers” yesterday and they said, “well nobody is perfect.” That was the response after I told them about the results of my current provider, but hadn’t yet told them about the results against their servers. They jumped all over this and told me that all these unanswered DNS queries were causing 404 errors on my pages. They back pedaled on that claim when I told them that their servers were only answering about 98 – 99% of the queries I was sending their way. That’s when he said nobody is perfect.
So, What Happens When DNS Queries Aren’t Answered?
This of course was a burning question I wanted answered. Since the salesman and so-called engineer claimed they would cause 404 or page not found errors I wanted to find out if that’s really the case. So I did some testing. I broke out the venerable Wireshark and took a look at just what was going on with DNS queries.
What I found is that the request is retried (at least on a Windows 2003 server) by the client if it isn’t answered, or times out after one second. While it did slow down the page load it did not cause the much-feared and promised 404.
So What If My DNS Queries Are A Little Slow
Well, it can actually cause or at least contribute to a poor user experience, which in turn can cause a reduction in traffic and ultimately a hit to the bottom line. In my case when a player loads on a page it loads elements from about seven locations (widgets, thumbnails, videos, analytics, etc.). Each one of these elements, since they each have their own host name within my domain causes a DNS request. If the elements are loaded sequentially and if each DNS request takes, say, 200 miliseconds that’s nearly a second-and-a-half just for DNS calls. That can be significant. In fact, with my testing I’m seeing times in the 50 – 300 milisecond per DNS lookup with my current DNS provider’s servers. While all of these elements don’t load sequentially, some load in tandem, I estimate the time to be between .5 and 1 second just for DNS on each page load. That’s significant.
NOTE: Let me just interject here that my tests are assuming all queries are going to my name servers, which I know isn’t the case. Again, this post isn’t going into the inner workings of DNS queries and resolution. I’m not addressing DNS caching, TTL, other DNS servers in the mix, etc.
The other important factor, besides the request being answered in the first place, is just how long, on average, the requests take. Again with my testing I’m seeing a fairly poor result with my current DNS provider and significantly better performance with their competitors.
It’s important to look at the average latency per request & add to that the reliability of the requests. If your provider has a low latency generally, but a high number of timeouts and subsequent retransmits that will add to the overall delay of the client requests. Obviously you want for each DNS query (which I’ve been calling requests) to be answered and answered quickly.
Who Are DNS Hosting Providers And Who Should I Use
As stated above most top sites handle their own DNS. However, there are a number of DNS providers and they get used increasingly as more sites are analyzed. I did a quick survey of the top 10,000 to get a pretty good idea of which are used and to what extent.
I found a pretty good list of DNS hosting providers which includes some I’m investigating, others I’ve heard of and some I’ve never seen. I found that most of these are being used by one or more of the top 10,000 sites.
Wait! Did I say I did a quick survey of 10,000 sites? How the hell? you say? It was actually pretty easy. I downloaded today’s top million sites from Quantcast & grabbed the top 10k from there which I put into a spreadsheet. Then I used nslookup (example: nslookup -querytype=soa google.com) to get the info, redirecting all 10k to a text file. Each record had a bunch of extra info so I used grep to extract the lines with the primary name server. The whole process took less than 30 minutes.
| DNS Provider | # From Top 1,000 | # From Top 10,000 | % of Top 10,000 |
|---|---|---|---|
| Domain Control (GoDaddy) | 17 | 496 | 4.96% |
| Ultra DNS | 89 | 441 | 4.41% |
| Rackspace | 29 | 367 | 3.67% |
| DNS Made Easy | 38 | 359 | 3.59% |
| World NIC (Network Solutions) | 179 | 1.79% | |
| EasyDNS | 134 | 1.34% | |
| Dynect (Dyn) | 30 | 126 | 1.26% |
| ZoneEdit | 63 | 0.63% | |
| mydyndns.org (Dyn) | 62 | 0.62% | |
| EveryDNS | 35 | 0.35% | |
| public-ns.com | 29 | 0.29% | |
| Akamai | 11 | 28 | 0.28% |
| TZO | 14 | 0.14% | |
| No-IP.com | 13 | 0.13% | |
| DNS Park | 13 | 0.13% | |
| Netriplex | 11 | 0.11% | |
| Nettica | 10 | 0.10% | |
| OpenDNS | 1 | 0.01% |
From this list Domain Control (GoDaddy) and WorldNIC (Network Solutions) are not options for me as their interfaces suck, they are clunky, they don’t have SLA’s, etc. Don’t get me wrong, I’ve used them both for DNS, in fact still do – GoDaddy is the registrar for this domain and I use their DNS, works great for this, however, I cannot run a large for-profit business website with their DNS. Also, Rackspace is out as I believe their service is primarily for existing customers. Again, nothing against Rackspace, I’m just not a regular customer of theirs so I’m not evaluating their DNS solution. That leaves me with UltraDNS, DNS Made Easy, EasyDNS, and Dyn. I was already checking out three of the four & now I’m evaluating all four. Actually, make that five. I’m evaluating Akamai’s DNS as they are the biggest player in the CDN space. Oh, and I’m a customer of theirs already.
Now that I have my list boiled down to UltraDNS, DNS Made Easy, EasyDNS, Akamai, and Dyn it’s time to vet each one a little more. At this point I’m comfortable that each could handle our current and projected volume, so check that off the list. Next is pricing, which I’m working on. It’s also important to me to get to know and feel the interface, support, etc. so I’m working on that with each as well. Last, but not least is performance of each. I’m already doing my own testing on three of the five. I’ll add the other two, let it run for a day or two then get back here to post the results and any other salient thoughts I might have.
For now, I’m out. . . .
See also:
- DNS Hosting Part II which contains the top 10 list of each of the DNS providers I’m investigating.
Great article but not 100% correct. If you look at the DNS for FaceBook and MySpace. Look at who owns the IPs of those name servers. It is actually UltraDNS.
Those services actually will rebrand the UltraDNS service.
Hope that does not over complicate anything.
Please consider 3Crowd. Their announcement (http://bit.ly/9tlW8v) may be new, but the founder/CEO Barrett Lyon has been around for a while and has an excellent track record. I'm biased, as Barrett is a friend of mine, but my comments are true. Give them a call, see what you think. I would like to know what your experience was like…
uatfig, thanks for the comment. I'll checkout 3Crowd as well.
Anonymous, thanks for the info. I didn't look at it to that level, by checking the IP addresses. I just looked at the authoritative name server & if it was the same name as the domain or other owned by the same company I assumed they were doing their own DNS. I'll have to take a closer look & update this article. Thanks.
"Unanswered DNS queries cause 404 errors"
This either reflects a profound misunderstanding of how things actually work on the part of the salesman or else he is blowing you-know-what up your you-know-where and hoping you don't know better.
A 404 comes from the webserver itself, before anything can come from the webserver, it has to be located via the DNS. If the DNS fails, you'll never get anything from the webserver. Not even a 404, you'll get one of those browser errors "Can't find the website".
These salesguys kill me. Thank god I don't employ any.
Mark, thanks for your comments. When the "sales engineer" said that I knew it was BS. It was especially gratifying when I pointed out that his company's DNS services were not answering 1-2% of the queries either, based on my own tests. He really started to back pedal, but didn't back down from his claim of 404's.
best best
best
PCA, have you reviewed http://www.dns.com's service? If so what did you think?
http://www.communitydns.net/ is very good.