ARTICLES

Recently, I’ve found myself typing quickly into my IM client to a few friends, explaining how a DNS change of some kind was going to work. It occurred to me that I should probably just write something up that would explain some DNS basics.

DNS is a big topic. I’m not even going to attempt to cover all of it here. If you really want to dig into it, then I recommend that you get and read the bugs book. A lot of this book is devoted to talking about the BIND software package (more on this later), but it’s still pretty much the reference for understanding DNS. If, on the other hand, you fall into the overwhelming majority of people who need to “get it”, but don’t need every detail, hopefully this article will help you out.

As noted, I’m not going to try and get into everything here. The thing with DNS, generally speaking, is that there’s really a lot of very small bits of information being passed around very, very quickly, and most things here in internet-landia rely on this to continue working. I’m going to talk about the following.

• Basic concepts and A Records
• TTL
• MX Records
• CNAMEs (a little)
• A brief bit about DNS server software

So without further adieu, here we go.

Basic Concepts

The DNS system is, at its heart, a translation mechanism between things humans are comfortable dealing with, and things that computers are comfortable dealing with. The human comfy part is essentially the domain name. Something like chrislea.com or nata2.org… something that we can see and recognize and remember. The thing is, your computer doesn’t care about those human readable domain names at all. The computer wants a number, since computers are pretty good at dealing with numbers. Specifically, the computer wants a number known as an IP address. That’s what DNS does.

The way that a domain name actually gets mapped to an IP for your computer is a little complicated, so I’m not going to talk about all the possibilities. The simplified version is that your computer will consult something called a “resolving nameserver”. These are different than “authoritative nameservers” which I’ll talk about in a bit. For now, there are two things to keep in mind.

1. Resolving nameservers are computers whose job it is to answer certain computers when they ask for a domain name to IP address mapping.
2. Your computer will typically use resolving nameservers that are provided by whatever ISP you are connecting to the internet through.

This means, for example, that if you are really unlucky and have Time Warner Cable as your ISP, then when your computer wants to know what IP address to use for say amazon.com, the Time Warner resolving nameservers are there to answer that question.

If you ask for a domain name and the resolvers don’t know the answer already, then the following sequence of things happens very quickly.

1. The resolver figures out what nameserver(s) know what IP is supposed to be used for the given domain.
2. The resolver asks that nameserver (which is the authoritative one) what the answer is.
3. The resolver stores the answer, and sends your computer the information it just got.

That is the basic life cycle of a DNS query. In general, you’ll be asking the resolver for what’s called an “A Record”. An A Record is an IP address that corresponds to the domain name you asked for.

There are a couple of things to pay attention to here. First off, the nameservers that your domain registrar made you pick as your authoritative nameservers isn’t actually what most client computers talk to. Those authoritative ones spend most of their time answering requests from resolvers. Another important point is that the resolvers store the answers for all the computers that use them. If you’re using those Time Warner resolvers, and somebody else who uses them has gone to Digg prior to you going there, then when you go you’ll get the already stored response. It doesn’t have to do the lookup each time. But, the resolvers will periodically do a fresh lookup, even if it already has an answer in hand. This brings us to our next topic.

TTL

So as noted, resolvers must periodically refresh what they know. Otherwise IP addresses couldn’t change reliably online. This is where the “TTL”, which stands for “time to live”, value comes in. When a resolver gets an answer from an authoritative nameserver, it gets more than just an IP address. It also gets a TTL value, which is measured in seconds. The TTL value tells the resolver “you don’t need to bother checking back in with me for at least X seconds”, where X is the TTL. In this way, the nameservers distribute data around the internet, and this keeps the authoritative nameservers for popular sites from dying under the strain. There’s no way that the Google nameservers could stay up if they got queried every single time anybody surfed to google.com.

Typical values for TTL might be 12 or 24 hours. So, if you were to change the IP address of a site you control, it would take potentially up to this long for the internet to adjust to the change. If you have control over the TTL value for a domain, and you want to change where it points, you may wish to lower the TTL value ahead of time. For example, if it’s set to 12 hours, then at least 12 hours before you want to make the update, change the TTL value to something low. This way, when you do the update, the internet will pick it up much more quickly. Then, after the update has happened, you can raise the TTL value to something normal again.

I recommend never lowering a TTL to anything smaller than 300 seconds, which is 5 minutes. This is because some resolvers are set up such that if they see a very low value, they assume it’s erroneous, and just ignore it and use their own default. If that happens, then you pretty much just invoked the exact opposite behavior of what you were aiming for.

MX Records

So it turns out there are different kinds of lookups you can do to a DNS server. The kind of lookup people generally think of corresponds to the A Record, which is what’s used for things like websites. I’m going to talk about two other kinds of records, the first of which is the MX Record.

In short, an MX (which stands for “mail exchanger”) Record is a special record that’s for email. When the thing that actually sends an email for you (called an MTA) is trying to figure out where the email should go, it checks the MX record for the domain to the right of the @ character in the address to see what to do. The MX record will point to at least one, and possibly several, domains. Then, it checks the A record for one of those domains to see what IP address to try and deliver the mail to. You can set a priority on the MX records, so that the mail sending program knows which to try first.

This accomplishes several things. First, it means that your mail doesn’t have to go to the same server as your website. That’s one possibility, but it could also go to a different computer altogether, such as to Google for GMail to handle, or to an exchange server, or any number of places. Second, because you can have more than one server listed to receive mail, you can get some built in resiliency. If the sending program can’t connect to the one with the highest priority it can try another one if it’s available.

CNAME Records

The last type of record I’ll mention is called a CNAME, which stands for Canonical Name Record. This type of record is an alias for another domain name. Why would you want to do this? Well, let’s find out.

Virb, which I work on, has a domain masking feature that’s in beta right now, but which will be rolling out to everybody soon. It lets you point a domain that you own at your Virb profile, and use it like it’s your own site. If I were going to do this, I’d set up a CNAME for the domain chrislea.com, and alias it to virb.com. Now, I could just set up an A record and point chrislea.com directly to the virb.com IP address, but that introduces a potential problem. If, in the future, we ever change the IP address for virb.com, I’d have to go and change my A record settings for things to keep working. By using a CNAME to alias my domain to just virb.com, we could change the IP and everything would keep working like it’s supposed to, so it’s a much better option for that sort of thing.

Server Software

I’m going to talk about the kinds of software that are used to run DNS servers just a little bit. Most of you will never have to deal with setting up or running nameservers, but it’s still worth a bit of discussion I think just so you’re familiar.

The most popular nameserver software is called BIND. It’s an old and very mature piece of software but has some significant problems. The way BIND works is that for every domain you want to serve up DNS information for, you create a flat text file called a zone file. The syntax for zone files is fairly cryptic, which is one problem. Another problem is that when BIND starts up, it loads the information for ALL the zone files it has into memory at once. This is fine if you don’t have very many domains, but when you have a lot you run into issues. One issue is that the main process starts using tons and tons of RAM. A second is that it takes BIND a long time to read in all the files, and to parse all the information into memory structures. at (mt) Media Temple, when we had something like 150,000 domains, it would take about 20 minutes to restart the BIND system we had in place at the time. Suffice to say, we got a new system in place.

As noted, DNS is a lookup system. You give the DNS server a domain name, and it gives you and IP address or another domain name back. This sounds a lot like a job for a database right? Thankfully, people much smarter than me also had that same idea. Enter PowerDNS, which is awesome. PowerDNS stores all the stuff that would be in zone files in BIND in a database, most often MySQL. When the software gets a request, it does a database query to get the information it needs and serves that back out. Whereas having 150,000 domains in BIND caused annoying problems, having 150,000 domains worth of info in a database like this barely even registers.

I have to put in a quick plug here. When were were looking to move away from BIND at (mt) Media Temple, we paid Netherlabs their consulting fees to have them evaluate our current setup and recommend a new architecture for us. It was worth every penny and then some. What we did put into production was almost exactly what they outlined and it’s been working flawlessly for over a year now, and we’ve added a considerable number of new zones in that time period on top of the ones we were starting with.

There are other options as well, but I like PowerDNS a lot, so that’s what I recommend you use if you need to run your own nameservers.

In Conclusion

There’s actually quite a bit more about DNS than what I’ve covered here. That said, I think this will probably answer a majority of questions people might have about how the DNS system works. If you really want to know more, that book I mentioned up at the start of the post is my recommendation to read.

If you need to set up nameservers, use PowerDNS. Or, if that doesn’t work for you, I urge you to use some DNS software that has a database backend.

Should you have any additional questions, please feel free to contact me.

Fedora 11 came out a few days ago (two weeks late, but who’s counting) so I went ahead and grabbed a copy and installed it. I figured I’d weigh in with my initial impressions since this is a pretty significant release. A few caveats to keep in mind for this.

• I’m going to talk really about the desktop experience here. I’ve not put it on a server and really beaten it up.
• I’ve installed it onto VirtualBox on my MacBook pro. Meaning, I have NOT installed it directly onto real hardware anywhere.
• As a consequence, I really haven’t bothered to even attempt testing out the Compiz stuff. Which is a shame becuase I love it.

The install was predictably painless. I intentionally clicked as few things as possible during the process and installed from the LiveCD, which is how I expect most people will be doing it. I think the only extra I may have done was to use NTP for getting the time. This is somewhat relevant because it means I got their default package selection for a desktop, as opposed to picking out all the things I know I’ll want at install time. Everything “just worked” the way it has with Fedora for a long time now, though I did feel like the process was a little faster than it had been on previous releases. This was surprising and pretty cool considering it was going onto VirtualBox and I didn’t give it a crazy amount of RAM or anything (768M). For the record, I’m running Leopard 10.5.7 on my MacBook and have the newest version of VirtualBox installed, which is 2.2.4 at the time of this writing.

I’ll start with the things I wasn’t that impressed with to get them out of the way.

For starters, I’m not a big fan of the default Fedora theme that was installed. I felt like the Fedora team was really going somewhere with Clearlooks back when it was first released, but since then, each “refinement” just feels a little more blue, and a little more cartoon-ish. It’s like they haven’t quite caught up with the reality that modern themes are moving towards more subtle gradients and bevels, with Apple’s design department as usual leading the charge. It’s not that this theme is bad, it just didn’t hold any “wow” factor for me. Furthermore, the installed additional options for themes, backgrounds, and so on was somewhat anaemic. Doing a yum search for more themes and installing the obvious choices didn’t really help. I’m still using the default Fedora theme because, although I don’t love it, it’s still my favourite choice.

The next nit-pick I have relates to Gnome-Do. Do is a truly awesome little piece of software that is similar to Quicksilver on OS X, but on steroids. I use it all the time, and I’m constantly trying to figure out new ways to get it to do cool things for me. It wasn’t installed by default, which I take no issue with as it’s a fairly new player on the Gnome desktop. The problem is that when I installed it, none of the plugins came with the installation. Nor was there a separate package for the plugins. This sucks, since the bulk of the cool stuff that you can do with Do is via plugins. A little searching indicated that the problem was, quite simply, nobody had bothered to package them for this release. Hopefully somebody will soon and they’ll get in there, but it’s a very sub-par experience using it compared to my Ubuntu desktop where all the usual plugins are easy to get and install via apt-get.

Finally, and this one I really do think is bad, none of the Openoffice software was installed by default. I have no idea if this was an oversight or, more likely, a decision, but I consider it pretty egregious in either case. Openoffice is certainly the de-facto office suite for the Gnome environment at this point, and not including it in the standard desktop install is a mistake. Users want it to be there. Abiword was there, which I personally use more than the OO word processor, but I’m and edge case and I know it. That should have been there.

That said, there’s a lot that I think they got right with this release. The update to the newest version of Gnome is an incremental one. Not much has changed, but it’s a bit smoother and faster. Users of previous releases will feel right at home, and I really don’t feel much of a shift doing normal things compared to my Ubuntu desktop. I consider that to be a very good thing.

True to Fedora form, the software that you can get is very much on the cutting edge. GCC is the brand spanking new 4.4 release, the python they ship is the 2.6 release designed to help people transition from the current 2.x series to the upcoming 3.x. Monodevelop (which I care about quite a bit personally) is the full fledged newest 2.0 release. I’m sure there’s going to be some issues because things are so new, but Fedora has always had the mindset of shipping the very newest versions of things it can get away with, and it’s continuing to do so here.

I have a few favourites in the newness category. The first is MySQL, which installs at version 5.1.32. Yes, I’m aware of all the hoopla surrounding the 5.1 release not being ready, but like it or not, it’s out there. I’m glad the Fedora crew shipped it, because even if there are bad bugs, the best way to get them fixed quickly is to get people using the software. The next one I love is Firefox, which installs at version 3.5 beta 4 It’s great. No problems thus far, and I get to play with the newest and shiniest toys.

Okay, so I just realized I lied. I have one more nitpick, which is that none of the beagle packages were installed by default. This one, I suppose, isn’t that big a deal, but I just installed it for the desktop, Evolution, Firefox, and Epiphany and the download was less than five megs. Search is getting more and more important, and users are really coming to expect it. I can’t see much argument for not including it by default, especially considering how nicely it works.

Everybody is expecting Fedora 11 to be the foundation for RHEL 6, whenever it comes out, and I think that this is a solid release for that purpose. There’s lots of updated software, but really it’s mostly an incremental release from Fedora 10 (yes, I know about ext4, no I don’t think it’s that drastic) and that means that the Red Hat crew is already dealing with a fairly mature and stable starting point. The amount of software available in the repositories is just staggering at this point, and things that don’t get into RHEL 6 will almost certainly be easily available from EPEL. In terms of ease of use, eye candy, and polish, I still have to give the nod to Ubuntu’s last release for desktop use. But Fedora has done a very good job I think of getting this amount of bleeding edge technology in front of people in a very attractive and usable way.

Like all properly enlightened techy types, I use Emacs as my editor of choice when I’m doing real coding work. I’ve been doing this for a while now, I’ve seen a lot of editors come and go, but Emacs (and XEmacs) have always been here for me. Yes, it’s ridiculously annoying to learn how to use properly. In fact, I’d hardly say I know how to use it “properly” considering what it’s capable of. But even with my limited knowledge of it, I’ve simply never found anything that really compares.

This sort of statement often gets other nerds riled up. The Vi vs. Emacs debate has been going on I think literally forever now, and I don’t really want to rehash any of it here. Suffice to say that I assume eventually the Vi people will figure out that they’re in the wrong and switch.

More interestingly of late has been the OS X people who are deeply, madly, rabidly in love with Textmate. I tried it out, and immediately felt like this guy:


Honestly, it was cute, but it took me less than three minutes to want to use a feature (split panes) that Emacs will happily do for me that it just doesn’t have. Since I needed to get actual work done, it was back to Emacs.

But I will say this for Textmate: the fonts looked beautiful. I mean, they were really lovely. Now, I like being a power user and I like getting stuff done efficiently, and therefore I like Emacs. But I also like things to look nice. I spent some time today trying to figure out how to make Emacs in Ubuntu look as pretty. This took me longer than I feel like it should have considering how easy the answer is.

At any rate, since it wasted my time, I thought I’d write it down so perhaps you won’t have to waste any of yours. On my Ubuntu 9.04 install, the following does the trick.


chl@melian:~$ sudo apt-get install emacs-snapshot-gtk
chl@melian:~$ echo "Emacs.font: Monospace-10" > ~/.Xresources
chl@melian:~$ xrdb -merge ~/.Xresources


Do that, start up emacs, and voila! The fonts look just about as nice as they did in Textmate using OS X.

Enjoy!

I wanted to get up on a soap box and talk about servers a bit. Getting disclosure out of the way as it pertains to what I’m going to talk about here:

• I work for (mt) Media Temple.
• The views expressed here are mine, not my employer’s.
• I set up the servers for the site Wefollow.
• I don’t work for Wefollow, Kevin Rose, or anybody affiliated.

(This will all make sense if you read to the end, promise…)

So today, I’m going to talk about servers. If you’re working on the web, you interact with servers. Even if you personally don’t deal with them (and the vast majority of you don’t) you still interact with them. They are an indispensable part of the web ecosystem. If you have a web site, then somewhere there is a server, or possibly a collection of servers, that feeds your site out to the Internet when asked. Most of you know this. But, I’d like to point out something that apparently is not properly understood.

Your server doesn’t scale.

Seriously. Your server doesn’t scale. Or, if you have a bigger site, then your servers (plural) don’t scale. They don’t scale ever.

Servers in a classical sense are physical devices. They are made of up hardware components, and each of those components has an ability to do some task at some speed, and that’s it. Nothing anywhere gains any sort of capacity in response to the amount of stuff it’s being asked to do. It’s not scaling, and it’s never going to. This is despite the fact that people wish they would, and as far as I can tell, a lot of people think that they ought to.

You’ll note I hedged on that last paragraph by saying “in a classical sense”. That’s because in this day and age, with the advent of virtualization, a “server” doesn’t always refer to a physical box anymore. The argument from the last paragraph still holds though. If you’re on a virtual server of some sort, there are defined amounts of physical resources assigned to it, and that’s what’s available. It still doesn’t have some inherent ability to adjust to the workload it’s being asked to perform.

A good analogy is with a car. If you go to the dealer and buy a Honda, you get a Honda. If, when you’re evaluating it, you ask the sales person, “Hey, if I really need to go fast, will this Civic scale into a Formula One Race Car?” you’d get a very funny look. Obviously, that’s not how it works. But for some reason, in server-landia, people have this idea that it can. Yes, it’s true that if you have a really good mechanic (analogous to a systems administrator) and a really good driver (analogous to a programmer) you might be able to get that Honda to go around a track faster than it did off the show room floor. But, it’s still never going get entered in a Grand Prix event.

What does scale on the web is architectures. People spend a lot of time on this, myself included. Architectures are about the only thing that you can honestly say scales. Everything else is hyperbole.

“But wait!” you say … “you work at an ISP, and didn’t you guys talk about your (gs) Grid-Service and say it was ready to scale?”

Well, yeah, we did. But the servers aren’t the parts that scale. That platform is a load balanced cluster with lots and lots of servers behind it, and that all “just works” for those customers transparently, as if it was one server. That’s the architecture that we developed, so that for the web servers, those customers have built in horizontal scaling over many nodes. The servers themselves don’t scale one whit, but the architecture does. Similarly, we developed technology for that platform to move databases around for customers transparently, and to put them into VPS servers with dedicated resources if a particular customer needs a lot of MySQL juice, all without any intervention. So there, we have an architecture that provides some automagic vertical scaling for customer databases.

There is a somewhat similar story for the (dv) Dedicated-Virtual servers that we sell at work. As noted, those are servers that are assigned defined amounts of physical resources. The software that does this allows us to very easily change those amounts, and to move the servers around between physical hardware that has sufficient resources to provide the assigned levels of things. Can this be done on-the-fly and automagically? Yes, it can be done. But if you figure out how to make that happen, you just implemented an architecture. And, for the record, if you do develop an architecture that just makes your VPS bigger when needed, you’ve implemented an architecture for vertical scaling. Vertical scaling is neat because you don’t have to change your app, but it will only take you so far, because eventually no one physical machine will be big enough anymore. In any case, the server is still always just a pool of hardware resources for your app.

So why am I here preaching about this today? Well, it’s because frankly I’m sort of sick of seeing things like “those servers don’t scale” on Twitter / blog posts / etc. Sure, they’re correct, “those servers don’t scale” because SERVERS DON’T SCALE. And the way that sentiment is generally announced is by somebody piping up and saying “FooBar site is down, and so their ISP sucks!!!”.

Case in point: http://wefollow.com.

Let’s take a quick walk through time and talk about that site. A few months back, Kevin hit me up and said he was going to need some hosting for a personal project. Since I work at (mt) Media Temple, I was able to facilitate that. Since Kevin’s a friend, I offered to help with setup and configuration. For the record, the site is hosted on two (dv) Rage servers. One is a webserver machine, the other is a database machine, both participate in a memcached pool.

I did some basic setup and then pretty much forgot about it for a while. When it was getting close to launch, he hit me back again and asked me to give things a once over, which I did, albeit not very thoroughly as we’ll see. The site was launched during SXSWi this year, and promptly crashed. Naturally, this was immediately twittered about and there was a bunch of “media temple sucks” tossed around. As if, somehow, (mt) was responsible for the site being down.

The site went down right away because I missed something fairly obvious that I really should have caught in the Apache configuration. That was it. When I found it and fixed it, the site popped right back up.

Until of course the next day, when it crashed again. Queue the cries of (mt) sucking from all over. That second spat of downtime was because the programmer for the site, Jeff Hodsdon, missed a database query that should have been getting cached in memcached but wasn’t. That query got hot, MySQL promptly went into a tailspin, and the site went down. It took me a little while to find the bad query, but once I did Jeff fixed it in about five minutes and things were okay again. [For the record, I'm definitely NOT dissing Jeff's coding chops. This was a side project and nobody had time for real QA. He missed one, it happens. If I had been as good at 19 as Jeff is, I'd probably be retired by now. Kid has some serious fu happening.]

After that, and I forget exactly when, there were two more brief periods of downtime. Both were because I tried doing some more intricate things with the Varnish configuration and I got it wrong. Since I got that sorted out, the site has been running smooth as teflon, even through the last traffic whompings when Kevin broadcasted something about the site to his eighteen-squajillion followers on Twitter.

The key point I’d like to highlight here is this: at no point did the two VPS servers change at all. Not a bit. Zero. Nada. Zilch. At one point during those last two oopses Kevin IM’d me and said something to the effect of “Hey man, I really don’t mind paying for bigger servers.” I’ve been doing this for a long time and given the amount of traffic the site is pushing, I knew that there was enough hardware there already. I asked him for another day to try and get things configured right, and (knock on wood) I did, and all’s been well. (And yes, for the astute readers out there, we have seen traffic on par with what we saw at launch since these last fixes and things have been fine).

But all was fine with the servers the entire time. The servers have always provided the amount of resources that they were advertised to have. When I provisioned them using the internal (mt) provisioning system, I got what I was supposed to get. So basically, the servers have always been doing exactly what they should have, (mt) did exactly what it should have, and the only entity responsible for the site going down is the person who configured things wrong. In this case, that was me. The servers never changed at all. What did change to get things nice and stable was that we fixed the code and configs, which is what currently constitutes the architecture for the site.

Again, just to be clear: we vastly increased the efficiency of the architecture that the site runs on, and that’s why it’s running well now. The servers are exactly the same as they were on day one, when everything died. The servers didn’t scale, because they’re just resource pools, but we figured out how to use the resources they provide properly.

“But what about The Cloud?” you say. Oh boy. Don’t get me started. I’ll save that one for a different post.