Packets of Death and Intel e1000 cards?

February 8, 2013 - No Comments

An interesting post is making the rounds in regard to a DoS vulnerability in Intel 82574L ethernet controllers.  It seems that at least one motherboard manufacturer was shipping these controllers in a configuration that would cause the receipt of a specific packet to cause the controller hang.  The upside of this is that in some certain configuration, this controller is vulnerable to a DoS attack that can drop it off the network.

Intel responded, but unfortunately their response is the classic head-in-the-sand security speak — which boils down to “don’t worry about it, nothing to see here.”  This isn’t good enough.  Do I have one of the motherboards at issue?  How can I check?  It seems like the author of the post isn’t the only one who has experienced this issue.  More information is needed.  I hope Intel steps up and provides the information that the security community needs to respond.

DDoS and DNS – Today’s AT&T DNS Outage

August 16, 2012 - No Comments

AT&T suffered a serious outage of their authoritative DNS services today, apparently due to a distributed denial of service (DDoS) attack.  Information is scarce thus far – InfoWorld is reporting on the outage with a few quotes from AT&T, but there doesn’t seem to be much official news from AT&T.

DDoS attacks against DNS are nothing new.  DNS is a fairly trivial protocol to DoS; it’s not very hard to produce large volumes of legitimate-looking queries.   The only real defense against such attacks is to provision lots of spare capacity.  Even the big guys suffer — witness Amazon’s outage back in 2009 for one example of the DNS DDoS fad.  So what should your Operations team know about DNS DDoS attacks?

First, the age of running your own name servers is over.  This may seem a counterintuitive recommendation in response to a major DNS provider outage.  Keep in mind, though, that if the big guys are having trouble responding to this type of attack, your site might be completely offline for days.  If you’re running an instance of bind (one of the most popular DNS servers) in a closet somewhere, it will work fine most of the time.  But it won’t stand up to even a small DoS attack.  Bind also has occasional security vulnerabilities that need to be patched — if you’re running your own server, you need to be sure that you’re updating it regularly.  There are companies that build DNS infrastructure for a living (such as Dyn, Neustar, and recent entrant Amazon), and their infrastructure is globally distributed, fast, and resilient to attack.  Take advantage of their expertise.  Using an outsourced service will often come with benefits beyond resiliency in the face of attack — you’ll also gain speed and access to advanced features like global load-balancing.

Second, make sure to work with a provider that is transparent in their operations.  It’s still early, but I’d be a bit surprised if AT&T issued a thorough postmortem report on this outage, as we’ve come to expect from folks like Amazon (example)  and Google (example).  Look for providers that admit to their shortcomings, and provide clear information on how they plan to address the issue going forward.  Avoid providers that are silent, or even worse, that loudly blame everyone else (examples withheld to protect the guilty).

Finally, work with more than one provider.  This is true for nearly every cloud or infrastructure service, but especially important for a service as critical as DNS.  If one provider is under attack, you’ll be glad you have a second.  Note that DNS doesn’t lend itself well to having a “cold standby” secondary provider, as DNS answers are often cached for 48 hours.  Switching to a cold standby may leave you offline for that whole time.  Best practice is to use two live providers, and keep information in sync between the providers (either manually, for DNS configurations that don’t change often, or via APIs or a “silent primary” for those that are more dynamic).

Attacks on DNS will continue; they’ve become a cost of doing business on the Internet.  Make sure you’re prepared, or you might see your site disappear from the Internet, with few options for speedy recovery.