the cj outage...

[V. Barbarino]

i know it's not your fault, by my rinky dink wesite has a back up from my host thousands of miles away.. one goes down, the other is up and running in mins...

seems your host sucks...

[Guest endymion]

Don't jump to conclusions, we are actually at the best host that I have ever encountered. Trust me I have encountered many. This is what happened, just announced by the team responsible for fixing the problem:

Dear Everyone,

Our apologies for the repeated outages today. It turned out our router problems _were_

caused by a DDOS (distributed denial of service) attack aimed at a site we hosted (note past

tense). It was a pretty massive attack, more than doubling the amount of inbound traffic we

normally receive.

As a result, our routers were overwhelmed. Due to the nature of the attack and the

circumstances surrounding it, we were not able to immediately trace down the cause of the

problems. We have now blocked the target's IP address and are keeping a close eye on our

inbound traffic so that we can act quickly to prevent this from impacting customers any

further.

We are also in the process of adding new routers and upgrading our existing routers to

provide more capacity so that in the future such attacks will be easier to withstand - so that

we are better able to react to similar attacks and take action before wide-scale problems

develop. Due to the unprecedented scale of this particular attack, we were caught

unprepared and for that we apologize.

If you have any questions or concerns, please do not hesitate to contact us at

support@dreamhost.com or via the web panel. Due to the large amount of correspondence

we have received from customers we are experiencing heavy support loads, but we will try

to get back to you as soon as we possibly can.

Sincerely,

The DreamHost Distributed-Denial-of-Sleep-Team

Basically somebody hit a router that happened to also serve us, trying to kill a site. The backup router took over and we came back up for a bit, then the ongoing attack killed the second router also.

A DDOS attack, for those not familiar, is when you take a bunch of zombie systems that you attack prior to the DDOS attack. You usually hack those with worms and viruses and trojans. Then you take all of the zombies and point them at some specific router all at once and just completely try as hard as you can to overload the router using all kinds of wicked tricks. It kills the router and knocks somebody's site offline, with no purpose other than just to take somebody out. We were not the target, somebody else in our network neighborhood was. The fallout took down us, my company, our attorney's web site, and lots of others.

Other notable DDOS attack today: http://www.riaa.com, the RIAA. As of this message they are still down.

[Guest pod]

An explanation of DOS and DDOS attacks.... 8)

[AndrewChibale]

i hope i never experience something like this on my website ..

i pay good money for it

[Guest pod]

Well, no one will likely target your site specifically. They may target your host, and there's not much you can do if that happens...

[Guest endymion]

i hope i never experience something like this on my website ..

i pay good money for it

Happens to everybody eventually, not much you can do about it.

[AndrewChibale]

Bellsouth better have some good firewall thingie going on ..

[Guest pod]

You host your site with Bellsouth?

Seriously, I'd advise moving it.

[AndrewChibale]

You host your site with Bellsouth?

Seriously, I'd advise moving it.

i never had any problem in the past ..

does it have such a bad reputation ?

[Guest pod]

Well, if the way they handle their ISP end is any indication. Plus, I don't like to keep all my eggs in one basket anyway.

[Guest endymion]

CoolJunkie is at DreamHost. They are a very good provider. The problem was not their fault. I am personally very happy that they got it back up without me having to deal with it. I trusted the team at DreamHost to handle things before this hacker attack and I still support them now.

[AndrewChibale]

their DSL access is the best in miami ..

thats why i kept both services .. besides, i also save some money

[MyIbiza]

Well, if the way they handle their ISP end is any indication. Plus, I don't like to keep all my eggs in one basket anyway.

Good point! Especially when dealing with internet and web hosting services.

[Guest pod]

their DSL access is the best in miami ..

Not even close. Try providers like Covad, Snappy, or COFS and then come back and talk to me. Better uptime, overall performance, and they usually give you what Bellsouth considers a "premium" router, as their standard router.

BTW, I used to work in DSL services (consultant/field tech), so I kinda have a background in it. 8)

As for saving money, sometimes spending a few bucks extra is worth it in the long run. I'd rather have someone whose sole business is the safety and upkeep of my website, rather than an all-in-one provider.

[AndrewChibale]

Not even close. Try providers like Covad, Snappy, or COFS and then come back and talk to me. Better uptime, overall performance, and they usually give you what Bellsouth considers a "premium" router, as their standard router.

BTW, I used to work in DSL services (consultant/field tech), so I kinda have a background in it. 8)

As for saving money, sometimes spending a few bucks extra is worth it in the long run. I'd rather have someone whose sole business is the safety and upkeep of my website, rather than an all-in-one provider.

Covad DSL

1.5 down

384 up

$64.95/month

keep in mind i only pay $39.99 for my access, and i get 1.4Mbps download (out of the 1.5 they all claim) and 256 kpbs up

more than enough for me .. and i dont need an static ip address either, i dont run a server or anything related (but still keep my pc on all the time)

btw, i couldn't get a hold of the other 2 companies u said about, and since u said u have a "background" on it .. it will be interesting to know a little more about that service .. i still wanna look for other companies and see what they have to offer

as for the "outage" you guys had .. damn i hope it doesnt happen again .. i like this place better than (you know where)

[Guest shannon_coolj.]

*yawn*

:

anyway, so cooljunkie went down

...now it's back up! problem solved

[Guest coach]

their DSL access is the best in miami ..

Nobody is safe from denial of service attacks, especially for only forty smackers. In any case, if you would like to do some research on your choices, check www.broadbandreports.com. Lots of good info there.

Thank god CJ is back up, though. I can't really get through a Monday without it.

[AndrewChibale]

*yawn*

:

anyway, so cooljunkie went down

...now it's back up! problem solved

should i really quote this ?

[Guest pod]

must not have looked too hard then.

www.snappydsl.net

www.cofs.net

are one and the same nowadays. COFS used to be independent but got bought out. BTW, COFS/Snappy is a local company. Keep the dollars at home, not feeding the fat cats at Hellsouth.

Now maybe all that you need is what BS gives you. I didn't say anything about price. With BS, you get what you pay for...$40 gets you best-effort service...they say 1.5 down 384 up, but hell, you may only get half that, depending on how shitty the wiring is, and so forth.

all of Covad's plans, while more expensive, give you what is known as a CIR, or Committed Information Rate, which means you get the speed advertised...at least from the DSLAM to the CPE. (switch to your house)...that being said, because things are so variable online, depending on the website, etc...If they cannot deliver it, they offer you a rate based on what they can deliver. Bellsouth? You don't like their DSL, they try and upsell you to a $1000/month T1 service. They don't like DSL, it cuts into their T1 profit margins heavily...they only have DSL since a lot of people want and need it...they got all pissy when telecom got deregulated in 1996...

Problem solved? For today. DDOS attacks are a reality of the internet. Kids with no lives and nothing better to do set these up, making life hard on everyone. Call me a bastard, but I'd love to find whomever did today's deal and throw them into a bathtub full of nails.

[V. Barbarino]

pod, they are not the reality of the internet. They are the reality of hosting companies who admitted they were not ready.... Call me crazy, but I wouldn't host with someone who admitted they weren't ready.....

since they weren't attacking your site, a backup should have gone online the very second their servers went down.. The attack was on one site, therefore with a backup, you'd have zero problems...

[Guest endymion]

since they weren't attacking your site, a backup should have gone online the very second their servers went down.. The attack was on one site, therefore with a backup, you'd have zero problems...

Our data center is multi-homed at three different major providers, so that if there are any problems on any one line, our data center is unaffected. I have been in that data center since 1998 with various other businesses and sites and it works. What happened yesterday is that the attackers used the zombies to alter the routing table in the router that connects DreamHost to Level3. They didn't just kill the router, they screwed with it so that it screwed with all three connections. We were connected the whole time but data packets to and from us were getting duplicated and routed all over the net randomly. The disruption was so severe that it hosed big portions of Level 3 and increased overall latency to Level 3 in general by almost 15% all day. An East Coast backup would not have solved this problem, we would have gone down just exactly the same way we did, for exactly the same amount of time. It was a serious attack that worked, all there is to say about that.

[Guest pod]

That's the deal with the attacks, just having an automatic backup doesn't solve the problem.

These attacks work by having various "zombie" computers (read the article I linked) barrage the site or provider with garbage data. Thousands of systems spewing out hundreds of megabits of garbage at one or two pieces of hardware, of course something will fail. And when the automatic rollover to a backup kicks in, the garbage data gets redirected.

The only solution really is to have more active monitoring, and more aggressive responses to these sorts of things.

It's hard though. The first D in DDOS means distributed, which means that there's no central point of attack, and it is that much harder to find who initiated it.

[Guest endymion]

These attacks work by having various "zombie" computers (read the article I linked) barrage the site or provider with garbage data.

In our case it was unfortunately not garbage data. I'm still getting details but they apparently reprogrammed the routers as the DDOS junk packets were flying. DreamHost blocked most of the junk traffic immediately as the backup router did its automatic failover. We came back up for a bit. Then they attacked the second router the same as the first and it became a zombie also. It took several hours to get the first router's firmware restored apparently, so that it could be put back online.

To clarify, 100% of our traffic does not flow through that one router. There are three different routers, each with an automatic failover backup, providing routing to three different major networks at a physically secure data center at a very well-connected NAP. If you do a traceroute to www.cooljunkie.com from your location three different times, you will most likely end up seeing three different packet routes. All of the precautions are in place to automatically defend against failures, and those failovers have been happening flawlessly for years. This attack worked because the attackers used the comandeered routers as part of the attack instead of simply knocking them offline by overloading them like you normally see. They were configured to actively send duplicate packets in random directions, which interfered with the other routers that were working normally.

A nice touch through all of this is that none of our mail bounced or got lost the whole time even though the mail clusters were going haywire because of the problem just like everything else was.

[ryan2772]

good lord i'm confused. ???

[Guest endymion]

good lord i'm confused. ???

The short version of the explanation is that you don't have to worry about it and it isn't likely to happen again any time soon.

Our outage yesterday was the equivalent of a business near ours burning down and the whole neighborhood full of businesses being inaccessible while it was going on. Or maybe more like somebody burning a neighboring business down on purpose.

I have had lots of sites and lots of business at DreamHost for six years. This is the second major outage, and if I recall correctly the first one was my fault. Not a bad record and I don't fault our IT staff. If you want to pick on somebody then pick on the hacker who did this.