CaryCitizen Hit By Server Mishap

Story by Hal Goodtree, Publisher of CaryCitizen. Picture from Skullbox.net.

Cary, NC – If you’ve been wondering where CaryCitizen has been for the past few days, the answer is sure to bring a shudder to many in the local IT community.

My Bad: I Paid for an Upgrade

For several weeks, we’ve been contemplating a server upgrade. Business is good and we need more capacity.

Some companies run their own server farmers, but the vast majority use commercial hosts. For almost a decade, we’ve hosted many of our sites at GoDaddy.

On Thursday, I called to order the server upgrade. GoDaddy assured me they took backups and mirror images of the server and that the upgrade was perfectly routine. In fact, we’d upgraded about a dozen times over the last few years without any problems.

Two hours later, we were shooting blank pages into cyberspace.

Recovery

Without getting too technical, the issue was that GoDaddy failed to connect the MySQL databases to our websites. These databases hold all the content for our sites.

If you visited CaryCitizen during the outage, you probably saw the text “Error Connecting to Database.” Those have to be the four scariest words in the English language for any digital publisher.

Fortunately, we keep our own backups of all our websites.

On Monday, GoDaddy reattached the MySQL databases. We were able to recover 8 out of 10 clients’ websites on the affected server that day. The final two websites and CaryCitizen were back online on Tuesday.

Our Great Clients

We informed those clients who were impacted last Thursday when the outage occurred and updated them until service was restored.

Many clients wrote to express support. Not a single one fired us :)

Likewise, CaryCitizen advertisers were also very understanding. Special thanks to Associate Publisher Lindsey Chester for communicating with our all our valued advertisers during the crisis.

To Our Readers

Finally, what can I say to our readers? I can only offer my deepest apologies if our service outage inconvenienced you in any way.

Some pictures and internal links in old stories may not be fully recovered yet. As well, we’ve had to republish the last few stories we’d posted before the outage. We’ll continue the cleanup over the next few days, but by-and-large things are back to normal.

We received many messages of support from readers and contributors and they were all greatly appreciated.

What Did We Learn?

As far as alternate servers and back-ups, I think we were we’ll prepared.

But looking ahead, I’d like to make better use of our other channels, particularly Facebook and Twitter. CaryCitizen is an amalgam – a mashup if you will – of content from many places, not just one server. We have a lot of channels and I’d like to be better prepared to switch to a back-up plan if GoDaddy borks it again.

Yes, we’re still with GoDaddy. In ten years, it was the first major outage. And breaking up is hard to do.

As for our new server, it’s four times faster than our old one. In fact, it’s not even an individual box anymore. It’s a grid server, a cloud of computers that rain torrential streams of data whatever the load. Or anyway, that’s the theory. Let us know if you think the site is faster (or not).

Just in case, we added another server to our cyberfarm. This one’s from a company in Albuquerque called FatCow. They run their servers from wind power.

Bottom line: It’s good to be baaaaaaack.