For once I actually have a decent excuse for not having posted lately — work has been completely hectic, and to be honest I’ve felt totally drained.
A couple of weeks ago, in the early hours of the morning, there was a big power outage at the data centre that’s home to our main server (and this site). Unfortunately, rather than the UPS taking over until the backup generator kicked in, something fried/died, and it took the hosting company — DSVR — nine hours to get everything up and running again.
Even once DSVR had got all the servers back online, web and FTP were crawling, and mail was totally dead (something on the internal system was utterly screwed, quite possibly related to DNS).
The problem meant that not only were we without our main email accounts for the best part of two days, but so were over a dozen of our clients. As you can imagine, this didn’t go down too well, and when we weren’t sat watching the queue of emails growing in Exim, we were trying to placate clients.
Luckily we have our .com domain mail server hosted on a completely different server, for contingencies such as this, so at least we had some way of getting emails out. Nevertheless, it took until after the weekend for the backlog of emails to clear from our main server.
It wasn’t always like this with DSVR; when we signed up with them some years ago, we chose them because they had what turned out to be a well-deserved reputation for excellent service. Yes, they were more expensive than some, but the good up-time, versatility and great support was well worth the money. Support tickets were answered quickly, and the staff took an active role in discussions on the customer forums.
On the rare occasions when there was a problem with the service, DSVR’s status page would contain detailed explanations of the problem, and the steps being taken to rectify it and ensure it didn’t re-occur. The support staff would also happily discuss the problem in the forums, often with a refreshing honesty.
Sadly DSVR were bought by Legend Communications (formerly Business Serve, subsequently bought by Thus) in 2004, and the service has been going downhill ever since. Earlier this year there were big problems when they moved data centres, and despite lots of promised improvements, things just seem to be going from bad to worse. There aren’t any of the original DSVR staff left either.
Leaving aside the point that a power outage shouldn’t be able to screw up the servers for a couple of days, to add insult to injury the communication from DSVR/Legend/Thus has been almost non-existent. Little more than “Blah, blah, power outage, some customers experiencing problems with email, engineers working to restore service as soon as possible” on the status page for a couple of days. Certainly nothing like a decent explanation.
Rumour has it, because Legend/Thus are a PLC, any communication (including the status page) has to ‘go upstairs’ for approval by management in case if impacts on share price. Hoorah for capitalism!
Legend weren’t any more forthcoming when the Register reported on the problems:
They gave us a standard “we apologise for the inconvenience”-type response which, after days of their customers losing business, is unlikely to smooth relations.
They added that the email issues were caused by high volumes of traffic. Now, we’re not systems engineers lads, but we thought your business was all about dealing with high volumes of traffic.
Indeed. Except that the email issues weren’t caused by high volumes of traffic, they were a symptom of it. The problems seem to be related to their DNS system getting hosed, which is pretty worrying.
Sadly this is the large bundle of straw that’s broken the camel’s back, and we’ve decided not to renew our contract with DSVR at the end of October. It’s a real shame, but with the unreliability of DSVR’s service reflecting badly on us to our clients, our loyalty has been stretched to its limit.
So we’ve now started migrating our clients to our new server, which means lots of extra work — moving databases and domains, changing secure certificates, getting new IP addresses, updating payment gateways etc.. We’ve been pretty snowed-under with work lately anyway, so the extra hassle of moving servers is something we need like a hole in the head (we haven’t had a weekend off for a few weeks now). Anyway, enough self-pity….
I was planning on switching from b2evolution to Wordpress soon anyway, so the move to the new server seems like as good a time as any to convert. I haven’t yet finished the new look for World of Badger, which does mean I might have to go with the default Wordpress look until I’m less exhausted. Do not be alarmed though, the drabness won’t be permanent.