A customer of mine had their own root server with 1and1. I got a call to investigate why none of their websites were working. After a bit of investigation, I realised I was looking at a half re-installed machine. It turned out that there had been a media failure on 1and1’s cluster thing where they keep their root servers, so 1and1 had replaced the faulty disc and kicked off a system re-install for all root servers that had space on the faulty drive - WITHOUT EVEN TELLING US. 1and1’s re-install didn’t work properly so there was bits of the new drive and bits of the old drive showing through. Of course we didn’t get the whole story straight away. They tried to blame it on us for the first few calls my customer made and it was only after I had pieced together what had happened that they actually got anywhere. That was with the hosting people, and a story like that just wouldn’t be complete without a 1&1 Customer Experience horror story so this is one of mine: http://www.bigsoft.co.uk/blog/index.php/2009/06/10/1-and-1-worlds-worst-customer-support