Originally I intended to give them time to respond, but then I wondered: What would Bruce Schneier say about that? Sunlight is the best disinfectant.
Of course it’s possible that I’m being targeted specifically, though that’d be weird. And it’s possible that I’m misinterpreting the data/events below. And it’s possible that I’m making this up, for all you know.
So, with those caveats, here’s the text of a ticket I opened with Rackspace this morning:
Figured you guys might be interested to know you’re probably being hacked by someone with access to your datacenters.
A while back (maybe a month?) I suddenly couldn’t log in via SSH to either of my virtual machines. They both require a cert for login. I got an error for both VMs that said there was a cert problem. I subsequently contacted you guys via chat & was told you weren’t doing any sort of MitM stuff, which left either somebody local to me, my ISP, or somebody who has access to both datacenters. (Neither of my VM had been “touched” by me).
Simultaneously I got a certificate warning when trying to use your web console app to take a look at my mail server. I chose not to proceed…but somehow the console opened anyway. Since I don’t actually have anything all that sensitive on either VM anyway, I logged into my mail server via the console.
Now I’m a few thousand miles from my previous location. The mail server suddenly accepts my cert for SSH login again (I haven’t tried for some time). All appears to be well. I could choose to believe it was a mysterious glitch that fixed itself, but I think it’s significant that I actually provided a password via the web console, via SSL using what purported to be your certificate, and only then was I mysteriously able to log in via whatever method I choose.
Because there’s a new wrinkle as of yesterday (at least that’s when I noticed it): My web server (different datacenter) suddenly has a new “fingerprint” for its key. The record on my laptop remains unchanged, and is in fact “synched” to other computers via encrypted online storage, so it can’t be an issue on the laptop itself. I don’t see how this can be anything but a MitM attack, carried out via various methods over a period of weeks (at least). Since I’ve tried connecting via multiple ISPs, from locations thousands of miles apart, I can only conclude the attack is fairly likely to originate within your own network–though if it’s not, it means somebody with serious backbone access is targeting at least one of your customers. Which sucks even harder.
As for me? I used to do “secure” web app development for various corporations & startups, and I still play around with Tor and VPNs terminating in different countries (of which neither affects the results I get when attempting to log in via SSH, pointing again to something local to your datacenters) and various other goofy stuff…but I’m doing it all strictly as a game. I mean, I write fiction these days. Damn near everything about me is now public (google “David Haywood Young” and you’ll see what I mean), so I doubt anybody cares about “hacking” me or my VMs specifically. I’ll bet the issue(s) is (are?) affecting lots of your other customers, though, and not in a good way.
Are you already aware? Do you have a plan to fix it? I’m willing to talk to you guys first, but I’ll be blogging about it all fairly soon. I think people should know. ATM I have to assume anything in either the Dallas or Chicago datacenter is compromised.
More on their cert below. No details about mine are forthcoming. 🙂