This has nothing to do with fiction, but I figure people ought to know. As I say below, I doubt this is aimed at me specifically.
Originally I intended to give them time to respond, but then I wondered: What would Bruce Schneier say about that? Sunlight is the best disinfectant.
Of course it’s possible that I’m being targeted specifically, though that’d be weird. And it’s possible that I’m misinterpreting the data/events below. And it’s possible that I’m making this up, for all you know.
So, with those caveats, here’s the text of a ticket I opened with Rackspace this morning:
Figured you guys might be interested to know you’re probably being hacked by someone with access to your datacenters.
A while back (maybe a month?) I suddenly couldn’t log in via SSH to either of my virtual machines. They both require a cert for login. I got an error for both VMs that said there was a cert problem. I subsequently contacted you guys via chat & was told you weren’t doing any sort of MitM stuff, which left either somebody local to me, my ISP, or somebody who has access to both datacenters. (Neither of my VM had been “touched” by me).
Simultaneously I got a certificate warning when trying to use your web console app to take a look at my mail server. I chose not to proceed…but somehow the console opened anyway. Since I don’t actually have anything all that sensitive on either VM anyway, I logged into my mail server via the console.
Now I’m a few thousand miles from my previous location. The mail server suddenly accepts my cert for SSH login again (I haven’t tried for some time). All appears to be well. I could choose to believe it was a mysterious glitch that fixed itself, but I think it’s significant that I actually provided a password via the web console, via SSL using what purported to be your certificate, and only then was I mysteriously able to log in via whatever method I choose.
Because there’s a new wrinkle as of yesterday (at least that’s when I noticed it): My web server (different datacenter) suddenly has a new “fingerprint” for its key. The record on my laptop remains unchanged, and is in fact “synched” to other computers via encrypted online storage, so it can’t be an issue on the laptop itself. I don’t see how this can be anything but a MitM attack, carried out via various methods over a period of weeks (at least). Since I’ve tried connecting via multiple ISPs, from locations thousands of miles apart, I can only conclude the attack is fairly likely to originate within your own network–though if it’s not, it means somebody with serious backbone access is targeting at least one of your customers. Which sucks even harder.
As for me? I used to do “secure” web app development for various corporations & startups, and I still play around with Tor and VPNs terminating in different countries (of which neither affects the results I get when attempting to log in via SSH, pointing again to something local to your datacenters) and various other goofy stuff…but I’m doing it all strictly as a game. I mean, I write fiction these days. Damn near everything about me is now public (google “David Haywood Young” and you’ll see what I mean), so I doubt anybody cares about “hacking” me or my VMs specifically. I’ll bet the issue(s) is (are?) affecting lots of your other customers, though, and not in a good way.
Are you already aware? Do you have a plan to fix it? I’m willing to talk to you guys first, but I’ll be blogging about it all fairly soon. I think people should know. ATM I have to assume anything in either the Dallas or Chicago datacenter is compromised.
More on their cert below. No details about mine are forthcoming. 🙂
Yep, sounds like a possible MitM to me. Both the State of Alaska and ANTHC tried to turn on an automated HTTPS MitM (Blue Coat and WebSense) which is only partially successful if they’re able to push out their own CA cert via group policy. Yes this is the same technique used by Iran (and now the NSA) to spy on its citizens. I have a giant rant on all the things that are broken with these corporate MitM cert spoofers if you’re interested. You can use a browser plugin such as Perspectives to be notified of spoofed certs.
Hey Oran! Didn’t know you read this blog. But yeah, serve me up a giant-sized rant. I’ve only gazed on such obnoxious devices from afar, and mostly looked away even so.
Meanwhile I got an unconvincing response from Rackspace, in which the guy focused on all the things it could be aside from a MitM attack, decided on the basis of experimentation/logic that they were unlikely, and concluded all was well in their datacenters. Which strikes me as both unjustified and unhelpful.
I mean…just for starters, if I were building/configuring such a device I’d only intercept traffic coming from -outside- the datacenter. That’d partly be ’cause it makes sense to do so from most perspectives I can think of, and also partly ’cause I might not have a choice (depending on architecture/access). So his playing around from the inside is interesting but far from conclusive.
Assuming he’s correct, though? I’m using AT&T at the moment but have had the same issues via Cable One and Comcast. And via VPNs terminating in Ireland, Canada, the US, Germany, and Latvia (because I like Latvia). It’s fairly likely that all of the above are traveling through the same pipe at some point, and I guess I could start doing traceroutes…but I wouldn’t trust the results and I’m not sure there’s much I could do about ’em anyway.
I’m very close to concluding that I should host my own data/servers literally in-house. As it is I do most of my browsing via Tor/Tails/Orbot, using XPrivacy on Android and also starting off with a variety of VPNs. And I messed with runlevels/scripts months ago on my Rackspace-hosted VMs so Rackspace’s “imaging” utilities don’t work (it’d also be very difficult for them to leverage “physical” access to get root). It just struck me that their ability to invisibly image a server and change the root pwd in the process was about the biggest security hole I could imagine, so I made sure they’d at least have to write a special script. I’ve even considered installing an OS on an encrypted partition that’d require me to log in and give a passphrase to boot it up, but if they can defeat my current setup they could easily compromise the outer non-encrypted device and set up–for example–a keylogger anyway. So it’d be fun but pointless.
At this point I don’t trust any variety of commercial cert, but have more confidence in my self-generated stuff. Which is being spoofed in a sorta clumsy way right now. So I’m very interested in your rant.
-D