Every week the AT&T Chief Security Office produces a series called ThreatTraq with helpful information and news commentary for InfoSec practitioners and researchers. I really enjoy them; you can subscribe to the Youtube channel to stay updated. This is a transcript of a recent feature on ThreatTraq. The video features John Markley, Principal Technology Security, AT&T, Manny Ortiz, Director Technology Security, AT&T and John Hogoboom, Lead Technology Security, AT&T.
John H.: Hey, John, I understand you're looking at a story about a kind of long-forgotten networking protocol that's still being used, and maybe has some vulnerabilities?
John M.: Yes I am, John. Back in summer, a company called Armis produced a security vulnerability, and prepared the report about a product called VxWorks. They did the due diligence. They did a great job of reporting this VxWorks vulnerability, and didn't think anything more than VxWorks was impacted.
They got a call from a hospital that their infusion pump had this vulnerability that they had identified for VxWorks, and they were a little confused. The thought was, "Well, this isn't VxWorks." What they discovered was is that a lot of these IoT devices are more like internet control systems and the real-time monitoring type tools that are out there are using a TCP/IP stack called IPnet that was developed 15 years ago, by a company called Interpeak. What they found is this vulnerability that Armis discovered is prevalent in every instance of this IPnet protocol. If you think back to the old days - internet control systems, doing real-time monitoring, didn't always have TCP/IP connections. But as we know, we're connecting all of these devices up to the internet, where we want them networked, and so this IPnet filled a void for those real-time monitoring systems.
But this company's now gone. The code is not supported, but this library for the IP stack is still present in many products. People just keep re-using this code, and it's not patched, it has a vulnerability, and it is in serious, critical systems like patient monitors, security cameras, and infusion pumps (as in this case). There's really no fix for it, because this company doesn't exist, nor does any support for this product exist. It's kind of scary, and I think of this as zombie code - which is appropriate for this time of the year in that we have these libraries like this one, but also things like OpenSSL, and some of the Adobe stuff. Libraries that either are older, or non-supported, and they’re included in products, and companies don't think to ever patch them, because while they may patch their product, they don't update or patch the library.
I think that at Defcon this year, they did a study at the Biohacking Village on this particular issue, and found that there are tons of IPnet vulnerabilities in many products. So I think it speaks to the issue that not only do we have to keep our products up to date, but we also need to check and make sure that the libraries they use are also up to date.
Manny: In this case, when you look at this thing, you have to figure out who ultimately owns this problem. You have to look at this manufacturer that decided that they were going to go off and use some of these pieces of code that are readily available and serve the purpose, which is the big thing there. So now, who owns that responsibility? I mean, I would think that if you decided to include that code into your device that you've now, in essence, bought into it, right? You now own it. You now have to go figure out how to keep this code up to date.
John H.: Right. All alternative that is actually still supported that can meet the same networking communication is what you do there.
John M.: But so many of these systems in this particular issue are real-time operating systems or real-time products, and they tend to be in manufacturing and medical healthcare...I had a job interview many years ago with a company in this space and recall the technology is very niche, and not always the best supported.
John H.: Right. Yeah, and that's the thing for all the embedded and other devices deployed out in the field with this software package. Even if there was a patch for it, there probably isn’t a good means to deploy that patch. Same with a lot of legacy equipment of these embedded things, even the routers you were talking about earlier - sometimes they don't have very good patching processes, or patching processes at all...
John M.: Can you take down hospital equipment, to patch - like this ssue with this hospital infusion pump, can you take the pump down to patch it for an extended period? What happens if the patch doesn't work right? Do you have a backup?
Manny: Yes, those are definitely risks that you run, and like you said, the patching process becomes totally ad hoc. You can't just go and just say, "Oh, yeah, on this night, when the hospital is busy, we’re going to push updates on all of them. You can’t.
John H.: You probably don't even know where they are.
John H.: And there's no means to do that, even if you did...
Manny: You’d have to have some massively coordinated effort, where you have people physically running around hospitals with thumb drives, right?
John H.: Right, doing it one by one. Right.
John M.: The thing is, internet control systems and real-time systems may have this, and the networking that connects to them is not always the most stable to begin with, so there's a lot of work that is done with thumb drives and direct access.
John H.: I guess they're going to have to figure out what to do here. My thought would be you have to come up with a new product and probably replace a lot of this tech that uses these legacy libraries, unless somebody out of their good nature of their heart is going to figure out a way to patch it all. Like write a patch for those libraries, and then deploy it out of the goodness of their heart to the world.
Manny: If you're running these type of devices, you should be running code reviews against them on some periodic basis - doing some sort of an audit. We're talking about code review audits, where somebody is literally reviewing code line by line, and making sure that they understand, "What's inside this code? What kind of libraries is it actually using?" And ensuring that there's basically a table of contents of all these different libraries the code contains, and we should be paying attention to whether patches are being shown for these particular libraries.