There are plenty examples of where Cyber Security failed and how expensive that could be. It is good that we point out these problems, especially when they could have been prevented. And there is another reason to expose Cyber Security issues and hacks. As a recent poll among Twitter users shows, trust in organizations and companies is low. Even so low that the participants expect that being exposed by hacks will have the biggest impact on how companies and organizations improve their cyber security… Read the poll results and comments here.
There is however a lot of good performance in the field of Cyber Security, even for the Industrial Legacy and here are some examples of doing the right things the right way and preventing a lot of costly damages!
Print server phone home
What appeared to be an advanced automated continuous monitoring system was in reality a lean Linux box and a bunch of smart scripts and tools, completed by a fast and straightforward web interface. These scripts and tools collected network information from different sources, were designed to separate normal from abnormal, and created push alerts for whatever was identified as abnormal. They called it Big Brother and BB was created and managed by a small team of enthusiastic network administrators.
People get tired of repeating non-alerts and then stop paying attention to all alerts. Even to the real alerts! This team learned that quickly and made sure to always fine tune the alerts and scripts. It was their Friday afternoon ritual to analyse the alerts of the past weeks for accuracy, compare data from different sources, and optimize their scripts accordingly. Sometimes they concluded that an alerts was only generating nonsense, sometimes they found better ways to detect anomalies. They’ve been doing this for a couple of years and kept their alert system lean and accurate.
Suddenly a strange anomaly popped up. Printer server devices within the network tried to create a SSH connection (which is encrypted!) to an outside IP address twice a day. What made it even more concerning is that those printer servers were installed throughout the company. In the factory, in the design and engineering departments, in the finance department, even in the conference center. Just everywhere!
The team noticed this because they setup the firewall to block all outgoing traffic from the network and only allowed specific channels and connections. The further down into the factory and technical systems, the less traffic was allowed. It was a lot of work, and it did create a lot of frustration among the users from time to time, but they kept this policy alive because they knew this was they only way to keep the network clean and under control.
BB, the network monitoring system, created alerts for attempts to outrun the rules and restrictions. And that is how the team noticed this anomaly. Why is a printer server device trying to make connections outside the network? OK, the connections were blocked, but what would it do when not? And who was it trying to connect with?
“Failed unauthorized attempt to create encrypted communication, target and content unknown” was the title of the ticket created by the team and that ticket lead to the regional CIO giving me a call after internal escalation to him.
“I’m not sure how to handle this, can you have a look at it? When I look at the ticket I should create a red alert so to speak, and get the corporate security guys involved. But shouldn’t we figure out what is happening before we alert the whole company?”
“Let’s take one step back. Your team detected a serious issue, possibly a breach of security protocols, do you agree with that?”
“Yes, but I don’t want to push all alarm buttons over what could be nothing. First I want to be sure this really is an issue and not some mistake.”
“Your team tells you this is an issue and wants you to escalate it. Are you telling me you don’t trust their expertise?”
“Of course not! But could you have a chat with the team before I decide?”
“Sure, I will look into it and let you know within the hour.”
The team had already done their homework. A quick check of the maintenance journal for the devices made clear that colleagues in the IT-Department had installed official firmware updates the week before. On online search showed some reports about an undocumented “feature” in the new firmware which tried to collect and sent meta data of printed documents to the vendor.
The part I liked most about their approach is that they didn’t wait for upper management to decide. There were procedures in place and this was a confirmed breach of security. Step 1 was to take all printer server devices offline and assign a team to figure out the quickest solution to get the shared printers back to work without the printer server devices. Step 2 was to inform the IT managers of all other factories and offices, with the urgent recommendation to take the printer servers offline until further notice.
The conversation with the still doubtful regional CIO was more efficient this time.
“Your team has followed procedures and taken appropriate measures. This should be handled as a serious security breach. Push it as critical until all devices are checked and taken offline.”
“Is it really that bad? Are you sure it is not a mistake?”
“Let me put it like this. If I was your boss, I would already challenge you now for not escalating this . It is also important that you give your team a clear signal that you take this very serious and that you fully support them. That signal must come very soon, now would be a great time!”
Wheels were set in motion, all the right people were informed and started to do their part. Other devices from this vendor were discovered and temporary decommissioned. The purchase department blocked the vendor in the system so no other devices could be ordered pending the investigation and final decision. Internal Auditing involved an independent certified network security expert to support the corporate and local network teams with their checks of logfiles. The legal team prepared a letter to the vendor demanding a formal written statement about the findings.
At first the vendor denied, then the vendor tried to explain this was just for service purposes, and finally the vendor provided updated firmware for the various devices “for selective customers only”. We assume that the vendor meant the customers who discovered that their data was being tapped.
My client wasn’t interested in any updates, all devices from this vendor were replaced immediately and the vendor was blacklisted. The network team got a big compliment for their disciplined and accurate work. BB, once a local project, is now a global corporate project and the team is very proud of that.
Significant detail: this factory was producing highly confidential parts for the defense industry!
Cyber Security is not just about keeping the bad guys out. Cyber Security is also about critical thinking and finding the flaws before they cost real money!
- Back to the Future Cyber Security – A manifesto for Cyber Security and the Industrial Legacy
- Back to the Future Cyber Security – All updates