Thumbs up when Cyber Security is done the right way! When Cyber Security is done the right way, we see a combination of a team taking their responsibilities serious, and a team which is using technology as part of the solution and not as goal. That is the only way to do the right things the right way. Hats off to all Cyber Security teams protecting their companies and organizations against Cyber Crime!
The network administrator at duty noticed an odd list of failed VPN login attempts at the service VPN entry point. Sunday evening, started at 22:00 and from the same IP address. The IP address traced back to the pool of what appeared to be a private internet provider. Normally this would be no big deal, it happens a lot. Port scanning junkies are part of every system administrators life but this one was different. Different because whomever this was, was actively trying to login to the VPN cluster and did so exactly five times, once every 2 minutes starting at 22:00.
What was happening here? A script kiddie who figured out this was a company? Or did someone in the IT team make a mistake? The admin wasn’t worried yet, the VPN cluster was controlled through an automated ticket system, and without an approved ticket there was no accepted VPN connection. All things done right the right way. Controlled VPN access, time restricted, source IP and target IP restricted, and always supervised. On top of that, a monitoring system that picked up attempts to cross the line.
Next Sunday, same time, same attempts. The week after that, same thing, so it was reported at the security review meeting. The source IP address changed every time but that is normal for German private internet connections. And it also makes it more difficult to track who was behind this, whatever it was. The log-files were scanned for more incidents involving these IP addresses. Nothing! It didn’t look like an attack yet, but it was something to keep an eye on. The team decided to increase the alert level for blocked attempts at this VPN cluster.
I got involved when the meeting the team had took a bit longer, and I was scheduled to join the next meeting with the team lead and his line manager in the same meeting room.
“Come in, we’re almost finished” the team lead said and I did.
“Care to brainstorm with us what this could be?” so I did.
“Always Sunday, always 22:00, always 5 attempts?”
“Looks like a script to me”.
“What makes you think that?”
Before I could answer, the other members started to pickup my line of thinking. Same time, same day, same amount of retries. Brainstorming all thinkable scenarios still didn’t bring us much closer to the answers we were looking for, but at least the defenses were working so far. The team agreed to set the logging level to verbose which basically means it will spit every little detail into the logs and terminal. Next meeting coming Sunday 21:30 to be there as it happens!
That Sunday, we got some surprised looks from the security guards when we showed in the evening. The plant was still relatively calm, the night shift would kick in soon to startup the plant. Tomorrow at 06:00 full scale production would resume but we only cared about 22:00 and the mysterious VPN connection attempts.
Point 22:00 there were 2 VPN connections. The first passed through without any issues, the second one appeared to be our VPN-bandit which immediately got full attention from the team.
“What’s with the other VPN connection?” I asked.
“Don’t worry about that one, that is our logistic provider picking up the inventory reports.”
“Is that connection also scripted?”
“Of course it is, this is their slot. Window opens every Sunday at 21:58 and closes at 22:08. They connect at 22:00 and need about 3 minutes to collect all data and send their confirmations. That is the regular connection, we are looking at the other connection.”
“How many other scripted connections are there on Sunday? How many at 22:00?”
“This is the only one at 22:00, we offset them in separate windows to keep the active connections at a minimum.”
There was some irritation in his voice because he wanted to focus on the VPN-Bandit and didn’t quite understand why I was even asking questions. His colleague jumped on my thoughts immediately and understood what I was thinking.
“Because we already drop the connection at the entry, we can’t see if he tries to use a certificate and what credentials he uses. Maybe we should allow him the first level to see what he’s up to, just to make sure this is not an orphan session or something like that.” he said.
“Why open the gate?” his colleague replied.
“When we block him upfront we can’t investigate further, and maybe we are looking at some orphan session. Just thinking, it seems a bit too much coincidence that he is fully in sync with the other session.”
“Do it, I don’t believe in coincidence! Hurry up, he has only 2 attempts left.”
It always great to see how teams learn how to use the magic words to get the attention of the other members. Commands were issued, settings changed. And even when time was running out, they kept the procedures. Log entries were made, change request approved, and changes validated before being pushed to the infrastructure. Impressive!
The 4th login attempt was made, data was logged and the connection was dropped again by the system.
Maximum allowed connections exceeded, certificate rejected.
“I told you this was an orphaned session! He even has a certificate!”
“Let’s not jump to conclusions. First undo the changes so we keep him out at the front door, and let’s analyse the data we collected.”
The data showed a very interesting picture of what was going on. The certificate was rejected because the source IP was not correct. The ID made clear that my client was the issuer of this certificate and the receiver was the logistics partner… Even worse, besides the source IP this certificate was valid! Someone had a valid certificate issued by this company! And that wasn’t even the biggest surprise. The credentials used to login were those of the logistics provider!
Someone somewhere had a script identical to the one used by the logistics partner, a valid certificate which was issued to the logistics partner, and even the credentials issued to the logistics provider.
Calls were made. The logistics partner wasn’t aware of any of this but issued an all hands on deck to investigate this. The IT-Director approved to block the accounts of the logistics provider and render the certificates invalid, which was executed immediately. There was even a procedure to handle communications and exchange of data when the main systems failed, so that was activated at both sides.
There it was again, the VPN-Bandit and this time the bandit connected perfectly without any issues. Data was exchanged, files were updated. I have not often seen so many surprised and shocked faces! Calls were made again. This time the IT-Department of the logistics partner called my client. And now all questions were answered at once…
“OUR FAULT! SORRY, THIS WAS ALL OUR FAULT! We tested the failover function last month and we didn’t recognize that the backup system remained active…”
Shock. Silence. Discussions. Confusion. Relief. Disbelief. Acceptance. Basically in that order and also some of it at the same time. There was one final question to be answered. Why did the backup system show up with dynamic IP’S and why was that certificate accepted? The logistic supplier had decided to use a mobile solution as backup for this critical connection. The moment the team was reminded of that, they also remembered that the certificate for the backup system was not IP bound for the connecting side.
It all made sense now for the team. When the logistic partner would activate the backup system for real, they would create a ticket and the IT team would process it. The main VPN connection would be deactivated and the backup VPN connection would be activated. Just like they did this evening. In the past weeks, both systems had been active and only one of them had authorization to access the VPN cluster.
When reading this you might think “so what?” but then you would be missing all the important Cyber Security steps in this process, and all the right things the team did the right way.
- The system administrator detected an anomaly which he didn’t understand so he brought it to the attention of his peers and team lead. Well done!
- The team couldn’t identify what cause the anomaly so they zoomed in on it.
- The team worked their way through this anomaly in a very disciplined manner, which reflects the discipline with which they always work.
- The team actively manages the technology in place, making it a solution!
- Most importantly, nobody on the team ignored this under the cover of “oh it will probably be nothing” which happens to be the start of most hacks which somehow showed up somewhere but nobody paid attention to it.