Nobody Was Looking

There’s a well-validated truism in security incident response: the evidence is always there. I’m regularly involved in investigating cases where computers have been compromised by attackers, ranging from opportunistic hackers, cyber crime gangs and even nation-state sponsored espionage.

Within the past couple of weeks, I’ve worked on a case where a company discovered some servers which had been compromised since 2015. The servers were hosted in a cloud environment, and questionable administration practices encouraged direct access over the Internet, relying only on a simple username and password. Attackers simply tried guessing passwords, over-and-over, day-after-day, until eventually found the winning combination.

Once they had gained access to the server, the attackers installed software that allowed them to use the server to target others on the Internet in what’s known as a Distributed Denial-of-Service (DDoS) attack.

So why had this gone undetected? The servers themselves were logging the failed attempts to login. They had recorded the exact date and time when the attacker successfully logged in. They even logged the fact that one of the attackers tools had crashed. The servers even attempted to email these logs to an administrator, but email was not setup properly when the servers were created, so the mail went nowhere.

The attacks were missed, because nobody was looking.

Modern-day IT systems are complex, and have many components. Detecting attacks is orders-of-magnitude easier when there is something looking at all the log data being produced by these components. The security industry has long had a solution for this: a Security Information & Event Monitoring (SIEM) system. The idea is simple: send all your security-relevant data to the SIEM and let it work out when something bad is happening!

In the case I’ve mentioned here, there was no SIEM. This is true in many IT systems, especially those in small business where the cost of SIEM solutions, both the tools themselves and the people to run them, can be excessive. Think about the servers running your web site; the systems running your accounting software; your own computer... who’s monitoring them?

SIEM solutions have their place, but there is often a lot that can be done for a fraction of the investment. Newer technologies like Elastic help store, organize and query large pools of data. This makes it easier to build a ‘single pane of glass’ to look at all the security information that computers and applications make available.

The old adage “You can’t manage what you can’t measure” also applies to security. If you lack a capability to see what your IT systems are doing, it’s possible you’ll miss hackers being in your computer systems for years, too.