In Praise of the Ultimate Survivor – The Humble Log File

The other day, I was talking to an acquaintance of mine who has been around the high-tech industry since the 1980s. We started to swap old war stories, and he told me about the time in the 1990s when he met with a supposedly visionary exec at a major US corporation. The exec was a fan of the latest technology for the sake of technology, and spent an hour or more explaining how “this old IP technology is going away” – and how “everything in future would run over ATM”. Now, my acquaintance was selling ATM gear at the time, so this was music to his ears – but even he knew that something didn’t add up.

Fast-forward 25 years, and we’ve seen the world fundamentally change. The web has evolved from a few pages at CERN in 1990 into the engine that arguably drives the world’s economy today. And, over all of that time, IP has been a constant. Sure, IPv6 is making some inroads and we shove IP into MPLS and DWDM behind the scenes – but it’s still IP, and it’s stronger than ever. ATM, on the other hand? It’s died a death, relegated mostly to a link layer protocol on DSL modems.

That got me to thinking about other technologies that just keep on going. SNMP is a good example – the first RFCs for SNMP appeared back in 1988. SNMP is still with us, it’s simple – the name says it all – and it works. Sure, security was appalling in v1, but v2 and v3 fixed that – and again, it’s still SNMP. There have been attempts to unseat it, but these have largely failed. CMIP is one example – a full seven-layer OSI stack that was really powerful – but also fatally complex. CMIP has also died a death – it never really got any traction in IT, and it’s now found in a few ancient telecom boxes that are waiting for retirement in your local central office.

And then, there is that ultimate survivor – the log file. Just like IP and SNMP, log files have been with us since time immemorial – and they’re still going strong. And, it’s for exactly the same reason – they are simple and they work. What could be easier to use than a plain text file that anyone can read? Log files are the ultimate testament to a simple fact: simplicity thrills and complexity kills.

Okay, that’s a bit over the top. But the truth is that log files are actually growing in importance – not diminishing. They may have started out as a basic way that developers could capture basic debugging information, but over the years we’ve found more and more uses for log file information. In fact, some would argue that log files are the next frontier of big data.

Here are just some of the reasons why the humble log file is never going away:

1. Troubleshooting IT issues – When an application stops responding or a server grinds to a halt, events can tell you what has happened. However, if you want to find out exactly why something happened, then log files are your best friends. Log files give you the detail you need to actually identify and fix an issue – they can even let you trace symptoms across your IT infrastructure until you pinpoint the root cause. That’s exactly why we make a point of collecting and analyzing log data here at Optanix.

2. Identifying configuration changes – Speaking of troubleshooting, configuration changes are responsible for more service outages than just about any other factor. How many times have you come in and found yourself wondering “what happened last night” when something doesn’t seem quite right? Chances are that someone changed the configuration without getting approval. If that’s the case, then configuration logs are usually the only way to identify the change.

3. Predicting future IT issues – No, log files aren’t a crystal ball. However, they can often give you an indication that something will go wrong in the future. That’s important, since this lets you take remedial action before any damage is done. By profiling your system logs, you can create a baseline view of normal log activity that happens all of the time. You can then use this baseline to spot anomalies – logs that don’t normally occur and indicate that something unusual is going on.

4. Compliance – Log files provide a reasonably complete record of who did what, where and when – assuming you turn them on. By analyzing these log files, you can create an audit trail across your IT environment, allowing you to demonstrate compliance with internal policies and external regulatory requirements. For example, your internal security policies may restrict access to certain systems – and logs can show you whether there have been any violations.

5. Security – Despite new IT technologies such as intrusion detection systems, security logs from firewalls, servers and other IT equipment still play an incredibly important role in uncovering and responding to security incidents. They provide the depth of information needed to get to the root of hostile activity, although the sheer volume of transaction information can be overwhelming. And, these logs are also an invaluable source of forensic evidence when things end up in court.

6. Web Analytics – There’s a whole industry growing up around web analytics – and, once again, log files are at the forefront. By analyzing web server log files, companies can do everything from optimize their website design through to measuring the results of marketing campaigns in collecting business intelligence. There’s just no other way to get these types of insights.

7. IoT – Okay, perhaps the Internet of Things is a little further out, but just think about all of the data that’s going to be generated by billions of connected devices – the sensor data alone is going to run into petabytes. The IoT is going to be the ultimate source of business, consumer and environmental intelligence – giving us unprecedented insights into the world around us. And, where is most of that data going to be held?

You guessed it – log files. I rest my case.