Event Management is Tedious, but it’s a Must Have – Here’s How You Can Keep Pace

Success in today’s digital business world demands a screaming IT infrastructure, providing reliable 24x7x365 access to a wide variety of applications, business services and data sources.

As you know, the modern-day IT platform is extremely diverse, with a blend of highly virtualized environments and public and private clouds. Corporate and line-of-business users are compounding the challenge by demanding a consumer-like experience from their handheld and mobile devices.

Fortunately, IT event and log management software systems are available that provide capabilities for anticipating and preventing service-impacting conditions. They also quickly detect and identify the causes of exceptional conditions when they do occur.

In fact, you can’t do without well-integrated event and log management. It’s a must-have. Everything in your enterprise’s IT environment – servers, firewalls, network equipment — generate logs about their activities and the events they process. These logs are stored on local and remote servers and can total in the terabytes.

There are a couple of drawbacks, however:

• Event logs create a tidal wave of information that is impossible for your staff – however sharp and dedicated they may be – to absorb, analyze and react to meaningfully.
• The rapid adoption of software-defined networks, and the rapid, often automated, deployment and provisioning of virtual servers has made the task of ensuring that you are monitoring everything in your network all the more complex.

Sometimes Simpler Is Better

This is where today’s event and log management applications can provide a solution. Given that this is a tedious area of technology, it may be tempting to consider outsourcing, since a fair number of vendors have an offering in this market. But they are only as good as the tools that they use.

Proceed with caution on outsourcing. Coupled with concerns of privacy and information security, which will restrict how much of your operation you can reasonably outsource, the cost of outsourcing might exceed the productivity benefits you could potentially achieve.

Whether in house or outsourced, your basic event and log management tools may only need to provide the following simple, basic features:

1. Log Management. Your log manager should be able to:
   • Provide both agentless and agent-based log collection
   • Aggregate logs from heterogeneous sources (Windows systems, Unix/Linux systems, applications, databases, routers and switches)
   • Provide an easy way to retain logs
   • Provide syslog management
2. File Integrity Monitoring. Your log manager should support real-time file integrity monitoring (FIM) by protecting sensitive data and meeting compliance requirements. You can now centrally track all changes happening to files and folders, such as when they are created, accessed, viewed, deleted, modified, renamed and much more.
3. Log Analysis. Event correlation enables user-defined rules and scripts to correlate events based on threshold conditions or anomalous events and notify in real-time for any threshold violations or network anomalies. 
4. Log Forensics. Today’s log forensics tools leverage powerful search functionality on both raw and formatted logs and instantly generate forensic reports based on the search results. This allows you to pinpoint the exact log entry that reported – or caused – the anomaly. In a security event for example, you can find the exact time at which the corresponding event happened and determine what initiated the activity – while isolating the location where the activity originated.
5. Real-Time Alerts. Real-time alerting allows your NOC staff to respond and escalate appropriately, if necessary.
6. IT Compliance. Whether your compliance requirements are dictated by PCI DSS, FISMA, HIPAA, SOX or GLBA regulations, all current-generation software solutions should provide the necessary reporting by populating the appropriate templates for you.

Security Alerting and SIEMs

While these tools were originally intended primarily for compliance and log management, given the comprehensive data collection that they entail, it has been a natural progression that they also become the aggregation point for security alerts.

Solutions in the security information and event management (SIEM) space take the basics further and add functionality. They analyze security event data and network flow data in real time for internal and external threat management. They collect, store, analyze and report on log data for incident response, forensics and regulatory compliance.

Improving threat intelligence and security analytics are where many of these resources are being focused. As you can imagine, early breach discovery requires effective user activity, data access and application activity monitoring – all in real time.

Organizations began using SIEM tools to detect and investigate attacks — but with limited success. Log-centric SIEMs make it difficult to detect and investigate today’s complex threats in a timely manner because they don’t provide full visibility across an enterprise.

What to Look For

The ability to incorporate multiple types of data – logs, packet, net flow and endpoint – gives a more comprehensive view of the overall infrastructure and trouble spots.

Top-tier SIEM tools can:

• Allow input of essential business context that, combined with automation and machine learning capabilities, help pinpoint and respond definitively to the threats that matter most.
• Provide automated thresholding and exception analysis of operational conditions during log analysis. Today, machine learning is being used to establish baselines and dynamically adjust thresholds, reducing the likelihood of event storms or missed events as your IT infrastructure scales.
• Identify threats from various analytics vectors including rules, threat intelligence, malware analysis and user and entity behavior analytics (UEBA) to provide truly sophisticated threat detection.

Careful analysis of your needs, with awareness of deployment challenges, will lead to the appropriate solution for your IT organization. Everything generates logs these days. With more devices generating more data, only through the adoption of a satisfactory SIEM can you be confident that you have full visibility into your infrastructure and are in a position to act on any issues that might arise.