Logs, logs, the audit trail
The more your parse, the more they fail
The more they fail, the less they plunder,
So let’s have logs to avoid a blunder
Will Alexander, NCC Group, 2014
In the age of Big Data, organisations are able to retrieve and store events from all manner of systems on their networks. From intrusion detection systems to the CEO’s laptop, many devices have the ability to log granular information that can aid security staff in detecting or responding to an incident; but as the amount of data increases, so does the amount of effort to perform proper log analysis. To aid in this analysis organisations buy, or develop in-house, log management solutions. These vary, but successful ones share a number of common features.
Correlation
Arguably, the most important feature of any log management solution is the ability to correlate events, often from disparate sources. Correlation involves linking events together into meaningful bundles by looking for common attribute values, such as IP addresses, ports, usernames, and time stamps. This process of weaving individual events together to form a greater understanding of what is happening on a network is what ultimately helps security staff identify and respond to security incidents.
Every other feature discussed here ultimately aids in providing the ability to correlate events.
Time Synchronisation
All logging devices should be synchronised to one central time authority using NTP. This ensures that events occurring at the same time on different devices share the same time stamp, with the goal of making time correlation much easier. Time-based correlation is further aided if all reporting devices share a common time zone, such as UTC. This ensures that devices in different geographic regions still share the same time.
If using NTP, updates should be authenticated wherever possible to ensure that only time from trusted sources is used and that attackers are unable to influence the times on devices.
Normalisation
A successful log management solution will allow logs to be normalised to a common format. That is, it should be able to map a field in a logged event to a common attribute shared across all devices. This makes correlation and searching a much easier affair. For example, a firewall might refer to the source IP address of a packet as src, whilst a web server might refer to the IP address of a client requesting a web resource as source. Both are referring to the same attribute, and it should be possible to discern this either by a process of normalisation or inference.
Although normalisation is important, it is also important to recognise that many security staff prefer to analyse raw logs, so both normalised and raw formats should be available.
Centralisation
In order to correlate events properly and efficiently, logs should be stored in a secure central repository. By putting all logs at security staff’s finger tips, and reducing the time between incidents occurring and their detection, a picture of the network is created that is closer to real time.
Only authorised security staff should be permitted to access the stored logs, as they might contain sensitive information, such as personally identifiable information or individuals’ web browsing habits. The ability to tamper with logs should be restricted to as few users as possible to ensure their integrity.
Also, ensure that the centralised repository has sufficient storage to retain data for a reasonable length of time. What is reasonable will be defined by your organisation’s requirements. For example, the PCI DSS dictates that logs should be available for at least one year, with the most recent three months available in easily-accessible storage. Organisations should carefully plan their storage requirements and ensure that their solution scales to allow for future growth.
Non-Repudiation
Attribution should be possible through the use of unique user accounts assigned to individuals. Shared or generic user accounts should be avoided wherever possible, to ensure that events can be attributed to a named individual and held to account. For example, access to the root account on Linux machines should be limited. Instead, users should be assigned individual user accounts with appropriate access to the sudo command to perform privileged actions.
Conclusion
By ensuring that log management solutions have the features discussed here, organisations can increase the efficiency of their security staff, and thus increase their return on investment in their security programs.
NCC Group can help design, deploy, and test custom or commercial off-the-shelf log management solutions. We also have expertise in performing log analysis to identify or triage security incidents. If you find yourself in need of help then give us a call.
Published date: 10 April 2014
Written by: Will Alexander
