In computer log management and intelligence , log analysis (gold system and network log analysis ) is an art and science seeking to make sense out of computer-generated records (also called Expired log or audit trail records). The process of creating such records is called data logging .
Typical reasons why people perform the log analysis are:
- Compliance with security policies
- Compliance with audit or regulation
- System troubleshooting
- Forensics (during investigations or in response to subpoena )
- Security incident response
- Understanding online user behavior
Logs are emitted by network devices, operating systems, applications and all manner of intelligent or programmable device. A stream of messages in time-sequence often understood by log. Logs can be directed to files and stored on disk, or directed to a network stream to a log collector.
Log messages must be understood with respect to the internal state of its source (eg, application) and announce security-relevant or operations-relevant events (eg, a user login, or a systems error).
Logs are often created by software developers to assist in the operation of an application or understanding of a user’s interaction with a system, such as search engine.  The syntax and semantics of data within log messages are usually applied or vendor-specific. Terminology may also vary; for example, the authentication of an application can be described as a login, a logon, a user connection or authentication event. Hence, log analysis must interpret messages within the context of an application, vendor, system or configuration in order to make useful comparisons to messages from different log sources.
Log message format or content may not be fully documented. A task of the log analyst is to induce the system to be of the full range of messages in which the message must be interpreted.
A log analyst can map varying terminology from different log sources to a uniform, normalized terminology so that reports and statistics can be derived from a heterogeneous environment. For example, log messages from Windows, Unix, network firewalls, databases can be aggregated into a “normalized” report for the auditor. Different systems may signal different messages with different vocabulary, such as “error” and “warning” vs. “err”, “warn”, and “critical”.
Hence, log analysis practices exist on the continuum from text retrieval to reverse engineering of software.
Functions and technologies
Pattern recognition is a function of selecting incoming messages and comparing patterns.
Normalization is the function of converting messages to the same format.
Classification and tagging is ordering messages with different keywords for later use (eg filtering or display).
Correlation analysis is a technology of collecting messages from different systems (eg, messages generated by malicious activity on different systems: network devices, firewalls, servers, etc.). It is usually connected with alerting systems.
Artificial Ignorance A type of machine learning which is a process of discarding log entries which are known to be uninteresting. Artificial ignorance is a method to detect abnormalities in a working system. In log analysis, this means recognizing and ignoring the regular, common log messages that result from the normal operation of the system, and therefore are not too interesting. However, new messages have not appeared in the logs before important events, and should be therefore investigated.   In addition to anomalies, the algorithm will identify common events that did not occur. For example, a system update that runs every week, and one week it was not run.
Log Analysis is Often Compared to other analytics tools Such As Application Performance Management (APM) and Error Monitoring. While much of their functionality is clear overlap. The difference is rooted in process. APM has an emphasis on performance, and most in production. Error monitoring is driven by developers versus operations, and integrates in code in exception handling blocks.
- Audit trail
- Data logging
- Data logger
- Server log
- System monitor
- Web log analysis software
- List of web analytics software
- Jump up^ Jansen, BJ (2008)The methodology of search log analysis. Handbook of research on Web log analysis, 100-123
- Jump up^ http://www.ranum.com/security/computer_security/papers/ai/index.html
- Jump up^ https://lwn.net/Articles/369075/