Log analytics is the assessment of a recorded set of information from one or more events, captured from a computer, network, application operating system (OS) or other IT ecosystem component. An organization can use log analytics to uncover patterns in user behaviors, identify problems, audit security activities or ensure compliance with established rules, and plan for capacity or IT infrastructure changes.
An event is an identifiable or significant occurrence within hardware or software, and the information about it is recorded in a log. A user or a computer system can generate an event. For example, a server hardware failure is an event.
Organizations rely on tools, such as Windows Event Viewer for Windows OS, or the application SolarWinds Event & Log Manager, to access, view and analyze logs.
Log analytic software collects and parses error logs to help an organization diagnose an issue: what caused the problem, where it is located and how serious it is. Log analytics can also aid a user to determine trends in an application or system's operation. Log analytic tools aggregate logs from disparate data sources, compiling a view of the widespread operation of an IT ecosystem.
How log analytics works
Log analytic software collects logs from events such as application installation, security breaches, system setup/startup operational information and more. An example security event is a system login attempts. An example operational event is when an application opens successfully. Setup events focus on the control of domains, such as where a log is stored after a disk configuration. System events focus on components such as CPU and storage.
A log entry includes such information as: the date and time the event occurred, the computer the event occurred on, an identification of the user, the category of the event (such as setup or security) and the program that initiated the event.
Log analytics occurs by organizing data via pattern recognition, classification and tagging, correlation analysis, and artificial ignorance. Pattern recognition is a way to compare incoming events to past events to determine which of those new occurrences will be significant. Classification and tagging puts events into ordered classes, and assigns a keyword or keywords to the event to describe it in a standardized way. Correlation analysis is a way to sort logs by warning events, and can alert the administrators to a widespread system error if a critical warning appears in multiple logs. Artificial ignorance, a machine learning program, discards log entries that occur regularly. It helps reduce noise and find uncommon events. Artificial ignorance is well suited to a system that operates consistently with a low amount of issues.
Each system that generates log messages writes them in a way that is specific to itself, so log analytics software must pull everything under cohesive terminology. For example, one application logs a moderate software failure event as a warning, while the same event is labeled as an error by another application.
Log analytics tool features and products
A log analytics tool performs log aggregation and gives users a query language to glean insights from the collected information. Log analytics tools can also automatically process logs for insights into specific events, or perform deeper analysis to extract meaningful conclusions or make predictions about the pattern of events taking place over time.
These tools typically tier events by level of urgency. For example, Windows Event Viewer uses information, warning, error, and critical urgency levels. Information is the least severe log entry, typically successful events. Warnings are meant to give attention to potential issues but do not indicate that something needs to be fixed. Error level events occur when an application starts to unexpectedly fail. A critical error happens when a program is forced to stop, and can no longer run properly without further attention.
Log analytics tools commonly offer GUI-based dashboards. Dashboards display the most relevant and critical information gleaned from log input, and can be tailored to the user. A dashboard might include total count of events, alerts, log search queries, filters for things like security or change management, and graphs. Graphs can show statistics on disk space, CPU status, event categories, and more. Some dashboards are customizable.
Log analytics tools commonly include a search functionality, which helps a user find a logged event. For example, if a log analytics tool uses classification and tagging, then the user can quickly search for a specific event by the given keyword.
A sampling of log analytics products includes Scalyr, SolarWinds Event & Log Manager and Microsoft Azure Log Analytics.
Scalyr is marketed as a DevOps log monitoring and analysis tool, as it consolidates logs for diagnosis and visualization from applications and systems. Scalyr can filter out user-specified logs, as well as graph metrics to show statistics such as percentiles, rates, distributions, and trends.
SolarWinds Log & Event Manager is an example of a tool with a customizable dashboard. For example, it can show data used by multiple accounts, filter events tied to security and more. LEM can alert users upon a warning event, which can be specified by the user. The dashboard can also contain a word cloud: a chart showing where the most logs are being generated.
Azure Log Analytics, not to be confused with the term log analytics, is part of a public cloud offering. It can be accessed independently or through other Azure products such as Azure Security Center. Azure Log Analytics can analyze virtual machines (VMs) via agents as well.