Open source log management tools collect and analyze system logs to aid everything from application updates to network and security optimization and IT performance. Everyone can use logs.
Open source log monitoring and management tools like Graylog, or the ELK stack of Elasticsearch, Logstash and Kibana, provide IT pros insights into their environments, offering performance data about what server in which location powered on or off, how many users logged in, when intrusion attempts were made on servers and other events.
The ELK stack's three different open source projects work together. Logstash is a highly configurable daemon installed on a VM or a bare-metal server. It allows you to specify what logs to collect, the type of logs and to which server the logs should be sent. The destination server runs Elasticsearch, which aggregates and stores those logs. Kibana is an open source log analyzer, which makes logs searchable and turns log data into charts, graphs and trendlines to reveal trends and patterns. Graylog offers similar features, but is powered by Elasticsearch and the open source document database MongoDB.
Log management tools deliver analysis and monitoring in complex IT infrastructures. For example, at Clemson University, the infrastructure code team uses the ELK stack to manage VMware ESX virtualization server logs. "When the storage went down, it would create a lot of errors on the ESX servers," said Nitin Madhok, a systems developer and programmer on the infrastructure code team there. He easily set up a dashboard using the three open source log management tools to show how many times in a month his ESX server experienced errors and charted those events based on when errors occurred.
In another example, an IT ops team working with a database server can pinpoint issues happening to the server based on frequency and time.
There are fewer differences between the ELK stack and Graylog than there are between open source tools and proprietary commercial log management tools. A commercial version of ELK stack from Elastic includes more features that organizations and enterprises need, Madhok noted. User management is only available in the supported commercial edition of Kibana, and the community version of Logstash creates indexes every day that the administrator must manually delete. Graylog addresses some of the drawbacks of the ELK stack, as it offers more features, such as user management, for free. While free is appealing, make sure open source tools meet compliance guidelines and security auditing requirements for your company.
Depending on how large your environment is, you may want to consider third-party tools in addition to the open source log management software to ease administration and remediation tasks.
"One person doing ELK stack by themselves is not something the university wants," Madhok said. Because of the limited manpower, Madhok purchased Splunk licenses to fill the gaps of ELK stack, but found them expensive. Splunk Enterprise is priced based on maximum daily aggregate volume of uncompressed data indexed, with the cost per gigabyte ranging from hundreds to thousands of dollars, depending on the plan.
Sometimes, open source software requires more involvement and more labor than commercially packaged software, said Jay Lyman, research manager for cloud management and containers at 451 Research. That roadblock has been winnowed away as open source adoption hits a maturity level that began with dabblers a decade ago. Those early adopters helped develop talent in-house and are now managing open source tool teams, said Stephen Elliot, vice president of research at IDC.
Today, corporate is accepting open source more than ever, and these open source tools enable new organizational and team structures. Investing time and energy into learning the most popular open source system management tools could pay off in new opportunities.