Technical Articles
ELK – Elasticsearch Logstash Kibana
Logstash is a very light weight component to ship the logs from one server to centralized servers. In the centralized server we might have run the logstash to apply the pattern and get the required info extracted and then send it to elastic search server.
We can configure logstash in any machine, now lets see how can we install and configure in windows
- Logstash: The server component of Logstash that processes incoming logs
- Elasticsearch: Stores all of the logs
- Kibana: Web interface for searching and visualizing logs
Elastic has created an end-to-end stack called ELK and elastic stack makes searching and analyzing of data easier than ever before
Screenshot below explains capabilities of each component :
Elasticsearch and Logstash require Java 7 so it has to be installed
Configuration of Logstash is shown below:
Example in Windows OS ,
Grok is the bread and butter of Logstash filters and is used ubiquitously to derive structure out of unstructured data , it is used as a filter to Parse arbitrary text and structure it
For additional knowledge on creating patterns one can go through this :
Grok Constructor
logstash-patterns-core/patterns at master · logstash-plugins/logstash-patterns-core · GitHub
If one expects to see the output on the console just un-comment stdout , one can validate against the output to see if it is as expected
## Install Java JRE
Using /s will do a silent installation without asking you any questions. It should be save. Haven’t had any additional browser toolbars installed afterwards
jre-windows-x64.exe /s INSTALLDIR=c:\java\jre
## Install NSSM Just extract the ZIP file to c:\nssm
## Logstash ### Prepare the directory structure
REM Base install dir
md c:\logstash
REM Extract Logstash to this directory
md c:\logstash\install
REM NSSM will save Logstash’s stdout/stderr here
md c:\logstash\nssm
REM Let’s keep Logstash’s config outside the install dir for easier updates
md c:\logstash\conf.d
Component Versions used
Elasticsearch 2.1.0 and 2.3.3
Logstash 1.5.4 and 2.3.2
Kibana-4.3.1-windows( has Sense which was useful in querying) and 4.5.1-windows(doesn’t have Sense)
### Install Logstash as a Windows Service
cd c:\nssm\win64
nssm install logstash C:\logstash\install\bin\logstash.bat
nssm set logstash AppParameters agent –config c:\logstash\conf.d
nssm set logstash AppDirectory C:\logstash\install
nssm set logstash AppEnvironmentExtra “JAVA_HOME=C:\java\jre”
nssm set logstash AppStdout c:\logstash\nssm\stdout.log
nssm set logstash AppStderr c:\logstash\nssm\stderr.log
REM Replace stdout and stderr files
nssm set logstash AppStdoutCreationDisposition 2
nssm set logstash AppStderrCreationDisposition 2
REM Disable WM_CLOSE, WM_QUIT in the Shutdown options.
Without it, NSSM can’t stop Logstash properly
nssm set logstash AppStopMethodSkip 6
REM Let’s start Logstash.
On completion of the above steps the configuration is in place to start with below steps:
net start logstash
### Remove Logstash’s Windows service
net stop logstash
cd c:\nssm\win64
nssm remove logstash
## Troubleshooting ### Have a look at Logstash’s stderr/stdout data first
type c:\logstash\nssm\stderr.log
type c:\logstash\nssm\stdout.log
### Is Java (64bit) installed correctly? c:\java\jre\bin\java -version java version “1.7.0_60” Java(TM) SE Runtime Environment (build 1.7.0_60-b19) Java HotSpot(TM) 64-Bit Server VM (build 24.60-b09, mixed mode)
### Test reading Security event logs on Windows input { eventlog { type => ‘Win32-EventLog’ logfile => ‘Security’ } } output { stdout {} }
==============================================================================
==============================================================================
Now lets see how can we do it in Monsoon (SAP Cloud Server)
In addition to ELK (versions can vary), logstash forwarder needs to be installed
- Logstash Forwarder: Installed on servers that will send their logs to Logstash, Logstash Forwarder serves as a log forwarding agent that utilizes the lumberjack networking protocol to communicate with Logstash
The Logstash Forwarder will be installed on all of the servers that we want to gather logs for, which we will refer to collectively as our Servers
My Area of work was on Chef cookbooks to develop a Performance service in HCP for HTTP and LJS logs using ELK and Ruby
Example in Monsoon
Screenshot of a working filter
Here is a quick demo of the steps involved on Windows (also available on Youtube)
ELK – Elasticsearch Logstash Kibana – Introduction on Windows – YouTube
Remember that you can send pretty much any type of log to Logstash, but the data becomes even more useful if it is parsed and structured with GROK
Thanks,
Dilip
SAP BLR
Hi,
Other useful links
Analyze Web Dispatcher logs with Kibana
SAP Business Intelligence with Elasticseach and Kibana
Hi Dilip,
Thanks for the nice blog on ELK. All the required information is collated at one place here...It is really useful for beginners like me.
Thank you , for more on their products and additional information please refer to their home page
Powering Data Search, Log Analysis, Analytics | Elastic