Learn how to extend and personalize SAP applications. Follow the SAP technology blog for insights into SAP BTP, ABAP, SAP Analytics Cloud, SAP HANA, and more.
Logstash is a very light weight component to ship the logs from one server to centralized servers. In the centralized server we might have run the logstash to apply the pattern and get the required info extracted and then send it to elastic search server.
We can configure logstash in any machine, now lets see how can we install and configure in windows
Logstash: The server component of Logstash that processes incoming logs
Elasticsearch: Stores all of the logs
Kibana: Web interface for searching and visualizing logs
Elastic has created an end-to-end stack called ELK and elastic stack makes searching and analyzing of data easier than ever before
Screenshot below explains capabilities of each component :
Elasticsearch and Logstash require Java 7 so it has to be installed
Configuration of Logstash is shown below:
Example in Windows OS ,
Grok is the bread and butter of Logstash filters and is used ubiquitously to derive structure out of unstructured data , it is used as a filter to Parse arbitrary text and structure it
For additional knowledge on creating patterns one can go through this :
If one expects to see the output on the console just un-comment stdout , one can validate against the output to see if it is as expected
## Install Java JRE
Using /s will do a silent installation without asking you any questions. It should be save. Haven't had any additional browser toolbars installed afterwards
jre-windows-x64.exe /s INSTALLDIR=c:\java\jre
## Install NSSM Just extract the ZIP file to c:\nssm
## Logstash ### Prepare the directory structure
REM Base install dir
md c:\logstash
REM Extract Logstash to this directory
md c:\logstash\install
REM NSSM will save Logstash's stdout/stderr here
md c:\logstash\nssm
REM Let's keep Logstash's config outside the install dir for easier updates
md c:\logstash\conf.d
Component Versions used
Elasticsearch 2.1.0 and 2.3.3
Logstash 1.5.4 and 2.3.2
Kibana-4.3.1-windows( has Sense which was useful in querying) and 4.5.1-windows(doesn't have Sense)
Now lets see how can we do it in Monsoon (SAP Cloud Server)
In addition to ELK (versions can vary), logstash forwarder needs to be installed
Logstash Forwarder: Installed on servers that will send their logs to Logstash, Logstash Forwarder serves as a log forwarding agent that utilizes the lumberjack networking protocol to communicate with Logstash
The Logstash Forwarder will be installed on all of the servers that we want to gather logs for, which we will refer to collectively as our Servers
My Area of work was on Chef cookbooks to develop a Performance service in HCP for HTTP and LJS logs using ELK and Ruby
Example in Monsoon
Screenshot of a working filter
Here is a quick demo of the steps involved on Windows (also available on Youtube)