Load Testing with JMeter: Test Results Visualization Using Kibana Dashboards
Update on 24.06.2017
The listener plugin has been re-implemented in order to reduce number of external dependencies that are not contained in JMeter distribution and require additional libraries: HTTP client was changed from Jersey to Elasticsearch Java REST (external dependency), JSON processor was changed from Gson to json-smart (contained in JMeter distribution).
Additional listener configuration parameter has been introduced to allow exclusion of specific attributes of sample result from being sent to Elasticsearch server.
Apache JMeter (Apache JMeter – Apache JMeter&trade;) is one of popular tools for load testing, which became a “Swiss Army knife” in the area of integration scenarios performance testing. It is open source, lightweight, supports variety of load generator types (called samplers in JMeter’s terminology), and it is flexibly extendible. Most commonly extension plugins are developed for samplers – so that JMeter can be used to generate load for specific technologies and protocols. But there is another area, which is worth of paying attention to – processing, presentation and visualization of test results. In JMeter, listener components are responsible for this task. JMeter is already equipped with some of them, 3rd party plugins enrich listeners capabilities by bringing new listeners – but in majority of cases, listeners are either visualizing test results in JMeter, or exporting them to external files. Test results visualization in JMeter is limited to user interface technologies used by JMeter, export to external files requires further data processing and rendering.
For those who would like to familiarize themselves with JMeter, there are technical materials describing its basic concepts, development and usage of test plans and their major components: a good starting point is documentation published on JMeter’s official web site, complemented by examples of JMeter test plans, which can be found over the Internet and on SCN.
In this blog, I would like to describe an alternative approach, which employs external data visualization tools for presentation of test results obtained from JMeter test plan executions in real time. I will use Kibana (Kibana: Explore, Visualize, Discover Data | Elastic), which is a part of ELK stack, but generally speaking, the approach can be easily adopted and other data visualization / dashboard tools can be utilized instead. In conjunction with Kibana, which is used for data visualization, I will utilize Elasticsearch for storing and indexing JMeter test results data (samples data). Principal flow diagram depicting involved components, is provided in the illustration below:
Custom developed JMeter listener for Elasticsearch sends test results data to an Elasticsearch server, Kibana uses indexed data persisted in Elasticsearch, when visualizing it in a prepared dashboard.
JMeter Listener for Elasticsearch
A developed JMeter listener for Elasticsearch is bundled as a JAR file. A Java class implementing listener, has to extend JMeter’s abstract class AbstractBackendListenerClient. Details on AbstractBackendListenerClient are provided in JavaDocs for JMeter and can be found at https://jmeter.apache.org/api/index.html?org/apache/jmeter/visualizers/backend/AbstractBackendListenerClient.html. Below are some key points regarding important methods of this class, that have to be implemented by custom listeners:
- getDefaultParameters(). Implements logic to set parameters and their default values. It is necessary to specify parameters programmatically so that they appear on the UI of the listener, default values can be overwritten later in JMeter in the listener UI. In the developed listener, parameters are used to specify connectivity and authentication configuration for a called Elastichsearch server, index to be used and timezone to be used for timestamps adjustment (if necessary);
- setupTest(). Implements initialization logic. Called once at the beginning of an executed test plan. In the developed listener, an Elasticsearch server ping test is conducted during initialization – if ping fails (for example, due to network connectivity issues between JMeter and an Elasticsearch server, an Elasticsearch server being down or incorrectly specified listener parameters), listener will not attempt to send samples data to an Elasticsearch server;
- teardownTest(). Implements cleanup logic. Called once at the end of an executed test plan. Not used in the developed listener;
- handleSampleResults(). Implements listener logic that is used to handle and process each sampler execution results. Called at each iteration of the test for each thread in the thread group. In the developed listener, sample data is marshalled into JSON format and sent to a specified Elasticsearch server by invoking its corresponding REST API over HTTP(S).
Following 3rd party libraries that are not contained in JMeter distribution, where used in implementation of a listener for Elasticsearch:
- Elasticsearch Java REST client library. Used to generate and send requests to Elasticsearch server using its API.
Other dependency libraries that are used in this listener, shall already be contained in JMeter distribution.
At least JRE 8 is required to use it.
Additionally, generation of a Java class reflecting structure of test results / sample data was done using http://www.jsonschema2pojo.org/.
Utility and dependency libraries (Elasticsearch Java REST client) shall be placed in directory <JMeter home>/lib.JMeter listener plugin binary shall be placed in directory <JMeter home>/lib/ext.
Source code of the developed listener, as well as compiled and built plugin (which is distributed as a JAR file) can be downloaded from GitHub.
The developed listener has been tested with JMeter version 3.2 running on Java 1.8 runtime.
I have prepared a JMeter test plan that consists of two samplers, which are responsible for generation of SOAP requests (being sent to a SAP PO system to trigger a simple interface) and HTTP requests (being sent to a public Internet service for retrieving current time).
Below is a screenshot of main components of a test plan and configuration of a developed listener for Elasticsearch. As it can be noticed, custom listener is added to a test plan using component Backend Listener and selecting a class that implements the required listener:
Later on, the parameter ‘result.attributes.excluded’ has been introduced and can be used to specify sample result attributes that do not need to be passed to Elasticsearch server. Names of attributes shall be comma-delimited. Full list of currently collected and available attributes, is: Timestamp, StartTime, EndTime, Time, Latency, ConnectTime, IdleTime, SampleLabel, GroupName, ThreadName, ResponseCode, IsResponseCodeOk, IsSuccessful, SampleCount, ErrorCount, ContentType, MediaType, DataType, RequestHeaders, ResponseHeaders, HeadersSize, SamplerData, ResponseMessage, ResponseData, BodySize, Bytes.
The test plan configuration implies execution of samplers several times – having done so, we are capable of collecting some volume of test data, that is sufficient for visualization demonstration in Kibana.
The developed listener produces log entries that can be accessed from JMeter – info / error messages for major events and debug messages in case details of listener execution need to be collected:
Configuration of Elasticsearch and Kibana, as well as development of visualizations and dashboards in Kibana are out of scope of this blog. Please refer to official documentation available on Elastic web site and tutorials published in Internet in case details are required in regards to these topics.
Let us now start test plan execution in JMeter and simulate both successful and failed sample executions in one of target systems. Below is an outlook at results visualized in a Kibana dashboard:
It shall be noted that just few attributes of retrieved test results were visualized and used in a demonstrated Kibana dashboard – there is much more data sent by a listener and indexed by Elasticsearch, which can be used for more precise analysis of test results. Below is a sample indexed document (test sample) containing various metrics and test sample request / response data, which were submitted by JMeter, persisted in Elasticsearch and available for visualization in Kibana:
I am trying to get your Jmeter ELK solution to work on my local but i am running into some issues. I have downloaded your Listener4Elasticsearch.jar and placed it under /lib/ext/. I am about to see the listener in the jmeter UI. However when I execute my test, no data is being display in Kibana.
The plugin has dependencies on several 3rd party libraries - particularly, Gson and Jersey 1.x - which are not a part of neither JMeter bundle, nor this plugin. This means, you have to download them separately and put them into directory <JMeter home>/lib (see remark on this in the blog).
If you already did this, please post an error message details from JMeter's log, it may bring clarity about origin of the error.
Please see the attached for the list of dependency i copied into the lib. Also I do not see any error messages with vadim.jmeter.listener.elasticsearchlistener in Jmeter logs.
From provided logs, I cannot see any evidence on invocation of the Backend Listener component by JMeter runtime. Can you please check details of JMeter test plan execution and see if attempt to call Backend Listener is actually done? Is corresponding listener component in the test plan enabled?
You may increase JMeter log level to DEBUG (e.g. in jmeter.properties or by using corresponding additional command line option when starting JMeter) and check if you get any additional information in the log.
The plugin is called as a Backend Listener implementation - but if JMeter does not call Backend Listener component at all when running your test plan, this means it does not even reach the plugin.
is there a way to configure a proxy in the Elasticsearch Listener that will be used by the Java REST client library? Currently, we need to go through a proxy to access the Elasticsearch server and the Listener is throwing an "407 Proxy Authentication Required" error message.
I've enhanced the listener and introduced support of proxy - the updated version (ver.0.3) with this feature is available at GitHub. The feature introduces three new configuration parameters (elasticsearch.connection.proxy.*): url, user and password. If these parameters are left empty, listener will not use proxy, if well-formed URL of the proxy server is specified, requests will be sent via proxy with no proxy authentication, if all three parameters are specified, requests will be sent via proxy with proxy authentication.
Hi Vadim, thank you for making such a quick fix.
I have downloaded ver 0.3 of the plugin and specified the new parameters, e.g.:
proxy.url = http://myProxy.local:8080
proxy.user = proxyUsername
proxy.password = proxyPwd
Unfortunately, the Elasticsearch ping test is still failing with following response "407 Proxy Authentication Required", "credentials could not be authenticated: "Credentials are missing.". You will not be permitted access until your credentials can be verified."
I am using library elasticsearch-rest-client-6.0.0-rc1.
Am I missing something in the setup?
That is interesting as I've tested this functionality before publishing the update and it worked fine for me. I used local setup (proxy and JMeter running locally), but I don't think it shall be major differentiating factor here. In my tests, I used CCProxy as a proxy, and Elasticsearch REST client library version 5.4.2.
Can I ask you to give a try to any request (such as GET request) to Elasticsearch server via proxy using Postman or browser? Given you use the same proxy settings (URL, user, password) in Postman / browser, does it work finely there? If so, can you please collect network trace for both successful (Postman / browser) and failed (JMeter) requests so that it can be compared if there is any difference between them?
Hi Vadim, I have sketched the exchange in the image below: it looks like that the Elasticlistener is not responding to the proxy authentication challenge and failing the test directly:
It looks to me like root cause analysis, reproduction in development and fixing this might take some time. Reason for this is, I tried to reproduce the issue locally and collected network traces for following two scenarios in JMeter (both using same proxy settings - URL and user/password): a) standard HTTP sampler, b) custom Elasticsearch listener. Looking in both traces, I can see packets to the proxy service and subsequent packets to the target Elasticsearch server with indication of used proxy and basic authentication for proxy in their headers, were captured. Traces are very similar and both suggest proxy was not bypassed and request for proxy authentication was fulfilled and delivered successfully.
In particular, this is an extract of network trace for the listener's test connection request (HTTP HEAD request to Elasticsearch server). In collected packets, it can be seen that firstly the request to Elasticsearch server (listening on port 9200) came (the first packet highlighted in red) - it targeted proxy server (listening on port 8080), which replied back with the response "Proxy Authentication Required" (the second packet highlighted in red). After few communication packets happening in between, the client (JMeter's listener) sent another request to proxy (the third packet highlighted in red), where in packet details (bottom part of the window) it can be seen that the request's TCP destination was still proxy (destination port 8080), whereas the actual HTTP request was instructed to be delivered to Elasticsearch server (http://localhost:9200/), and the request contained Proxy-Authorization header with valid proxy credentials (the bottommost highlighted part of the packet)
Similar trace and observations I got for HTTP sampler.
This all together leads me to conclusion that I'm still unable to reproduce the issue, and the network level behaviour of the plugin is as expected - at least, in the local development environment I used so far. I will take some more time to see if I can reproduce the issue you face, but would appreciate if we could move forward with some more details about setup of your environment and see where meaningful discrepancies might come from. We can progress here, but my suggestion would be not to overload blog comments with detailed discussions of this particular issue, and switch to alternative means of communication, such as e-mail, where we can exchange full traces and investigate it in more details. If that option is suitable for you, can you please share your e-mail address (for example, via direct message) and we can progress further. I would be thankful if you could collect similar traces on your side so that we can compare them and figure out differences. For your reference, I used Wireshark for collection and analysis of network traces, together with Npcap library (to enable Wireshark to trace loopback interface). Npcap usage was only required due to all components (JMeter, proxy, Elasticsearch) talking to each other locally, via loopback interface, and not via network interfaces - hence, in case of distributed installation, it might not be required.
As discussed over mails, currently Elasticsearch REST client library provides support of basic authentication mechanism - although the underlying used library (Apache HttpClient) has capabilities to support alternative authentication mechanisms (NTLM being named as one of them), it has to be investigated further if and how this is feasible to utilize the feature in Elasticsearch REST client.
if i try your plugin i get the following error:
2017-10-12 13:21:18,424 INFO c.d.j.l.e.c.d.j.l.e.ElasticsearchListener: Elasticsearch ping test: Successful
2017-10-12 13:21:18,424 INFO o.a.j.g.u.JMeterMenuBar: setRunning(true, *local*)
2017-10-12 13:21:18,512 INFO o.a.j.e.StandardJMeterEngine: Starting ThreadGroup: 1 : GetVersion
2017-10-12 13:21:18,512 INFO o.a.j.e.StandardJMeterEngine: Starting 1 threads for group GetVersion.
2017-10-12 13:21:18,512 INFO o.a.j.e.StandardJMeterEngine: Thread will continue on error
2017-10-12 13:21:18,512 INFO o.a.j.t.ThreadGroup: Starting thread group... number=1 threads=1 ramp-up=1 perThread=1000.0 delayedStart=false
2017-10-12 13:21:18,514 INFO o.a.j.t.ThreadGroup: Started thread group number 1
2017-10-12 13:21:18,514 INFO o.a.j.e.StandardJMeterEngine: All thread groups have been started
2017-10-12 13:21:18,516 INFO o.a.j.t.JMeterThread: Thread started: GetVersion 1-1
2017-10-12 13:21:18,517 INFO o.a.j.s.FileServer: Stored: Data/PreOrderCheck_buy.csv
2017-10-12 13:21:18,522 INFO o.a.j.p.h.s.HTTPHCAbstractImpl: Local host = ProTest
2017-10-12 13:21:18,524 INFO o.a.j.p.h.s.HTTPHC4Impl: HTTP request retry count = 0
2017-10-12 13:21:18,741 ERROR o.a.j.JMeter: Uncaught exception:
java.lang.StringIndexOutOfBoundsException: String index out of range: -1
at java.lang.String.substring(String.java:1967) ~[?:1.8.0_131]
at com.doc.jmeter.listeners.elasticsearch.com.doc.jmeter.listeners.elasticsearch.ElasticsearchListener.getSampleResult4External(ElasticsearchListener.java:333) ~[JMeter_ElasticsearchListener.jar:?]
at com.doc.jmeter.listeners.elasticsearch.com.doc.jmeter.listeners.elasticsearch.ElasticsearchListener.handleSampleResults(ElasticsearchListener.java:140) ~[JMeter_ElasticsearchListener.jar:?]
at org.apache.jmeter.visualizers.backend.BackendListener.sendToListener(BackendListener.java:274) ~[ApacheJMeter_components.jar:3.2 r1790748]
at org.apache.jmeter.visualizers.backend.BackendListener$Worker.run(BackendListener.java:246) ~[ApacheJMeter_components.jar:3.2 r1790748]
2017-10-12 13:21:28,330 INFO o.a.j.t.JMeterThread: Thread is done: GetVersion 1-1
Im using your 0.3 release on Ubuntu 16.04 with jMeter 3.2 and java 1.8.
Can you please check that both thread group and sampler components in the JMeter plan are named / have labels (this is mandatory for this listener plugin)? For every sample result submitted by the sampler, this listener retrieves information about thread group name in which context it is executed, and sampler name. By default, both of them are submitted to Elasticsearch server so that if there are different thread groups within single test plan and they contain samplers with same names, there will be no confusion about which sample results are analyzed (documents / entries in Elasticsearch could be filtered further by thread group name / sampler name).
From the exception you provided, it looks to me as the thread group is named / has label 'GetVersion', but sampler is not named / doesn't have label.
To illustrate this idea, the below extract of the test plan, containing thread group and sampler in it, is correct:
In contrast, the following extract of the test plan (label of the sampler being removed) will cause error:
thanks for your replay and its solve my issue 🙂 the problem was that the thread and the sampler both had the same name so i changed the sampler name and after that it works!
thanks again for the solution
I downloaded elastic search from here: https://www.elastic.co/downloads/elasticsearch
as a zip file.
its includes plenty of files and not a single jar,
I tried to follow exactly after the instructions.
I am using jmeter 3.2, and downloaded elasticsearch-5.6.4.zip.
I extract the zip and put the folder elasticsearch-5.6.4 in C:\Jmeter\apache-jmeter-3.2\lib
I downloaded the jar JMeter_ElasticsearchListener.jar and put it in C:\Jmeter\apache-jmeter-3.2\lib\ext
I close jmeter and opened it, pressed add -> listener -> beckend listener and get this errors:
can someone explain what I am missing?
It seems you downloaded Elasticsearch server distributive, while here you need client libraries - namely, the Elasticsearch JMeter listener plugin and Elasticsearch Java REST client, which you can download, for example, from Maven repository (https://mvnrepository.com/artifact/org.elasticsearch.client/rest). You only need libraries' JAR files to be placed in corresponding folders of JMeter installation (Elasticsearch listener plugin - in directory /lib/ext, Elasticsearch Java REST client library - in directory /lib).
I downloaded the jars for the elastic search + dependencies.
I should put only the 3 jars in the bin folder? am I correct?
No, you don't need that many dependency JARs: besides the listener plugin JAR, you only need one dependency's JAR (which has to be placed in /lib, not /bin) for Elasticsearch Java REST client library - have a look into my earlier comment, it contains link to Maven repository from where you can download it.
I try to work with this listener, and I saw that the UI is changed.
where I need to put the scheme name?
if scheme name = superDB
IP = 22.214.171.124
port = 98
what to put in the elastic search URL?
what to put in elastic search type?
Elasticsearch client doesn't use database schema indication - so I might need to get context of schema 'superDB' that you refer to - what is that and what do you use it for?
In the recent release, listener plugin was reworked and its parameterization simplified - hence, you now see a single parameter for Elasticsearch server's URL, which shall be the value in format <scheme>://<host>:<port>.
Elasticsearch type is name of the documents' type that shall be used in the specified Elasticsearch index, when sending sample results of JMeter test run (where sample results are ultimately converted into Elasticsearch documents).
You can find more extensive description of all current parameters acceptable by the listener plugin, in ReadMe documentation in the GitHub repository: https://github.com/vadim-klimov/apache-jmeter-listener-elasticsearch.
Thanks a lot, this is really a very helpful article.
One quick question, my Jmeter log file is getting very huge size because the Listener is logging everything in the Log as well. Do you have any quick solution to stop logging the Listener in the Log file?
The listener uses different log levels when generating log entries - debug level for most precise information about listener execution, should you require to obtain the most detailed logs, info level to log major steps of listener execution, warning level for situations when there were some recoverable deviations and error level for severe error occurrences. You can adjust and set an appropriate log level in JMeter configuration (Options > Log Level). Hence, if you don't want to log verbose details of listener execution and only get informed on severe errors, set the log level to ERROR.
Thanks a lot.
But I am running from the jmtere maven plugin with Jmtere version 3.3. And I am not able to set the Log level as we can do in the GUI. Tried with Log4J2 config but still the Listener logs are getting printed. Hence, thought to check with you.
But I understand that is not directly related to the Listener and more generic. I will try to find the solution.
I could resolve it after changing Log4J2 config only.
Can we also capture total users CPU memory usage and other system metrics?
Yes, have a look in PerfMon plugin for JMeter - it allows collection of important information on server load and health metrics for servers where PerfMon agent is running.
Thank you how did you get such beautiful dashboards in kibana I managed to get data into elastic search but clueless on how to pull data into kibana
I would suggest you checking Kibana installation and configuration guide - it describes how you can connect Kibana to a given running Elasticsearch server(s), and then how you can visualize documents indexed by Elasticsearch, in Kibana dashboards.
I want to send some additional log parameters, is it possible to do that?
Yes, it is possible to enrich an entry that is sent to Elasticsearch, with additional attributes. JMeter sampler produces SampleResult that is available to listeners and contains an outcome of a sample execution - it is possible to use its attributes, as well as add custom attributes based on your specific requirements. This can be achieved by extending implemented logic contained in the method ElasticsearchListener.getSampleResult4External(), that produces String representation of the entry to be sent to Elasticsearch.
Thanks for your quick response and detailed explanation.
I am new into JAVA , able to extend the code but finding difficulty to create a jar file.
It is giving compilation error because we have used third party modules.
If you can give me the steps to create JAR file it would be great.
I used your JAR file it worked file, tried to create JAR file from your source code but failed to do it.
The listener uses Elasticsearch Java REST client library (https://www.elastic.co/guide/en/elasticsearch/client/java-rest/index.html), so please make sure it is added as a dependency to the project before you run compilation and assembly into the JAR file.
Hi, I would require sample report template.json file(NDJSON) contains performance testing dashboard that can be used in the ES/ KIbana Dashboard for jmeter load testing. I tried a lot to find it, but no luck. COuld you please help in this?