Skip to Content

Update on 24.06.2017

The listener plugin has been re-implemented in order to reduce number of external dependencies that are not contained in JMeter distribution and require additional libraries: HTTP client was changed from Jersey to Elasticsearch Java REST (external dependency), JSON processor was changed from Gson to json-smart (contained in JMeter distribution).

Additional listener configuration parameter has been introduced to allow exclusion of specific attributes of sample result from being sent to Elasticsearch server.

 

 

Intro

Apache JMeter (Apache JMeter – Apache JMeter™) is one of popular tools for load testing, which became a “Swiss Army knife” in the area of integration scenarios performance testing. It is open source, lightweight, supports variety of load generator types (called samplers in JMeter’s terminology), and it is flexibly extendible. Most commonly extension plugins are developed for samplers – so that JMeter can be used to generate load for specific technologies and protocols. But there is another area, which is worth of paying attention to – processing, presentation and visualization  of test results. In JMeter, listener components are responsible for this task. JMeter is already equipped with some of them, 3rd party plugins enrich listeners capabilities by bringing new listeners – but in majority of cases, listeners are either visualizing test results in JMeter, or exporting them to external files. Test results visualization in JMeter is limited to user interface technologies used by JMeter, export to external files requires further data processing and rendering.

 

For those who would like to familiarize themselves with JMeter, there are technical materials describing its basic concepts, development and usage of test plans and their major components: a good starting point is documentation published on JMeter’s official web site, complemented by examples of JMeter test plans, which can be found over the Internet and on SCN.
In this blog, I would like to describe an alternative approach, which employs external data visualization tools for presentation of test results obtained from JMeter test plan executions in real time. I will use Kibana (Kibana: Explore, Visualize, Discover Data | Elastic), which is a part of ELK stack, but generally speaking, the approach can be easily adopted and other data visualization / dashboard tools can be utilized instead. In conjunction with Kibana, which is used for data visualization, I will utilize Elasticsearch for storing and indexing JMeter test results data (samples data). Principal flow diagram depicting involved components, is provided in the illustration below:

Flow.png

Custom developed JMeter listener for Elasticsearch sends test results data to an Elasticsearch server, Kibana uses indexed data persisted in Elasticsearch, when visualizing it in a prepared dashboard.

 

 

JMeter Listener for Elasticsearch

A developed JMeter listener for Elasticsearch is bundled as a JAR file. A Java class implementing listener, has to extend JMeter’s abstract class AbstractBackendListenerClient. Details on AbstractBackendListenerClient are provided in JavaDocs for JMeter and can be found at https://jmeter.apache.org/api/index.html?org/apache/jmeter/visualizers/backend/AbstractBackendListenerClient.html. Below are some key points regarding important methods of this class, that have to be implemented by custom listeners:

  • getDefaultParameters(). Implements logic to set parameters and their default values. It is necessary to specify parameters programmatically so that they appear on the UI of the listener, default values can be overwritten later in JMeter in the listener UI. In the developed listener, parameters are used to specify connectivity and authentication configuration for a called Elastichsearch server, index to be used and timezone to be used for timestamps adjustment (if necessary);
  • setupTest(). Implements initialization logic. Called once at the beginning of an executed test plan. In the developed listener, an Elasticsearch server ping test is conducted during initialization – if ping fails (for example, due to network connectivity issues between JMeter and an Elasticsearch server, an Elasticsearch server being down or incorrectly specified listener parameters), listener will not attempt to send samples data to an Elasticsearch server;
  • teardownTest(). Implements cleanup logic. Called once at the end of an executed test plan. Not used in the developed listener;
  • handleSampleResults(). Implements listener logic that is used to handle and process each sampler execution results. Called at each iteration of the test for each thread in the thread group. In the developed listener, sample data is marshalled into JSON format and sent to a specified Elasticsearch server by invoking its corresponding REST API over HTTP(S).

 

Following 3rd party libraries that are not contained in JMeter distribution, where used in implementation of a listener for Elasticsearch:

Other dependency libraries that are used in this listener, shall already be contained in JMeter distribution.

At least JRE 8 is required to use it.

Additionally, generation of a Java class reflecting structure of test results / sample data was done using http://www.jsonschema2pojo.org/.

Utility and dependency libraries (Elasticsearch Java REST client) shall be placed in directory <JMeter home>/lib.JMeter listener plugin binary shall be placed in directory <JMeter home>/lib/ext.

 

Source code of the developed listener, as well as compiled and built plugin (which is distributed as a JAR file) can be downloaded from GitHub.

 

The developed listener has been tested with JMeter version 3.2 running on Java 1.8 runtime.

 

 

Demo

I have prepared a JMeter test plan that consists of two samplers, which are responsible for generation of SOAP requests (being sent to a SAP PO system to trigger a simple interface) and HTTP requests (being sent to a public Internet service for retrieving current time).

 

Below is a screenshot of main components of a test plan and configuration of a developed listener for Elasticsearch. As it can be noticed, custom listener is added to a test plan using component Backend Listener and selecting a class that implements the required listener:

Listener configuration.png

Later on, the parameter ‘result.attributes.excluded’ has been introduced and can be used to specify sample result attributes that do not need to be passed to Elasticsearch server. Names of attributes shall be comma-delimited. Full list of currently collected and available attributes, is: Timestamp, StartTime, EndTime, Time, Latency, ConnectTime, IdleTime, SampleLabel, GroupName, ThreadName, ResponseCode, IsResponseCodeOk, IsSuccessful, SampleCount, ErrorCount, ContentType, MediaType, DataType, RequestHeaders, ResponseHeaders, HeadersSize, SamplerData, ResponseMessage, ResponseData, BodySize, Bytes.

The test plan configuration implies execution of samplers several times – having done so, we are capable of collecting some volume of test data, that is sufficient for visualization demonstration in Kibana.

 

The developed listener produces log entries that can be accessed from JMeter – info / error messages for major events and debug messages in case details of listener execution need to be collected:

JMeter log.png

 

Configuration of Elasticsearch and Kibana, as well as development of visualizations and dashboards in Kibana are out of scope of this blog. Please refer to official documentation available on Elastic web site and tutorials published in Internet in case details are required in regards to these topics.

 

Let us now start test plan execution in JMeter and simulate both successful and failed sample executions in one of target systems. Below is an outlook at results visualized in a Kibana dashboard:Kibana dashboard.png 

It shall be noted that just few attributes of retrieved test results were visualized and used in a demonstrated Kibana dashboard – there is much more data sent by a listener and indexed by Elasticsearch, which can be used for more precise analysis of test results. Below is a sample indexed document (test sample) containing various metrics and test sample request / response data, which were submitted by JMeter, persisted in Elasticsearch and available for visualization in Kibana:Indexed document.png

To report this post you need to login first.

14 Comments

You must be Logged on to comment or reply to a post.

  1. tony nguyen

    Hi Vadim,

    I am trying to get your Jmeter ELK solution to work on my local but i am running into some issues. I have downloaded your Listener4Elasticsearch.jar and placed it under /lib/ext/. I am about to see the listener in the jmeter UI. However when I execute my test, no data is being display in Kibana.

    (0) 
    1. Vadim Klimov Post author

      Hi Tony,

      The plugin has dependencies on several 3rd party libraries – particularly, Gson and Jersey 1.x – which are not a part of neither JMeter bundle, nor this plugin. This means, you have to download them separately and put them into directory <JMeter home>/lib  (see remark on this in the blog).

      If you already did this, please post an error message details from JMeter’s log, it may bring clarity about origin of the error.

      Regards,

      Vadim

      (0) 
      1. tony nguyen

        Vadim,

        Please see the attached for the list of dependency i copied into the lib. Also I do not see any error messages with vadim.jmeter.listener.elasticsearchlistener in Jmeter logs.

        Screen Shot 2016-08-10 at 1.41.44 PM.pngScreen Shot 2016-08-10 at 1.42.05 PM.pngScreen Shot 2016-08-10 at 1.44.09 PM.png

        (0) 
        1. Vadim Klimov Post author

          Hello Tony,

          From provided logs, I cannot see any evidence on invocation of the Backend Listener component by JMeter runtime. Can you please check details of JMeter test plan execution and see if attempt to call Backend Listener is actually done? Is corresponding listener component in the test plan enabled?

          You may increase JMeter log level to DEBUG (e.g. in jmeter.properties or by using corresponding additional command line option when starting JMeter) and check if you get any additional information in the log.

          The plugin is called as a Backend Listener implementation – but if JMeter does not call Backend Listener component at all when running your test plan, this means it does not even reach the plugin.

          Regards,

          Vadim

          (0) 
  2. Robert Bourgault du Coudray

    Hi Vadim,

    is there a way to configure a proxy in the Elasticsearch Listener that will be used by the Java REST client library? Currently, we need to go through a proxy to access the Elasticsearch server and the Listener is throwing an “407 Proxy Authentication Required” error message.

    (0) 
    1. Vadim Klimov Post author

      Hello Robert,

      I’ve enhanced the listener and introduced support of proxy – the updated version (ver.0.3) with this feature is available at GitHub. The feature introduces three new configuration parameters (elasticsearch.connection.proxy.*): url, user and password. If these parameters are left empty, listener will not use proxy, if well-formed URL of the proxy server is specified, requests will be sent via proxy with no proxy authentication, if all three parameters are specified, requests will be sent via proxy with proxy authentication.

      Regards,

      Vadim

      (0) 
      1. Robert Bourgault du Coudray

        Hi Vadim, thank you for making such a quick fix.

        I have downloaded ver 0.3 of the plugin and specified the new parameters, e.g.:

        proxy.url = http://myProxy.local:8080

        proxy.user = proxyUsername

        proxy.password = proxyPwd

        Unfortunately, the Elasticsearch ping test is still failing with following response “407 Proxy Authentication Required”, “credentials could not be authenticated: “Credentials are missing.”. You will not be permitted access until your credentials can be verified.”

        I am using library elasticsearch-rest-client-6.0.0-rc1.

        Am I missing something in the setup?

        Thanks.

        (0) 
        1. Vadim Klimov Post author

          Hi Robert,

          That is interesting as I’ve tested this functionality before publishing the update and it worked fine for me. I used local setup (proxy and JMeter running locally), but I don’t think it shall be major differentiating factor here. In my tests, I used CCProxy as a proxy, and Elasticsearch REST client library version 5.4.2.

          Can I ask you to give a try to any request (such as GET request) to Elasticsearch server via proxy using Postman or browser? Given you use the same proxy settings (URL, user, password) in Postman / browser, does it work finely there? If so, can you please collect network trace for both successful (Postman / browser) and failed (JMeter) requests so that it can be compared if there is any difference between them?

          Regards,

          Vadim

          (0) 
            1. Vadim Klimov Post author

              Hi Robert,

              It looks to me like root cause analysis, reproduction in development and fixing this might take some time. Reason for this is, I tried to reproduce the issue locally and collected network traces for following two scenarios in JMeter (both using same proxy settings – URL and user/password): a) standard HTTP sampler, b) custom Elasticsearch listener. Looking in both traces, I can see packets to the proxy service and subsequent packets to the target Elasticsearch server with indication of used proxy and basic authentication for proxy in their headers, were captured. Traces are very similar and both suggest proxy was not bypassed and request for proxy authentication was fulfilled and delivered successfully.

              In particular, this is an extract of network trace for the listener’s test connection request (HTTP HEAD request to Elasticsearch server). In collected packets, it can be seen that firstly the request to Elasticsearch server (listening on port 9200) came (the first packet highlighted in red) – it targeted proxy server (listening on port 8080), which replied back with the response “Proxy Authentication Required” (the second packet highlighted in red). After few communication packets happening in between, the client (JMeter’s listener) sent another request to proxy (the third packet highlighted in red), where in packet details (bottom part of the window) it can be seen that the request’s TCP destination was still proxy (destination port 8080), whereas the actual HTTP request was instructed to be delivered to Elasticsearch server (http://localhost:9200/), and the request contained Proxy-Authorization header with valid proxy credentials (the bottommost highlighted part of the packet)

              Similar trace and observations I got for HTTP sampler.

              This all together leads me to conclusion that I’m still unable to reproduce the issue, and the network level behaviour of the plugin is as expected – at least, in the local development environment I used so far. I will take some more time to see if I can reproduce the issue you face, but would appreciate if we could move forward with some more details about setup of your environment and see where meaningful discrepancies might come from. We can progress here, but my suggestion would be not to overload blog comments with detailed discussions of this particular issue, and switch to alternative means of communication, such as e-mail, where we can exchange full traces and investigate it in more details. If that option is suitable for you, can you please share your e-mail address (for example, via direct message) and we can progress further. I would be thankful if you could collect similar traces on your side so that we can compare them and figure out differences. For your reference, I used Wireshark for collection and analysis of network traces, together with Npcap library (to enable Wireshark to trace loopback interface). Npcap usage was only required due to all components (JMeter, proxy, Elasticsearch) talking to each other locally, via loopback interface, and not via network interfaces – hence, in case of distributed installation, it might not be required.

              Regards,
              Vadim

              (0) 
              1. Vadim Klimov Post author

                Hi Robert,

                As discussed over mails, currently Elasticsearch REST client library provides support of basic authentication mechanism – although the underlying used library (Apache HttpClient) has capabilities to support alternative authentication mechanisms (NTLM being named as one of them), it has to be investigated further if and how this is feasible to utilize the feature in Elasticsearch REST client.

                Regards,

                Vadim

                (0) 
  3. paul velzeboer

     

    Hi Vadim,

     

    if i try your plugin i get the following error:

     

    2017-10-12 13:21:18,424 INFO c.d.j.l.e.c.d.j.l.e.ElasticsearchListener: Elasticsearch ping test: Successful
    2017-10-12 13:21:18,424 INFO o.a.j.g.u.JMeterMenuBar: setRunning(true, *local*)
    2017-10-12 13:21:18,512 INFO o.a.j.e.StandardJMeterEngine: Starting ThreadGroup: 1 : GetVersion
    2017-10-12 13:21:18,512 INFO o.a.j.e.StandardJMeterEngine: Starting 1 threads for group GetVersion.
    2017-10-12 13:21:18,512 INFO o.a.j.e.StandardJMeterEngine: Thread will continue on error
    2017-10-12 13:21:18,512 INFO o.a.j.t.ThreadGroup: Starting thread group… number=1 threads=1 ramp-up=1 perThread=1000.0 delayedStart=false
    2017-10-12 13:21:18,514 INFO o.a.j.t.ThreadGroup: Started thread group number 1
    2017-10-12 13:21:18,514 INFO o.a.j.e.StandardJMeterEngine: All thread groups have been started
    2017-10-12 13:21:18,516 INFO o.a.j.t.JMeterThread: Thread started: GetVersion 1-1
    2017-10-12 13:21:18,517 INFO o.a.j.s.FileServer: Stored: Data/PreOrderCheck_buy.csv
    2017-10-12 13:21:18,522 INFO o.a.j.p.h.s.HTTPHCAbstractImpl: Local host = ProTest
    2017-10-12 13:21:18,524 INFO o.a.j.p.h.s.HTTPHC4Impl: HTTP request retry count = 0
    2017-10-12 13:21:18,741 ERROR o.a.j.JMeter: Uncaught exception:
    java.lang.StringIndexOutOfBoundsException: String index out of range: -1
     at java.lang.String.substring(String.java:1967) ~[?:1.8.0_131]
     at com.doc.jmeter.listeners.elasticsearch.com.doc.jmeter.listeners.elasticsearch.ElasticsearchListener.getSampleResult4External(ElasticsearchListener.java:333) ~[JMeter_ElasticsearchListener.jar:?]
     at com.doc.jmeter.listeners.elasticsearch.com.doc.jmeter.listeners.elasticsearch.ElasticsearchListener.handleSampleResults(ElasticsearchListener.java:140) ~[JMeter_ElasticsearchListener.jar:?]
     at org.apache.jmeter.visualizers.backend.BackendListener.sendToListener(BackendListener.java:274) ~[ApacheJMeter_components.jar:3.2 r1790748]
     at org.apache.jmeter.visualizers.backend.BackendListener$Worker.run(BackendListener.java:246) ~[ApacheJMeter_components.jar:3.2 r1790748]
    2017-10-12 13:21:28,330 INFO o.a.j.t.JMeterThread: Thread is done: GetVersion 1-1

     

    Im using your 0.3 release on Ubuntu 16.04 with jMeter 3.2 and java 1.8.

    (0) 
    1. Vadim Klimov Post author

      Hi Paul,

      Can you please check that both thread group and sampler components in the JMeter plan are named / have labels (this is mandatory for this listener plugin)? For every sample result submitted by the sampler, this listener retrieves information about thread group name in which context it is executed, and sampler name. By default, both of them are submitted to Elasticsearch server so that if there are different thread groups within single test plan and they contain samplers with same names, there will be no confusion about which sample results are analyzed (documents / entries in Elasticsearch could be filtered further by thread group name / sampler name).

      From the exception you provided, it looks to me as the thread group is named / has label ‘GetVersion’, but sampler is not named / doesn’t have label.

      To illustrate this idea, the below extract of the test plan, containing thread group and sampler in it, is correct:

      In contrast, the following extract of the test plan (label of the sampler being removed) will cause error:

      Regards,

      Vadim

      (0) 
      1. paul velzeboer

         

        Hi Vadim,

         

        thanks for your replay and its solve my issue 🙂 the problem was that the thread and the sampler both had the same name so i changed the sampler name and after that it works!

        thanks again for the solution

         

        (0) 

Leave a Reply