NOTE: This blog is intended for: Developers, Software Testers and Architects. This includes frameworks built by SAP and the Open-Source Community.

Where all the enterprise software manufacturers are moving towards cloud, most of their services are given as APIs and Web Applications. Creating those Web UIs is an on-going process and maintaining the source codes is a huge burden on the team. In an era where the code repositories are growing big in size, developing a modular software is well adopted by big companies and the start-ups. There’s a need to create quality software codes and for this, companies must move on from a “Waterfall” development to “Agile” software development. Agile methodology follows “Iterative” and “Incremental” software development methods.

In Agile methodology, the development cycles are small and fast-paced. Developing a defect-free and efficient software code requires thorough testing of our software as and when any changes are made to the codes. A Software needs to be tested at code-level as well as module-level, and hence there’re these three types:

  1. Unit Testing,
  2. Integration Testing and
  3. End-2-End UI Testing

This is required to make sure that all the software modules together are working as required.

Test Driven Development (TDD) is one of the Agile methodologies which follows very short development cycles each comprising a test case which needs to be full-filled by writing a working code for it. It follows the Test-First approach of software development. A developer lists the test cases needed by the business and then starts writing just-enough codes to “pass” the test case.

Similar to TDD mentioned above, we have one more method called, Behavioral Driven Development (BDD), which is a development methodology that involves writing down the software behaviors in business oriented terms which bridges the understanding of a Technical Developer and a Business Person.

Any standard software has multiple modules in it, which in itself comprises of many more units. As a developer of a software, it’s our duty to make our code testable and to assist us in this process, there are multiple openly shared Test Automation tools and frameworks available. These frameworks allow the developers to follow TDD effectively.

In this blog, I’m putting down all the available Test Automation tools/frameworks which a developer can use to go Agile. There’s a pattern which is followed to make the blog precise and to-the-point, mentioning the required details and giving an analysis of various tools. All the references of the tool and its website is also mentioned. As a developer you can go through following to get a glimpse of the Tools and its features and further go through the links provided to get a hands on experience.

List of Framework/Tools Covered

  • VYPER – SAP’s Internally Developed Integration and End-2-End Testing
  • Protractor – End-2-End and Integration test Automation
  • KARMA – Test-Runner for Fast-And-Frequent software Unit development
  • Jasmine – Unit Testing Framework
  • Mocha – Unit Testing Framework
  • START – SAP’s internal code-less End-2-End UI Test Automation Tool

(Few more to be added – QUnit and SAHI)

All these tools are well maintained by big sponsors making it a future proof solution to your Test Automation Need. The comparison parameters used are:

  • Test-Case Development Methods
  • Maintainability of Test Cases
  • Code Sharing
  • Customizable Test Cases,
  • Licensing: Open-Source/Premium
  • Community support

 

Let’s dig down each of them one by one –

1 VYPER https://vyper.mo.sap.corp
    Born on February 2016 (Internally Published – https://blogs.sap.com/2016/02/10/sap-ui5-automation/)
    Created By Internal SAP Labs India Team, Bangalore.
    Latest Version VYPER v1.1
       
    Brief VYPER is a test automation framework based on Protractor (http://protractortest.org) for automating UI5 applications. It is a simple browser extension which enables automation of Business scenarios in UI5 control format. It’s based on Selenium WebDriver and Protractor which specifically targets Automation of the UI5 Applications. VYPER has been growing internally with 15+ teams using it to Automate their End-2-End UI Testing.
    Test Development Control Selection based Auto-Script-Generation (i.e. User Selects the UI5 Element and chooses the Action from the Menu). The script is generated and contains the required information of the control which can be used by the Selenium Web Driver to start the Automated execution of the Test scripts.
    Maintainability The scripts can be easily maintained as it has UI5 syntax and can be grouped together for an application.
    Code Storage The scripts which is generated can be easily saved in text files.
    Code Sharing Provides in-built Git Repository Integration for Better Collaboration.
    Easiness Easy to use as it’s a Chrome Extension which can easily be installed from Chrome Store.(Not now though)
    Custom Codes The Scripts generated can be edited and saved.
    Control Identification Uses Selenium Web Driver for locating the UI Controls using the collected UI Properties during recording.
    Installation Local Installation, Pre-configured VM(Premium Option)
    Licensing (Internally Available) Premium
    Used For Automation of Integration and End-to-End scenarios.
    Features Git Integration, Easy to Install and Use, Scripting in UI5, Accurate Control Identification, and Test Script Sync with Protractor.
    Compatibility with UI5 Fully Compatible with UI5 Applications.
    Learning Links

https://vyper.mo.sap.corp/

https://vyper.mo.sap.corp/Vyper/GettingStarted.html

https://github.wdf.sap.corp/i047466/Vyper

 

 

2 PROTRACTOR (with Jasmine) http://www.protractortest.org/
     

 

Fig. Jasmine System Architecture

     
  • Protractor uses Selenium’s WebDriverJS to trigger Web Events by requesting it.
  • It uses Jasmine for its test syntax like ‘describe’, ‘it’, ‘expect’ and ‘toBe’ etc.
    Born on June 2013
    Created By Julie Ralph, Google.
    Latest Version Major version – Protractor v4.0
       
    Brief “PROTRACTOR” is a widely used End-2-End testing framework for AngularJS Applications. It’s a NodeJS program which uses the Jasmine Syntax and Selenium’s WebDriverJS to run a test script. It uses the ‘ControlFlow’ concept to provide coordination between asynchronous JavaScript events/function calls. It’s Customized for AngularJS UI controls with ng-attributes. It basically is a test script runner for AngularJS based Web Application.
    Test Development There’re majorly two types of files in Protractor. Spec.js (for Automation Logic) and Conf.js (Refers to spec.js and Selenium server address along with Browser lists) files. Test script development is done in Spec.js file using Jasmine syntax, i.e. using ‘describe’, ‘it’, ‘expect’ etc. The developer needs to write their own Test script using Protractor’s global objects like ‘browser’, ‘element’ and ‘by’ etc. In the Conf.js file we put the details of selenium server address and list of browsers to be supported.
    Maintainability

The scripts written in Jasmine-like syntax ensures that all the test cases are logically clubbed and in sequence inside the same file. Also you can check this blog on how to write more maintainable Protractor Scripts using the ‘PageObjectsdesign Pattern

Protractor PageDesign for BDD

    Code Storage The test scripts are stored locally and can be executed.
    Code Sharing The code can be shared directly or with tools like Git Repository.
    Easiness As a developer of test scripts, you are expected to know the syntax of Jasmine test framework and be able to write a well-maintained scripts. You should be comfortable with NodeJS package installation steps.
    Custom Codes A developer with basic understanding of JavaScript can easily customize. There’s no Record-n-Play feature here.
    Control Identification The Protractor framework is made for Angular JS Web Apps and can be used to automate Angular JS codes easily. For this it uses Protractor’s global objects like ‘browser’, ‘element’ and ‘by’ etc. which identifies the UI controls, later this can be fed to WebDriverJS to run the scripts. Angular JS UI controls can be searched by ID, Model, Binding and other ng-attributes.
    Installation

Installation can be trickier if new to concept of NodeJS package manager. Kindly refer the tutorial to install Protractor, Selenium’s WebDriver:

http://www.protractortest.org/#/tutorial

    Licensing Open Source as based on NodeJS and AngularJS with huge community support.
    Used For Mainly for End-to-End AngularJS App test Automation.
    Features In-Sync UI Automation with ControlFlow concept, Community Support, Scalable for Large Projects, Use Selenium Grid(Time Saver), Can Run multiple Instances of Test on Multiple OS, Browser at same Time using WebDriver’s Hub/Nodes concepts.
    Compatibility with UI5

Not compatible for UI5 specific Controls. Works only with Angular JS Apps. Can try identifying the UI controls from this tutorial –

Protractor with Angular and Non-Angular Sites

    Learning Links

Writing Better Protractor:

Protractor PageDesign for BDD

http://www.protractortest.org/#/tutorial

Testing with Protractor – YouTube Tutorial

Protractor for Beginners(Part 1 & 2)

Testing for Unit and End-2-End:

https://blog.brunoscopelliti.com/test-angularjs-app-with-karma-jasmine-and-protractor/

Selenium Grid Tutorials:

http://blog.yodersolutions.com/run-protractor-tests-in-parallel/

http://www.guru99.com/introduction-to-selenium-grid.html

 

 

3 KARMA

(Earlier Testacular i.e. Test + Spectacular – with Jasmine)

https://karma-runner.github.io/1.0/

Born on March, 2012 (as TestacularJS – Vojta Jina – YouTube )
Created By Angular JS Team, By Google (Vojta Jina).
Latest Version v1.4.0
 
Brief

‘KARMA’, simply putting is a TDD enabler Test-Runner which runs on NodeJS server and executes the Jasmine Test scripts. It mocks the server requests which otherwise takes long time, thereby making ‘KARMA’ a fast script runner. It can’t be used for E-2-E testing (As it doesn’t has Selenium Integration). Protractor is the Option for E-2-E testing. It can be used to run your scripts on Multiple Browser and Multiple Platforms (Desktop, Tablet, Phones). Specifically it’s used in a situation where you need to make changes to your code and do the fast & frequent testing.

Test Development

Test Scripts can be developed using any available test frameworks, like, Jasmine, Mocha, and QUnit. Details can be found here: https://github.com/karma-runner/karma

Maintainability

Test Scripts written in Jasmine and other frameworks are run on the KARMA server on NodeJS Platform. Scripts can be grouped as per business functionality. Also PageObjects design patterns will be useful while writing the Test Scripts.

Code Storage

Code is Stored locally and can be executed by the KARMA Test Runner locally or remotely(using the KARMA WebDriver Launcher Plugin –

https://github.com/karma-runner/karma-webdriver-launcher)

Code Sharing

Codes can be shared by exporting to Git Hub as there’s no in-built Git Integration.

Easiness

Easy to Write scripts in preferred Test Framework and run it Fast-And-Frequent using the KARMA Test Runner.

Custom Codes

Developer can write and customize the Test Script in JavaScript. There’s no Record-n-Play feature here.

Control Identification

KARMA can be used for AngularJS Apps and only for Unit or Integration Testing. Hence no need for UI Control Identification.

Installation

KARMA can be installed as a Node Plugin using NPM (Node Plugin Manager). For more details:

http://karma-runner.github.io/0.12/intro/installation.html

Licensing Open Source.
Used For

Fast-n-Frequent Unit-Test and Integration Test scenarios for any JavaScript Projects. Run with Jasmine and other Test Frameworks.

Features

Fast-n-Frequent Test Runner, Test on Real Device (Desktop, Mobile, Tablet etc.), Easily plugs-in with Continuous Integration Tools (Jenkins), AngularJS, RequireJS etc. Debug the Test Scripts (Jasmine etc.), Google Talk First Demo: https://www.youtube.com/watch?v=YG5DEzaQBIc

Compatibility with UI5

Can be used to test the JavaScript Units, but not for E-2-E UI testing.

Learning Links

·   Test Runner for Jasmine Test Cases in AngularJS Using KARMA:

http://www.bradoncode.com/blog/2015/05/19/karma-angularjs-testing/

·   Complete Guide for KARMA: http://www.methodsandtools.com/tools/karma.php

·   KARMA – How to get HTML Report of Test Results: https://npmjs.org/package/karma-htmlfile-reporter

·   KARMA & WebStorm IDE Integration: https://www.youtube.com/watch?v=MVw8N3hTfCI

Compare PROTRACTOR & KARMA

·   Protractor is a test Runner which is built on top of NodeJS Server.

·   It uses syntax from Jasmine framework.

·   It can be used with NodeJS applications.

·   Protractor uses Selenium’s WebDriverJS to trigger Web Events by requesting it.

·   It has two files called Spec.js and Conf.js (to refer spec.js file and the selenium server).

·   Protractor is a NodeJS program which uses Jasmine and Runs the test scripts while hosted on Node Server.

·   Protractor uses Jasmine for its test syntax like ‘describe’, ‘it’, ‘expect’ and ‘toBe’ etc.

KARMA is a Test Runner like Protractor as they both use the Jasmine tests.

 

 

4 JASMINE A Testing Framework – https://jasmine.github.io/
Born on September, 2010
Created By Pivotal Labs(USA)
Latest Version v2.5.2
 
Brief

“JASMINE” is a Unit testing Framework based on Behavioral Driven Development (BDD) Concept. It can be used with any JavaScript codes. Jasmine has its own Syntax, Assertion Libraries (Matcher functions) and can be easily integrated with any Programming Language with the help of Language-Plugins available. Jasmine has some global functions like ‘it’ and ‘describe’ in which you can write your testing logic. Other Alternatives to Jasmine are ‘Mocha’ and ‘QUnit’.

Test Development

Development is super-easy with Jasmine with its simple syntax and logical grouping of Test Specifications. You write the Test Cases in the ‘Spec’ files which uses global ‘it’ function to implement the test case. Test Suites can be defined using the ‘describe’ function. All the ‘spec’ test cases are executed using the ‘SpecRunner.html’ file. For more info, refer:

http://howtodoinjava.com/scripting/javascript/jasmine-javascript-unit-testing-tutorial/

Maintainability

Codes written are clubbed in the Test-Suite using ‘describe’ and contains multiple ‘it’ method to implement the Test Cases. Easy to manage the Project.

Code Storage

Code is written in JavaScript files which are stored locally.

Code Sharing

Code is stored locally and can be shared using Git Repository.

Easiness

Jasmine has a very simple syntax for writing Test Specifications and gives elaborated output in the browser. Easy to trace logs help debug the application faster.

Custom Codes

Developer can write their test specs in JavaScript language and customize as they wish. There is no auto-generation of codes.

Control Identification

Jasmine is a Unit Test framework and doesn’t deal with UI Controls. Hence no need for UI Control Identification.

Installation

You can download the Jasmine Framework tools from here:

https://github.com/jasmine/jasmine/releases

Follow this:

http://howtodoinjava.com/scripting/javascript/jasmine-javascript-unit-testing-tutorial/

Licensing Open Source.
Used For

Unit Test, Integration Testing for any JavaScript application.

Features

DOM-Less JavaScript Unit Testing, Any Browser, Widely used with any Test Runner like KARMA and Protractor etc. Custom Matcher functions, Spy (to Imitate any function), For BDD test development.

Compatibility with UI5

Unit test your JavaScript logic, but not for E-2-E UI testing.

Learning Links

Hands-On Guide: https://jasmine.github.io/2.1/introduction

Jasmine with Protractor: http://engineering.wingify.com/posts/e2e-testing-with-webdriverjs-jasmine/

Elaborated Blog:

https://code.tutsplus.com/tutorials/testing-your-javascript-with-jasmine–net-21229

Jasmine for Beginner:

http://howtodoinjava.com/scripting/javascript/jasmine-javascript-unit-testing-tutorial/

Conclusion

·      Jasmine is a Language of Automation Testing of Web Apps.

·      It can be used for Unit and Integration Testing of Web Apps.

·      So Jasmine is a Test Framework while it can also be used along with Protractor or KARMA Runners. They use Jasmine test scripts along with Common WebDriverJS language binding to run the test scripts in an Automated Way for End-2-End testing.

·      ‘Mocha’ is just like ‘Jasmine’ a test framework with Jasmine-like scripting syntax. It’s used along with “KARMA Test Runner” for E-2-E testing. Mocha doesn’t have its own ‘Assertion’ methods, hence uses ‘Chai (http://chaijs.com/)’ or ‘SinonJS (http://sinonjs.org/)’ assertion libraries.

 

 

5 MOCHA A Testing Framework – https://mochajs.org/
Born on November, 2011
Created By Open Source Community(Sponsored by SauceLabs and Yahoo)
Latest Version v3.2.0
 
Brief

“MOCHA JS” is a test library for Unit and Integration testing of JavaScript applications. It is a simple and feature-rich testing framework which runs on NodeJS as well as in Browser (Using HTML Test Runner). It has similar syntax like ‘Jasmine’ and can be used with multiple plugins. It has its own test runner command line utility in NodeJS.

Test Development

Test scripts can be written in JavaScript and customized as per requirement. There’s no Auto-Code-Generation in Mocha framework. Just like Jasmine, Mocha too has global functions like ‘describe’ and ‘it’, thanks to various assertion libraries which it supports (viz. Chai, Expect.js etc.)

Maintainability

It also uses the concept of Test Suites and Specs (or Test Case) which logically clubs the test scenarios in a single file for test management.

Code Storage

Test Scripts can be stored locally and executed in either mode Viz. NodeJS server (Using Mocha CLI (Command Line Interface) utility – https://mochajs.org/) or Browser (Using HTML TestRunner file – https://www.sitepoint.com/unit-test-javascript-mocha-chai/ or https://mochajs.org/#running-mocha-in-the-browser)

Code Sharing

Codes are stored either in Node server local directory structure or as a JavaScript project with HTML Test Runner. Using Git Repository is advised for sharing the files.

Easiness

The learning curve for Mocha is similar to Jasmine as both of these can be either run in NodeJS or independently on browser.

Custom Codes

Developer can write their test specs in JavaScript language and customize as they wish. There is no auto-generation of codes.

Control Identification

‘Mocha’ is a Unit Test framework and doesn’t deal with UI Controls. Hence no need for UI Control Identification.

Installation

The ‘NPM‘ packages for ‘Mocha’ can be installed after having Node on your system, unlike Jasmine, where you can also install your local version and after unzipping the files, you can start off. Mocha Projects follow the Node Directory structure for proper project management.

Licensing Open Source.
Used For

Unit and Integration Testing with an In-Built Test Runner or With KARMA.

Features

Integration with Multiple Plugins and APIs (like ChaiJS, ExpectJS, SinonJS etc.), Asynchronous Script Execution, Hooks (Like beforeEach() and afterEach() functions in Jasmine), Test Result Reporter in multiple output(like Markdown-With TableOfContents, HTML, JSON etc.).

Compatibility with UI5

Unit test your JavaScript logic, but not for E-2-E UI testing.

Learning Links

Installation & Unit Testing: http://unitjs.com/guide/mocha.html and https://mochajs.org/

Using Mocha with HTML Runner: https://www.sitepoint.com/unit-test-javascript-mocha-chai/

Using Mocha for TDD with Node: https://semaphoreci.com/community/tutorials/getting-started-with-node-js-and-mocha (With Color Code Example)

Difference between Mocha and Jasmine Test Framework: https://www.codementor.io/javascript/tutorial/javascript-testing-framework-comparison-jasmine-vs-mocha

 

 

6 START https://wiki.wdf.sap.corp/wiki/display/STARTtool
    Born on March, 2015
    Created By Internally by SAP.
    Latest Version Production Version Available (No Versions Specified)
       
    Brief SAP “START” or “Simple Test Automation for Regression Tests” is a widely used UI Automation Framework developed internally to automate the testing of application in any SAP UI technology (FIORI, UI5, HTML GUI etc). START tool offers generic approach to record user actions to create a test case, this can be executed for different test phases/types. Test automates are created the way end user interacts with the application. While START Tool support applications developed using FIORI, UI5, HTMLGUI etc. It uses eCATT, Chrome WebDrivers, Selenium JAR files, and Other DLLs which are used to record the User actions on the Browser in Windows Platform.
    Test Development

“START” is an absolute-zero coding test case development framework which uses .NET Framework and FIORI-Selenium WebDrivers to record-and-play certain user action on the Web UI. After installing the START Tool(follow the given URL tutorial), the user needs to follow the given tutorial Video on this “Link” to understand the process of capturing the user actions and replaying it to know the test results and logs. Test Scripts which are generated by START Engine are stored on server side reducing the load on client and making test case reusable. Developers can also add their custom codes to the existing test execution flow.

Pre-Requisites

·         .NET Framework 4.0 should be installed

·         Install JDK(64-Bit) and Uninstall 32 Version(Compulsory)

·         IE 10, Chrome-36 and Firefox 30 and higher.

For more detailed and Step-By-Step Installation, visit official Wiki page here:

Install START

    Maintainability Developers need not worry about how the test scripts is getting generated as START is a Code-less and follows Record-n-Play test automation. Test Scripts are stored on Server Side and well-maintained by START Engine utilities. Test Cases can be customized to add certain extra features. Versioning is supported to manage the Test cases.
    Code Storage “START” tool installation has light-client and most of the framework libraries are installed in the server side which reduces the load on client side. Test Scripts are also stored on the Server systems which can be re-used by other developers to re-run the tests.
    Code Sharing Being an internal SAP Tool, it has been deeply integrated SAP Systems and SAP eCATT (extended Computer Aided Test Tool) framework. Unlike other Open Source Automation tool, test scripts are stored in the server end which can be used by any other SAP Developer to run the test.
    Easiness Very easy to use and requires no coding background. Functional people can very well use this tool to generate their UI Automation test cases. Integration with SAP test workbench to debug the test cases and Track them later.
    Custom Codes Works on default-provided test libraries of START Framework with flexibility to run test cases with multiple set of inputs. Records the output to a variable which can further be reused for next testing scenarios. Custom codes can be added to support features and complex UI actions. More details: Custom Coding in START
    Control Identification Using the Chrome WebDrivers and .NET Framework tools and FIORI-Selenium JAR files, the START framework identifies the FIORI UI Controls and stores them in a the Selenium Scripts which are run by the START Engine.
    Installation START tool can be installed on any Windows 64-bit system. Follow the link for steps: Installing START Tool
    Licensing SAP’s Internal Tool/Premium
    Used For Code-Less, End-2-End UI Automation Testing.
    Features Code-less Test Case & Record-n-Play Test Automation of FIORI Apps, Multi-Browser Support. For more features have a look at the Wiki here: START Tool Features
    Compatibility with UI5 FIORI (Smart Template) Compatible.
    Learning Links

SAP Internal START Tool Home Page Link: START Home Page

User Guide: START User Guide

Guidelines for Developers: Guidelines

SAP GS Internal START WIKI Link: GS Internal link for START

Conclusion

To conclude the blog for now (as we’ll add SAHI and QUnit), Following are the main feature which makes a good Automation Testing Framework – (among Jasmine, Mocha and QUnit)

  • APIs for Assertion
  • Test Doubles(Using Spy, Stub or Mocks)
  • Asynchronous Testing
  • Faking Server Responses
  • Test Runner or Integration with Test Runners like KARMA, Protractor etc.

Also as a developer if you’re getting confused about the multiple choices available, here’s an example scenario of choosing the test tools –

  • If Jasmine is a Framework of your choice, then you can go with –
    • KARMA (for the test runner)
    • SinonJS (possibly for its fake server unless your framework provides an equivalent, like $httpBackend if you are using Angular)
  • If you are going with Mocha, you have –
    • CHAI (for assertions libraries)
    • SinonJS (for test doubles and its fake server)
    • KARMA or MOCHA CLI (for the test runner)

Hope you got a basic idea about which tool to use for your Next Big Project!

Feel free to ask questions and provide feedback to improve the blog contents.

 

 

Last Modified 16th Feb, 2017
Document Version 1.1

 

To report this post you need to login first.

7 Comments

You must be Logged on to comment or reply to a post.

  1. Stefan Schnell

    Hello Prakash,

    thank you very much for this interesting information.

    I would like to check the SAP Start Tool, do you have a link for external access?

    Cheers
    Stefan

     

    Addition 2017/02/02:

    Hello Prakash,

    you removed all information about the SAP Start Tool, do you have special reasons to do that?

    Cheers
    Stefan

    (1) 
  2. Aavishkar Bharara

    Hi Prakash,

    Good compilation.

    Can you please also create a good decision matrix to make easy decision on usage of tool based on the user needs.

    Regards

    Aavishkar

     

    (1) 

Leave a Reply