E2E Testing Part 6: SAP Analytics Cloud / SAP Lumira

[TimoStark]

This is the 6th part of the blog-series about automated testing (end to end and integration-tests) using testcafe. The blog-series has the following parts:

1. Part 1: Overview and first test


2. Part 2: Enhanced Selection, Action and Assertion Possibilities.


3. Part 3: Structure your test and common best-practices


4. Part 4: Code-Coverage E2E tests, Integration Tests


5. Part 5: Integration in CI (Jenkins / Github Actions) / Authentification


6. Part 6: Use testcafe for SAP Analytics Cloud,SAP Lumira


7. Part 7: Use testcafe for SAPUI5 Web Components

Inside this 6th part we will discuss the integration of all what we learned so far in SAP analytics products, namely: SAP Analytics Cloud and SAP Lumira.

Testing of Reporting Applications

In my current experience with BI projects I never saw any report which was automatically tested by integration or end-to-end test. Certainly this is caused by the missing tool-chains (ever tried to implement unit test for Lumira? 🙂 ), but especially also by the missing awareness by the project teams.

I personally don't think that just skipping testing on reporting solutions is the way we should go in future. While we really have to pay attention to not make data-point specific testing (i.E. check if column 20, row 30 has value 1.333,22), smoke-tests can be of great help in the day-to-day business work. Especially for very complex integration scenarios like SAC and Lumira are offering - there are always issues in the daily work.

• Some report don't come up at all, due to connectivity issues (Live Data Connection down? System overloaded)


• Some reports have issues in the scripting functionality


• Changes in reusable widgets are causing crashes


• ...

Especially with a large set of reports, smoke-testing everything manually before deployment can be a pain - Therefore I am a big fan of the idea to at least implement smoke E2E tests for all reporting solutions. These smoke-tests should include:

• Open Report with a predefined very static dataset


• Click through the report, so that at least all charts and tables were visible once


• Validate that all charts & tables are showing data


• In case of scripted functionality: Smoke-Test the most important scripts

In this blog we will therefore discuss testing for the two most prominent/used reporting solutions: SAP Lumira and SAP Analytics Cloud.

SAP Analytics Cloud

Remark: SAC heavily relies on service workers. Service-Worker support is a pretty new feature in testcafe. Currently there are two open bugs, which are preventing testcafe to search with SAC. Fixes you have to apply manually are described in the corresponding issues (Issue 2538, 2524).

SAC is the leading analytics solution by SAP. SAC is based on SAPUI5, therefore in theory the ui5 enhancements for testcafe can be used directly.

Unfortunately SAC makes testing very complex, by not providing any static-ids and a big set of custom controls, which are not following UI5 standard about data-storage and binding. While it is completely understandable that a big solution like Analytics Cloud needs custom controls and solutions for high speed access, it makes testing a little bit more complex. Additionally it is very annoying that no single element can be addressed by normal-UI5 ids. And that is not only valid for your developments (e.g. Charts, ..) but also for basic functionality like page-filter.

Anyways there are of course ways to identify elements and charts - these are however not provided via public interfaces - therefore a word of warning: Technically as of now the integration with Cloud Analytics is working - if SAP is deciding to change their ID generation, you might have issues. Therefore please really focus on smoke-tests for the moment.

Technically SAC is using a Component/Widget based approach. Everything you are building - no matter if it is a text element, chart or table is an own widget, with own controller and implementation logic. All data and properties are encapsulated inside this widget (which also allows the wonderful option to write own widgets).

Startup: Similar to Lumira and Launchpad the ui5 enhancements library is providing a startup script which you can utilize. It is logging in with the provided username and password, and waits for the story to be available.

ui5Test('Open Report', async u => {

    await ui5SAC.startup({ user: "username", password: "password" });

    //...

}); 

Similar to the other logins this is only providing basic authentication (SAP-User Login). If you have 2-factor authentication or custom login-authentication you have to develop the authentication via typescript and NodeJs as described in blog 3.

Selecting Elements: As already said SAC is constantly using auto generated IDs. I personally havn't found any way of overwriting them in Stories or Analytical Applications. Every single widget however has a static Widget-ID, which can be accessed via public-API.

 

Based on this widget-id we can "navigate into" the widgets and identify data and selectors for the given Chart, Table, Text [...]. In this example we will utilize this knowledge using a very simple SAC Analytical Application, providing two charts and one table.

 

The test execution will look like in the following gif:

We want to do a very simple test, by adding an automatically calculated column to the table (Accumulative Sum), and filtering on a data-point in the second chart.

Let's start with the table, and the check if the table is initially filled:

const tableSelector = ui5().sac().widgetId("da7cc240-d431-4561-884a-27c3cfe565fa").widgetTable();

const dataTbl = await tableSelector.data();

await u.expectAny(dataTbl.sac.tableData.length).greater(0, "Data must be filled on startup");

• We are using the known standard API, but adding the "sac()" instruction to provide SAC specific selectors


• We are selecting using the widget-id, and provide the selector the information, that this ID is a table.


• We are retrieving the data assigned to this control using the standard API


• All SAC specific data is stored inside the "sac" attribute. TableData is containg the column/row based selection of the table. There are of course additional datasets as shown in the following screenshot

After knowing that we have data in our table (assertion 1), we are trying to add a key-figure (btw: This testcase does not really make sense, as we are actually testing functionality of SAC, but for the sake of this example, we are doing such a test).

await u.rightClick(tableSelector.dataPoint(2, 2));

await u.hover(ui5().element('sap.fpa.ui.control.commons.MenuItem').property('text', 'Add calculation'));

await u.hover(ui5().element('sap.fpa.ui.control.commons.MenuItem').property('text', 'Accumulative Sum'));

await u.click(ui5().element('sap.fpa.ui.control.commons.MenuItem').property('text', 'Repeating'));

• We have to rightclick on the Key-Figure name, to add a calculated column to it. We are using the data-point coordinates here. Remark: we could also use the Name of the content.


• Afterwards we have to hover through the menu, to press the Repeating accumulative Sum for the selected key-figure.

For the chart we first have to identify the first data point, to be able to select on the coordinates.

const intf = ui5().sac().widgetId("6ddc7d54-5e1b-4e01-b76e-adb0d54840e9").widgetChart();

const chartData = await intf.data();

const firstDataPoint = chartData.sac.dataPoints[0];



//rightclick on data point

await u.rightClick(ui5().sac().widgetChart().dataPoint(firstDataPoint.xValue, firstDataPoint.yValue).widgetId("6ddc7d54-5e1b-4e01-b76e-adb0d54840e9"));



//filter

await u.click(ui5().element('sap.fpa.ui.control.commons.MenuItem').property('text', 'Filter'));

• For the chart we are using the Widget-ID API again


• We are again getting the assigned data of the chart, and afterwards rightclick via the coordinates of the first data-point (we cloud ofc also make some complex logic here


• To filter we are again using the "normal" API selecting over the text

After getting the data we want to remove the filter again. To make our life easier, the "data" function is returning a relatedElements object, which is a list of UI5-IDs, of relevant elements "around" the chart.

const dataAfterFilter = await intf.data();

await u.hover(ui5().id(dataAfterFilter.sac.relatedElements.filterOpenLink));

await u.click(ui5().id(dataAfterFilter.sac.relatedElements.filterDeleteLink));

• We first have to hover over the Filter-Link to be able to see the "delete filter" button


• Afterwards we can simply press the delete button

SAP Lumira

SAP Lumira was the leading analytics solution by SAP - before SAC entered the game 🙂 Anyhow SAP Lumira solutions are of course still available at a lot of customers. We will start with SAP Lumira as the testing is more straightforward and simple. SAP Lumira reports are based on UI5 and provides a pretty easy interface to identify Lumira elements.

Unfortunately in this chapter I can not share the source code or videos of testing SAP Lumira (sensitive data) - but at least we can discuss the API you can use for developing tests with SAP Lumira.

Startup: Normally Lumira applications are implemented within a Business Objects server. Identical to SAP Launchpad, the ui5 enhancements framework is offering an API to login into a Business Objects Server. The startup script is expecting a username and password. Optionally you can provide parameters, in case you have a prompt screen on startup.

await ui5Lumira.startup({

       password: user.pw,

       user: user.user,

       parameter: [ { number: 0, value: "0000000000 } ]

});

This will login into the following Dialog and wait until the report is up and running (by waiting until the busy indicator is not visible anymore).



Creating Selectors: First the very good news: SAP Lumira is reusing the IDs you are providing in SAP Lumira Designer inside the UI5 application. The main rule is therefore to provide meaningful IDs during development. You can utilize these IDs inside the test selector than.

In the following example we are checking, that the element with the id CROSSTAB_TEST is visible.

u.expectVisible(ui5().id("CROSSTAB_TEST")).ok();

Assertions: The u.data() function is giving you a lot of insights for lumira specific data. Most specifically the following data will be filled depending on the object type:

• For all Lumira-Elements:
  
  
  
  • metadata.lumiraType: The Lumira Element Type (e.g. CROSSTAB, CHART, TEXT, ..)
  
  
  • identifier.lumiraId: The static ID you've provided inside the lumira designer
  
  
  
  


• For Crosstabs:
  
  
  
  • lumiraProperty.numberOfDimensionsOnRow: Number of dimensions on row axis
  
  
  • lumiraProperty.numberOfDimensionsOnCol: Number of dimensions on column axis
  
  
  • lumiraProperty.numberOfRows: Number of row loaded
  
  
  • lumiraProperty.numberOfCols: Number of columns loaded
  
  
  • lumiraProperty.numberOfDataCells: Number of cells loaded
  
  
  
  


• For Charts:
  
  
  
  • lumiraProperty.chartTitle: the chart title
  
  
  • lumiraProperty.chartType: the chart type
  
  
  • 
    
    lumiraProperty.dimensionCount: the amount of dimensions inside the chart
  
  
  • lumiraProperty.numberOfDataCells: Number of available data-points
  
  
  • lumiraProperty.dimensionsCount: Number of used dimensions
  
  
  • lumiraProperty.measuresCount: Number of used measures
  
  
  • feeds: Contains all used feeds inside the Chart control
  
  
  
  


• For Charts and Crosstabs
  
  
  
  • data: Contains all the data shown inside the table or char
  
  
  
  

As always, to get the data just use the data API.

const data = await ui5().id("CROSSTAB_TEST").data();

The following two screenshots will give you an idea about what kind of data is published:

Of course you can now do normal assertions based on both the properties and assertions from here, as shown within the next code-sample.

await u.expectAny(data.lumiraProperty.numberOfDataCells).greater(0,"we expect to at least have one data cell");

await u.expectAny(data.tableData.data[0]["INFO_OBJECT"]).equal("test", "we expect the first row, with col INFO_OBJECT to have a value of test");

Creating Selectors for Charts:

When testing reporting applications you normally also want to validate filtering actions. Especially in reporting applications those actions are often caused by selecting any data-point inside a chart. Charts are technically rendered with SVG - therefore they can't be easily selected using any available ui5-selector definition (or even using xPath).

To make your life easier here, the data API is returning a _SELECTOR attribute for every single data-point, which is giving you the complete dom-identifier of the svg element, which is visualizing the data-point. You can use the domQuery, to build the selector and click on the data-point.

const data = await ui5().id("CHART").data();

await u.click(

     ui5().domQuery(data.tableData.data[0]["KEY_FIGURE_TO_CLICK_SELECTOR"]));

 

That's it for this blog post. You should now know, that you can reuse the end-to-end testing library for UI5 also for very complex products like e.g. SAC or SAP Lumira - as long as these products are UI5 enabled. In the last blog post following, we will have a look on the end-to-end testing if you are using SAPUI5 Web-Components.