Skip to Content
Author's profile photo Former Member

Results Determination, powered by SAP HANA

Introduction

During its SAP implementation, Cambridge Assessment faced the challenge of how to implement a new solution to determine candidate examination results.

Following analysis and proof of concept work, a joint Cambridge Assessment and Cognizant Technology Solutions project team is delivering a Native SAP HANA solution, which will represent a significant business change to examinations processing.

Following a proof of concept in 2014 and a full design and build, the solution is currently in the testing phase before rollout in September 2015.

Background – What we do

Cambridge Assessment (part of the University of Cambridge) is a not for profit organisation that creates and administers examinations for learners and assessment organisations across the globe.

Viewed at a distance, the examination process can be broken down into the following steps:

HLProcess3.jpg.png


Results Determination – Current Challenges

Results Determination (The process of turning marks into results) involves a number of challenges:

High volume and throughput:

–       10,000,000 candidate exams taken each year

–       800,000,000 marks received each year

–       Peak processing of 7,000,000 marks/hour, 2,000 marks/second

–       Results Determination processes which run over period of days

·     Legacy systems have product structures held in code which limits the business ability to be able to create new products which would ‘break’ the code quickly

·     Complex Business Rules – Calculation of each candidate’s results involve the combination of approximately 60 raw marks (in green) and the calculation of 20 new measures (in greenthrough a series of mathematical formulae, as illustrated by the diagram below. 


Sample Recipe.png


The Way forward for Results Determination

When Cambridge Assessment decided to implement SAP, replacing its legacy systems, this raised the question of how Results Determination could be achieved in a new technology that would tackle the challenges experienced previously.

Following analysis and proof of concept work, the project team started to consider SAP HANA as an alternative to the initial SAP SLcM implementation. The attraction to HANA was clear:

  • HANA database allows Cambridge Assessment product rules to be defined in data, as product data recipes.
  • Conceptually, by processing data ‘in memory’ HANA could reduce the amount of time spent in costly Results Determination read and write processes
  • HANA hardware is configured to support parallel processing, which supports the high-volume of candidate processing against product rules.
  • The columnar data store offers further improvements in the performance of read processes.

Cambridge Assessment engaged in a three-stage Proof of Concept to prove the data-definition concept, volume processing and data transfer in and out of HANA, after which the decision was taken to move forward with SAP HANA.

Solution Architecture

The Results Determination Solution has the following features:

  • Results Determination process will be executed within native SAP HANA with raw marks flowing in from a variety of third-party marking systems and additional reference data (for e.g. candidates, centres, booking etc.) fed real-time from SAP Business Suite, third-party databases & excel files upload via user interfaces.
  • The end to end process consists of steps such as validation & reconciliation of raw marks followed by execution core calculation algorithms on vast amounts of marks for numerous candidates.
  • The results further pass through a serious of post-calculation check points before being packaged and sent back to SAP Business Suite from where the results are published to the candidates.

The technologies involved in the solution are illustrated below:

Technology Diagram2.png

  • The HANA box is built on a single-node IBM X6 processor with SUSE Linux OS and 512GB memory. The appliances (purchased from and installed by Computer Services Integration Ltd), were the first X6 appliances in Europe!
  • The box itself is a stand-alone NetWeaver 7.4 on HANA but with no standard package installed. Most of the building blocks are within Native HANA making use of features such as Information Views, stored procedures, column stores etc. with the NetWeaver layer on top primarily used for BRFplus decision tables, user interfaces and job orchestration.
  • All of the real-time data intensive operational reporting is built using various reporting tools available within the SAP Business Objects 4.1 suite. The reports and other interactive user interfaces are further combined within a single view on SAP Enterprise Portal.
  • The application depends on external data being passed into it at real-time using SAP LT Replication technology from SAP Business Suite and SAP Data Services 4.2 for data from other third-party databases.
  • Large volumes of granular raw marks are streamed into the application from a variety of marking sources through the SAP Process Integration middle-ware with volume data buffered in the interim. Examination results once processed & calculated are sent back using SAP Data Services to SAP Business Suite for completion of the business process and to SAP BW & third-party systems to facilitate offline statistical reporting & analytics and long-term storage.
  • Although built on a stand-alone system for now, the use of right and advanced systems positions us to easily integrate & scale the application with packaged solutions in future such as SAP Suite on  HANA or BW on HANA if need be.

Interim Project Outcome

As previously stated, the solution is currently in the testing phase, so the business benefits of the project deliverables are unproven. Still the interim project outcomes are as follows:

  • Complex business rules shown above have been defined and stored as data defined recipes in HANA, which allows the business the flexibility to change/add product recipes using the definitions of calculation and measure types built and combining them in different ways.
  • Compared to previous solutions which involved significant effort to hardcode product structures, HANA with the NetWeaver 7.4 front end delivers a User Interface with the capability to load new product structures.
  • Processing throughput has improved by several thousand times and performance on hardware scaled for the initial implementation already exceeds the full performance requirement for Cambridge Assessment.

Existing solution

Target Performance

Current Interim Test Results using HANA

·     Initial SLCM Implementation: <1 Marks/Sec

·     Current phase target: 300 Marks/Sec

·     Final state: 2000 Marks/Sec

·     Current PSV rate: 8000 Marks/Sec

·     Key process speed is business enabler

Further Information


Mark Taylor (Cambridge Assessment) and Anooj Behanan (Cognizant Technology Solutions) recently presented the Results Determination solution to Cambridge Assessment’s Regional SAP User Group (HERUG). The recording of the presentation is here:

Assigned Tags

      2 Comments
      You must be Logged on to comment or reply to a post.
      Author's profile photo Rahul Aware
      Rahul Aware

      Hi Mark,

      I am excited about this use case. By data defined recipes in HANA - do you mean decision tables in HANA? If not then did you consider using HANA's native decision table capability? I see you are using BRFPlus at Netweaver level - which also means that rule evaluation happens at application server instead of database - which my experience says is performance intensive as it loops sequentially over the decision table for every decision to be made. So, if you have 1000 records in a decision table- for very evaluation it will loop 1000 times which can't be avoided.

      I would really appreciate if you can give more details around data defined recipes and usage of BRFPlus.

      Thanks in advance and happy HANAfying 🙂

      Regards,

      Rahul.

      Author's profile photo Former Member
      Former Member

      Hi Rahul,

      Apologies for the late response to this question.

      Complex calculation rules (Mark Taylor has added an example screenshot of a moderately complex recipe in the blog above) have been moved from programmatic logic into essentially data. Each of the calculation entities shown in the example has been converted into data entities with calculation types forming its attributes and they have been realized using a combination of HANA information views and stored procedures.

      BRFPlus decision tables have been used for validation of incoming marks. The prior phase (non-HANA & realized using SLCM) had used BRFPlus extensively which the Business had become familiar with and were keen to continue with the new proposed HANA solution. Therefore this was something we had to carry into the new solution and to minimize the potential performance impact (as you rightly pointed out), we have kept the decision table framework of BRFPlus quite simple with them being realized as stored procedures in HANA. The data entities pushed up into BRFPlus are quite minimal and the execution of rules for all marks are entirely pushed down into HANA avoiding the need for loops on marks within the NetWeaver layer. In the subsequent phases of the solution, we are exploring both the latest version of BRFPlus/DSM that's tightly integrated with SAP HANA & SAP HANA Rules Framework (which had not released at the time we were doing our Proof Of Concept of the above solution).

      Hope that helps!

      Thanks,

      Anooj