Results Determination, powered by SAP HANA

[Former Member]

Introduction

During its SAP implementation, Cambridge Assessment faced the challenge of how to implement a new solution to determine candidate examination results.

Following analysis and proof of concept work, a joint Cambridge Assessment and Cognizant Technology Solutions project team is delivering a Native SAP HANA solution, which will represent a significant business change to examinations processing.

Following a proof of concept in 2014 and a full design and build, the solution is currently in the testing phase before rollout in September 2015.

Background – What we do

Cambridge Assessment (part of the University of Cambridge) is a not for profit organisation that creates and administers examinations for learners and assessment organisations across the globe.

Viewed at a distance, the examination process can be broken down into the following steps:

Results Determination - Current Challenges

Results Determination (The process of turning marks into results) involves a number of challenges:

High volume and throughput:

-       10,000,000 candidate exams taken each year

-       800,000,000 marks received each year

-       Peak processing of 7,000,000 marks/hour, 2,000 marks/second

-       Results Determination processes which run over period of days

·     Legacy systems have product structures held in code which limits the business ability to be able to create new products which would ‘break’ the code quickly

·     Complex Business Rules – Calculation of each candidate's results involve the combination of approximately 60 raw marks (in green) and the calculation of 20 new measures (in greenthrough a series of mathematical formulae, as illustrated by the diagram below. 

The Way forward for Results Determination

When Cambridge Assessment decided to implement SAP, replacing its legacy systems, this raised the question of how Results Determination could be achieved in a new technology that would tackle the challenges experienced previously.

Following analysis and proof of concept work, the project team started to consider SAP HANA as an alternative to the initial SAP SLcM implementation. The attraction to HANA was clear:

• HANA database allows Cambridge Assessment product rules to be defined in data, as product data recipes.
• Conceptually, by processing data ‘in memory’ HANA could reduce the amount of time spent in costly Results Determination read and write processes
• HANA hardware is configured to support parallel processing, which supports the high-volume of candidate processing against product rules.
• The columnar data store offers further improvements in the performance of read processes.

Cambridge Assessment engaged in a three-stage Proof of Concept to prove the data-definition concept, volume processing and data transfer in and out of HANA, after which the decision was taken to move forward with SAP HANA.

Solution Architecture

The Results Determination Solution has the following features:

• Results Determination process will be executed within native SAP HANA with raw marks flowing in from a variety of third-party marking systems and additional reference data (for e.g. candidates, centres, booking etc.) fed real-time from SAP Business Suite, third-party databases & excel files upload via user interfaces.
• The end to end process consists of steps such as validation & reconciliation of raw marks followed by execution core calculation algorithms on vast amounts of marks for numerous candidates.
• The results further pass through a serious of post-calculation check points before being packaged and sent back to SAP Business Suite from where the results are published to the candidates.

The technologies involved in the solution are illustrated below:

• The HANA box is built on a single-node IBM X6 processor with SUSE Linux OS and 512GB memory. The appliances (purchased from and installed by Computer Services Integration Ltd), were the first X6 appliances in Europe!
• The box itself is a stand-alone NetWeaver 7.4 on HANA but with no standard package installed. Most of the building blocks are within Native HANA making use of features such as Information Views, stored procedures, column stores etc. with the NetWeaver layer on top primarily used for BRFplus decision tables, user interfaces and job orchestration.
• All of the real-time data intensive operational reporting is built using various reporting tools available within the SAP Business Objects 4.1 suite. The reports and other interactive user interfaces are further combined within a single view on SAP Enterprise Portal.
• The application depends on external data being passed into it at real-time using SAP LT Replication technology from SAP Business Suite and SAP Data Services 4.2 for data from other third-party databases.
• Large volumes of granular raw marks are streamed into the application from a variety of marking sources through the SAP Process Integration middle-ware with volume data buffered in the interim. Examination results once processed & calculated are sent back using SAP Data Services to SAP Business Suite for completion of the business process and to SAP BW & third-party systems to facilitate offline statistical reporting & analytics and long-term storage.
• Although built on a stand-alone system for now, the use of right and advanced systems positions us to easily integrate & scale the application with packaged solutions in future such as SAP Suite on  HANA or BW on HANA if need be.

Interim Project Outcome

As previously stated, the solution is currently in the testing phase, so the business benefits of the project deliverables are unproven. Still the interim project outcomes are as follows:

• Complex business rules shown above have been defined and stored as data defined recipes in HANA, which allows the business the flexibility to change/add product recipes using the definitions of calculation and measure types built and combining them in different ways.
• Compared to previous solutions which involved significant effort to hardcode product structures, HANA with the NetWeaver 7.4 front end delivers a User Interface with the capability to load new product structures.
• Processing throughput has improved by several thousand times and performance on hardware scaled for the initial implementation already exceeds the full performance requirement for Cambridge Assessment.

Existing solution	Target Performance	Current Interim Test Results using HANA
·     Initial SLCM Implementation: <1 Marks/Sec	·     Current phase target: 300 Marks/Sec ·     Final state: 2000 Marks/Sec	·     Current PSV rate: 8000 Marks/Sec ·     Key process speed is business enabler

Further Information

Mark Taylor (Cambridge Assessment) and Anooj Behanan (Cognizant Technology Solutions) recently presented the Results Determination solution to Cambridge Assessment's Regional SAP User Group (HERUG). The recording of the presentation is here: