Robin Hagemans already told something about Alliander, the largest electricity and gas network company in the Netherlands, in his blog Revolution ? We are at the start of a energy revolution and need to be able to handle the big data wave that will accompany it. Preparing our network for local energy production and electric cars among others will be a costly exercise. Therefore we are trying to cut down costs and postpone large investments if possible.
One of our largest expenses is the upgrading of substations in our electricity grid. These stations transform electricity from a high voltage to a lower one, so it can be distributed to our customers. When new houses or factories are built in a certain area or when consumption of electricity increases, a substation has to be upgraded to be able to fulfill the hunger for electricity. A, now annual, forecasting process tries to determine the ideal moment to start the upgrading. If we upgrade too soon, we could have spent the money at others things, start too late, we face the risks of blackouts in the affected area. Because of the high gain, the already large amount of data involved in this process coming from our grid sensors (about 1.500.000.000 sensor readings per year) and knowing that the amount of data would increase explosively due to the introduction of smarter grid sensors (measuring each second instead of each 5 minutes), we picked this process as a first test case for SAP HANA. The goals: speed up the process, build a solution that can handle more and more data and improve the forecasting results.
Together with the consultants from SAP Data Science, we build (and brought live in the beginning of 2013) a SAP HANA powered solution for the first part of the process: analyzing last year grid sensor data. This solution consisted of:
- ETL jobs build with SAP Data Services to ingest the grid sensor data in SAP HANA.
- A data model build in SAP HANA with tables and information views to aggregate the large amount of data.
- Two algorithms build in the R programming language which took data out of SAP HANA, analyzed it for outliers and load shifts (one substation temporary taking over the work of another) and applied markers in the original tables if outliers or load shifts were detected.
- A SAP BusinessObjects Dashboards user interface for the business users to analyze the data and filter out other errors themselves.
- Java Servlets which allowed two-way communication between the dashboards and the SAP HANA database so we could store input from the users in the database.
- A handful of SAP BusinessObjects Web Intelligence reports that calculated all the necessary key figures automatically.
SAP BusinessObjects Dashboards user interface for Load Forecasting phase 1
This first phase successfully accelerated this first part of the process by a factor of 2,5 and made way for a second phase. In this second phase data was added about the consumption and production of electricity by large consumers, like factories, the interface was improved even more and a similar application was built for the load forecasting of gas substations.
The most notable difference with the first phase, apart from the even larger amount of data, was the switch from SAP BusinessObjects Dashboards front-end to a SAPUI5 front-end build directly on the SAP HANA database with its XS application server. Not only was it possible to build a more flexible application from an end-user perspective in SAPUI5, but also the application architecture was simplified having only 1 server: SAP HANA. Less application layers also meant a faster user interface and thus happier end-users.
Another simplification in our architecture was the rewriting of the R algorithms in algorithms that used the SAP HANA Predictive Analysis Libraries. Now our algorithms didn’t have to run on a separate R server anymore, but could be executed directly on the SAP HANA database, again eliminating another server.
SAPUI5 user interface running on SAP HANA’s XS engine for Load Forecasting phase 2
The electricity application for Load Forecasting phase 2 went live in February 2014 and the gas application went live in April. Unfortunately due to changes in the organization, it’s hard to tell at the moment what the effect is of working with these new applications, but one thing is sure, we have improved the load forecasting process even further. Next phases will probably include the forecasting part of the business process (which is currently still done in Excel), adding of geo-spatial information and maybe even the introduction of Hadoop in conjunction with SAP HANA to be ready for the big data boom that accompanies our new smart grid sensor.
There’s a lot of work to be done, so it’s great that SAP is making the work of our IT department easier by providing us with a fast in-memory database and a simplified application landscape!
Stefan, Pieter and Robin, The IT Alliander HANA based innovators.
Vote for us at the SAP HANA Innovation Awards