Skip to Content

Following up from Looking at my own water consumption with SAP Lumira & SAP Predictive Analytics this time I tried using SAP Lumira Desktop first.  I copied my electricity usage and average daily usage from my online bill and copied that to Lumira.  SAP Lumira has a predictive features but you need a time hierarchy and have to use a line chart.  Your other option is to use SAP Predictive Analytics.

Here’s the sample raw data:

/wp-content/uploads/2015/07/sample_751992.jpg

I bring that into SAP Lumira, create a time hierarchy on meter reading date, and then a line chart.

/wp-content/uploads/2015/07/1fig_751993.jpg

The gear then shows the options to forecast with SAP Predictive or Linear Regression.

/wp-content/uploads/2015/07/2fig_752000.jpg

I go with the SAP Predictive Analytics option,, to forecast 6 months out (how will my electric bill look at Christmas time, when the property tax bill is due?)

/wp-content/uploads/2015/07/3fig_752001.jpg

So you can see from the above my electricity is high in the winter (I have electric heat). I’ve looked at less inexpensive options for heat in the winter but they are not available in my area.  In the summer, my electricity is low, as I don’t like having the air conditioner on (this while we’re undergoing a heat advisory today)

The forecast for December looks similar to December 2014, so on the surface the forecast looks pretty accurate.

/wp-content/uploads/2015/07/4fig_752002.jpg

The forecast for average daily usage for December 2015 is similar December 2014.  This has allowed me to take a closer look at electricity, one of my most expensive bills.

It’s not enough (in my opinion) to enter the Data Genius contest, but maybe this will give you some ideas.

Reference

Tomorrow: Bridging the Business Analyst-Data Scientist Gap using SAP Lumira and SAP Predictive ASUG Webcast

DataGenius IV: Home

Upcoming ASUG Business Intelligence Community W… | ASUG

Add these SAP Lumira Sessions at #SABOUC ASUG Conference in Austin

To report this post you need to login first.

9 Comments

You must be Logged on to comment or reply to a post.

  1. Aaron Williams

    this is really cool.  I wonder if you could pull the data from your utility via an API.  And if so, there will soon be a hundred iPhone/Android apps to do this. 

    (0) 
  2. Ethan Jewett

    Hi Tammy,

    As always with you, this is a great demo of the functionality of the tool. It also nicely illustrates some of the problems with the tool and a bit of the danger of SAP’s drive to include predictive features without thinking really really hard about the defaults and without putting guard rails in place to protect users from mistakes. For SAP’s use, here are the issues I see right off the bat:

    1. By far the biggest problem: There is no prediction interval provided for the forecast and no confidence interval for the model fit with actuals. How can a predictive tool not at the very least provide a confidence interval and at least an estimated prediction interval to give some sense of the reliability of its forecast?
    2. The SPA forecast appears to be using the wrong auto-correlation model (it probably shouldn’t be using one at all, but rather using a pure seasonality model), I assume, but doesn’t give you options to choose a different model or parameterize the model differently. One result of the wrong model choice is that the forecast lags the actual by about a month.
    3. Probably another symptom of bad model choice: the forecast for summer 2015 regresses to the mean, when the historical data indicates that usage should stay low throughout the summer. Indicates a very poor modeling or parameterization choice, but you are never given the choice, so this isn’t the user’s fault!

    Hopefully we’ll see a real effort to include more responsible predictive features in the … wait for it … future.

    Best,

    Ethan

    (0) 
    1. Ethan Jewett

      Actually, I should amend this. After asking someone who actually knows about this stuff, #3 isn’t so much of a problem. These forecasts will usually regress to the mean. #3 is actually more of a symptom of #1 (lack of error bars), as the reason it regresses to the mean so fast is probably that there is so little data, so the prediction error bars are likely huge after the first couple of months.

      I’d also suggest that application should explicitly warn the user when attempting to forecast based on insufficient data. In the case of a seasonal model, I’d think at least a few cycles would be necessary.

      (0) 
      1. Antoine CHABERT

        Hi Ethan,

        Here is my point of view on your comment. 

        By far the biggest problem: There is no prediction interval provided for the forecast and no confidence interval for the model fit with actuals. How can a predictive tool not at the very least provide a confidence interval and at least an estimated prediction interval to give some sense of the reliability of its forecast?

        I agree with you, information is lacking to the end-user to properly interpret the results he is getting.

        Running the same tests using the latest version of our SAP Predictive Analytics (Automated Analytics mode) solution provides the end-user with all the missing information, especially in terms of model components and quality, estimated forecast & reliability.

        Please refer to the content that I provided in appendix for more details.

        The SPA forecast appears to be using the wrong auto-correlation model

        (it probably shouldn’t be using one at all, but rather using a pure seasonality model),

        I assume, but doesn’t give you options to choose a different model or parameterize the model differently.

        One result of the wrong model choice is that the forecast lags the actual by about a month.

        It is a core strength of our SAP Predictive Analytics product to automate the choice of the model for the end-user.

        The average quality of the model that is generated in this specific example is due:

        1. to the small size of the dataset being used
        2. to the fact that there are no other input variables that can be used beyond the sole date to predict the electricity consumption. Other useful input variables could be the outside temperature, the number of days where a person is in its house, the average number of hours that the sun is shining in the month… etc….

        SAP Predictive Analytics (Automated Analytics mode) lets you use other input variables as well, which can improve the quality of the time series model. This is not possible in the current SAP Lumira integration.

        I’d also suggest that application should explicitly warn the user when attempting to forecast based on insufficient data. In the case of a seasonal model, I’d think at least a few cycles would be necessary.

        For me some potential axis to improve the current integration of predictive into Lumira should be:

        • telling the end-user what is the maximal limit of reliable forecasts that can be made. Today there is no safe guard or warning.
        • telling the end-user about the quality of the model that was generated.
        • being able to display a prediction interval for the values that are forecasted. 
        • being able to use other variables (not only the time hierarchy) in order to improve the model and forecast accuracy.

        As I have illustrated in my answer, all these features are covered by SAP Predictive Analytics (Automated Analytics mode).

        Best regards & thank you,

        Antoine

        Appendix

        In the appendix, I detail the results I get on the dataset that Tammy used, using our SAP Predictive Analytics 2.2 (Automated Analytics mode).


        Usage (kWh)

        model horizon 3.png

        The model generated enables only to forecast with reliability the next three months (maximal horizon). Not the next six months.

        model performance.PNG

        The model takes into account only 32% of the signal, as the horizon-wide MAPE is equal to 0,683.

        model components.PNG

        The model is purely auto-regressive (see AR(2) mention) – no trend, no cycles detected.

        Forecasts vs Signal.jpg

        Here is the confidence interval around the forecasted values. You can see the interval around the forecast is quite huge. In the table below, you see the forecasted values, err_max and err_min for the three coming months.

        Meter Read Date

        Usage (kWh)

        Usage (kWh)_Forecasts

        Err_Max

        Err_Min

        2015-07-19

        459,99

        1363,514

        -443,53

        2015-08-19

        666,21

        2240,7

        -908,284

        2015-09-19

        779,98

        2552,503

        -992,541

        Avg. Daily Usage

        model horizon 3.png

        It means that the model enables only to forecast with reliability the next three months. Not the next six months.

        Model Components and Performance.PNG

        The model takes into account only 51% of the signal.

        Forecasts vs Signal 2.jpg.png

        Here is the confidence interval around the forecasted values.

        You can see the interval around the forecast is quite huge.

        In the table below, you see the forecasted values, err_max and err_min for the three coming months.

        Table 2.PNG

        (0) 
        1. Ethan Jewett

          Hi Antoine,

          Thanks for the detailed response. I basically agree with everything your are saying and I appreciate you showing exactly how it works in PA as opposed to Lumira. In this case, if the Lumira integration had surfaced information about either a prediction interval (since it’s so large) or the model choice (because it’s non-cyclical and the user knows that this is a cyclical process), it would have been clear to the end-user that this forecast was not reliable.

          I’m aware that PA is a good tool for this kind of thing, but it’s also clear that the integration of predictive functionality into Lumira is severely and dangerously flawed as it currently stands.

          Thanks,

          Ethan

          (0) 
          1. Antoine CHABERT

            Hi Ethan,

            My personal point of view is that the Lumira integration has room for improvement, in the directions I outlined in my answer:

            • telling the end-user what is the maximal limit of reliable forecasts that can be made. Today there is no safe guard or warning.
            • telling the end-user about the quality of the model that was generated.
            • being able to display a prediction interval for the values that are forecasted.
            • being able to use other variables (not only the time hierarchy) in order to improve the model and forecast accuracy.

            Thanks

            Antoine

            (0) 

Leave a Reply