Model Compare as a new feature in PA 2.2
Intro
PA 2.2 has been released, I have recently installed and seen a nice new feature called model compare that exists in the expert side of predictive.
Model Compare
Basically this feature tells you which algorithm has the strongest KI and KR. The algorithm that has the best KI and KR score should be the most accurate. We must remember that KI is a measurement of predictive power while KR is the measurement prediction confidence. The closer these are to 1 the stronger the model is.
Below is an example where data is run through three different algorithms, once run the algorithm with the highest KI and KR will get a star.
The results will display the best algorithm and the supporting KI and KR scores per algorithm.
You will also be able to review the results per algorithm if needed.
Hope you find this helpful.
Thanks.
Concise & clear! Nice one Louis.
Thanks. Wonder who pointed out to me this new functionality 😉
Mystery 🙂
it is a mystery 10 hour version - YouTube
Cheers,
Antoine
PS/ still working on these email questions you sent me 🙂
Also check this nice blog entry from Hillary Bliss here: Best New Features in SAP Predictive Analytics 2.2 | SAP BI BLOG
Best regards
Antoine
Louis De Gouveia
I realized that the model statistics and model comparison cannot be used for time series algorithms, only for classification or regression scenarios.
See our user guide for expert analytics on page 230: http://help.sap.com/businessobject/product_guides/pa22/en/pa22_expert_user_en.pdf
Model Statistics is a component that calculates performance statistics on datasets that are generated by algorithms. It can calculate statistics for two algorithm types, classification and regression.
It would make sense to update the screenshots in this blog to avoid any confusion.
Sorry for the inconvenience.
Antoine
Hi Antoine
Well spotted. KI and KR is used as a measurement for the accuracy of classification and regression algorithms. Time series we should make use of MAPE to determine time series model accuracy.
I have updated the screenshots accordingly.
I look forward to future versions of SAP PA throwing an exception or warning should a logical error occur like this, or maybe even the comparison at MAPE level 😉
Thanks for spotting.
+1. I raised the same ask to engineering teams 2 hours ago :-).
Feel free to raise any product improvement idea in our idea place: Predictive Analytics: Home
I see two potential asks in this area, here we are discussing the first one mostly.
a. making the comparison of time series algorithms possible in expert - at MAPE level
b. bringing time series automated algorithms in expert analytics.
Thanks & regards
Antoine
Hi
Agree fully with point A.
Just created the idea, https://ideas.sap.com/D27885
Please vote up.
I'm new at this area and perhaps I am misunderstanding the concept of KI, but, when I tried to do an action using the SAP PA Association Rules, I got in some cases a negative KI.
How is it possible? And what does it mean?
Thanks in advance.
Hi Jedson, can you please create a dedicated question? Ideally include screenshots and the underlying dataset if possible.
Thanks & regards
Antoine
Sure.
Sorry about that.
No problem - we just need to make sure we comply with SCN rules of engagement