SAP PA 3.2 Feedback: Awesome tool but lacks on a few basics.

​Dear SAP PA Team,

This comes up as a user / consultant feedback for the SAP PA 3.2, when we tried to use it for implementing a real-time business case end to end in SAP HANA 1610 with APL / PAL installed.

Use Case: To identify invoices which can be cleared or kept open( Classification problem ).

System setup: SAP HANA 1610, APL, PAL, SAP PA 3.2

Data: Training data size 16300 records, test data 1200 records.

ETA for demo: 8 days

Our experience:

Automated models “Expert analytics” in SAP PA 3.2:

1. So we started with the idea to train our model with SAP APL as we were less on time, hence used PA 3.2 as our base. We started with “Expert analytics” because as per documentation and help guidance, this is where we can export the automatic “model”to HANA directly with a few clicks, without much manual hassle.
2. We built a model in “Expert analytics” and saw that the expert model of “Auto classification” gave just 60% of hit rate, without any option to tune it. Fine, we first wanted to try end to end export of atleast one model to HANA.
3. [Feedback]: No option to tune further the “Auto classification” model ? Automated analytics running on same APL kernel has a lot of features but these are missing in “Expert” analytics.
4. Using documentation, we connected our HANA to tool and exported the model to HANA as a procedure, we could not because it gave us an error with absolutely no explanation. For that we raised an SCN ​​post ​and feedback received was that APL and PA should be of similar version. We are already at PA 3.2, what higher can we go ? One expects that the latest version of a tool is backwards compatible.
5. [Feedback]: PA 3.2 lacks backward compatibility for previous versions of HANA in this case. Moreover we are already on HANA 1610 so should not have been a compatibility issue !
6. [Feedback]: The tool should be smart enough to give an error during connection step only that this version of the PA tool is not compatible with the HANA backend.

Non automated models “Expert analytics” in SAP PA 3.2:

1. Had to revert back and built a Naive Bayes PAL model for starters, exported the model. Everything looked fine in the tool.
2. Went to the HANA DB and were shocked to see what’s going here! When it says “Model export”, one expects a prediction function. We got a procedure that simply just queries the model’s prediction result table. Where is the prediction function ?
3. [Feedback]: Model export feature is not exactly a model export but a results export.
4. We tried retraining this model through SQL code as per SAP PAL documentation but that’s very time consuming, if I want to do an accuracy test and all.
5. However, we didn’t have so much time now to try traditional algorithms so PAL was not an option to us anymore.
6. Ok so, till here after wasting nearly 4 days. We came up to the conclusion that we have to try something else. We raised an SCN question again and started looking at the “Automated analytics” as advised to us by an expert, because we were lesser on time for our demo.

Automated analytics:

1. Lost on hopes, we went to “Automated analytics”, created a quick classification model, exported it as HANA UDF and things looked great till here. This is the only section that worked end to end for us.
2. [Question]: However with this feature we don’t know what is the automatic model retraining option available here, we cant keep on creating new UDFs on retrains ?
3. Next, we thought this is a good ground to move and started building complex models, we spent nearly 2 days to tune the model and went as high as 3rd degree.
4. Final Step: We went to “generate code” option and were shocked, there was no option to export HANA SQL for models beyond 1 degree. Why ? We raised an SCN post and came to know that the product lacks it.
5. [Feedback]: Expert analytics needs to have option to export higher degree models as HANA UDF.
6. Till here, we were lost on hopes and just 1 day left for demo. After spending nearly 7 days in all non-modeling tasks, we had to switch back to Automated analytics, degree 1 model using HANA UDF.
7. Quiet an unexpected trip we had with PA 3.2.

General feedback: Overall it’s a good software to play with the data, do analytics, play with graphs( loved it ), do general analytics.

However there’s still a lot of room for improvement. It would be much appreciated, if the tool focuses more on building models which can be easily plugged in to SAP HANA as SAP customers are the first audience of this tool and they would focus more on trying to connect it with ML capabilities given in SAP HANA( PAL / APL ).

I think some of these are really important to be considered from the real project implementation perspective.

Thanks,

Hasan Rafiq