I had an epiphany the other day. It finally became clear to me that what I was trying to do would never work. I wanted to publish a ‘sentiment analysis’ report based on Twitter, using just Data Services and some reporting tool from the Business Objects toolset. Yesterday I found out – or actually, the person asking the question on SDN pointed out to me – that one critical tool was missing in this solution. Let me explain.
‘Twitter’ analysis demo’s at TechEd and User Group meetings
Earlier this year, I witnessed demo’s about how Twitter feeds were used for creating reports showing ‘the voice of the customer’. I saw the demo first at the SAP User Group meeting, even before BI4 was released. At TechEd, I attended another session with a similar presentation. At both events, the audience was awed by this new capability of DS, and so was I.
Building my own demo (#fail)
Spurred on by what I had seen, and convinced that somehow this was all very relevant and had great business potential, I eagerly waited the BI4 launch and the subsequent installation of a demo environment on one of Bluefin’s internal systems. I couldn’t wait to build my own ‘Twitter’ report, but I do have a daytime job (and a life outside work, my partner keeps reminding me) so I couldn’t really spend as much time as I wanted to set this up. Instead, I asked for someone on the bench to help me out. I explained what I wanted, pointed her in the direction of manuals and the wiki on Text Analysis, and continued with my own work. After a few days my colleague was pulled into chargeable work so she gave me a status update about what she had managed to do. Despite her best efforts (and she is a knowledgeable consultant!) little progress had been made: The manuals had been read, the concept was understood, but very little information was found about how to get started. The question how to get data from Twitter in DS was still unanswered.
The weeks after this I used all my spare moments to look for a step-by-step guide how to connect ‘Twitter’ to Data Services. Call me naive, but I really thought that this was now an out-of-the box solution (and I bet this is the same for most of the other people present at the SAP demo’s) and I was very disappointed that I could not find any documentation about how to do this.
Fast Forward December ’11: The Partnership with NetBase is announced
Although progress had stalled, my ambition to build a Twitter report was still alive, and when I read about the NetBase partnership I was intrigued: How does this fit in the picture, since SAP already has Data Services to do Sentiment Analysis?
A new effort was made to find out more about how Social Media data can be leveraged for sentiment reporting using Data Services. Still, not much more information was available. What I did found though was that someone on an SDN forum had asked more or less the same question as I had. First, I thought the contributor had not done their homework: The question was if DS 4.0 was able to capture data from social networking sites. My answer: Yes, DS comes with sentiment analysis, and I replied with links to the “voice of the customer” example and the text data analysis wiki. “Hang on”, was the reply, “I know all that” (well in fact she was far too polite to actually say that), “but DS starts with analysing data in a database. How do you get your Twitter data etc in a database in the first place?”. The penny still didn’t drop at my end. Surely, you can configure that in DS, can’t you? It was only when she explained about the use of API’s and web crawlers that the fog finally lifted. Of course! How could I have been so naive! No, you can’t actually hook up DS to a Twitter feed or Facebook status updates. You will have to get this data in a database first, and how you do that, well… it looks like you are on your own for that, as there does not seem to be any documentation from SAP how to do this.
Now the partnership announcement with NetBase makes a lot more sense. Netbase is not only able to get the data in a database but is also superior in analysing this data, rendering the same functionality in Data Services obsolete.
Where does that leave Data Services?
With NetBase providing the service for extracting data from social networking sites, and doing the ‘sentiment analysis’ for you, do you still need DS? Well, yes, you do. First of all, Data Services is much more than a tool for Text Analysis. In fact, I would go as far as to say that Text Analysis is functionality in the periphery of the DS toolset. Data Services is the strategic Data Integration platform in a SAP landscape and provides comprehensive ETL functionality. On top of this platform additional tools are available for data quality management, master data management and data governance. The text analysis functionality spans a wider area than ‘just’ social networking sites: It is also capable of analysing your documents on your intranet and shared drives.
In fact, what I hope is that somehow, you will be able to use the results of the NetBase analysis as input in a Data Services dataflow, so you can easily load these results to whatever system it is relevant for, be it CRM, BI or a Portal or web page.
Not giving up on building the ‘Twitter’ report with Data Services
The use of NetBase will come at a price, and not everyone will be willing to pay for this.
I know DS is very capable of doing sentiment analysis, I have seen the demo and I was, and still am, very excited about it. There is just one extra hurdle in the process which needs to be taken, and although this came to me as an unpleasant surprise, I’m sure I will manage. Watch this space to find out more. I will let you know how to do it as soon as I find out.