Textual Analytics and textual Business Intelligence

[Former Member]

Bill Inmon published on his website two documents about textual Business Intelligence and Textual Analytics for download after registering. You can also find the article about Textual Analytics in SDN. 

In summary, classic data warehouses collect mainly numerical data. But in your business many information is exchanged as texts, e.g. mail, contracts, technical and non-technical documentations and many more. The knowledge of this information can lead to better decisions. Bill Inmon uses 360° view for an customer as an example. Through mails, documents etc. you get many more information about your customer than through your ERP which stores numerical values, like day of birth, gender and zip code. Modern search engnies and parsing algorithms allow to extract the textual information and store it into databases. As a result you can analyze the data and visualize it. Bill Inmon demonstrates visualization with a so-called Self-Organized-Map (SOM). You can find a SOM sample applet on the research pages of Laboratory of Computer and Information Science (CIS) in Finland. SOMS are widely used and successfull in the area of data mining. The idea behind a Self-Organized Map are neural network which can be trained. Maybe it's oversimplified but you can imagine a SOM as a heat map. To store, parse and analyze texts in a database summarizes Bill to textual Business Intelligence. Bill concludes that classical BI and textual BI are complementary elements.

Which technology is provided by SAP for textual BI and Textual Analytics?

In my point of view, SAP Enterprise Search and SAP BO Business Explorer allows to search texts and documents for keywords. With integration into Knowledemanagement you can store your documents in SAP Portal. An additional tool is SAP BusinessObjects Text Analysis which allows to parse unstructured data and store it structured in SAP BW. Here is the question, which tool fulfills most of the requirements and should be used. To solve this you can setup a simple decision matrix. Define your requirements, select the available products and appraise for each requirement a numerical value between 1 and 10 for each tool. At the end you get a weighted result and can choose which tools fit your requirements the best.

You may ask, what about SOMs? The good news: SOMs are available. Vienna University of Technology has developed an open-source SOM Toolbox in Java. If you use Eclipse framework as base for integration of the two worlds, it is possible to integrate the SOM Toolbox into SAP applications. If you use Crystal for Eclipse SDK it is possible to set up SOMs with respect to the appropriate SOM viewer, in Crystal Reports or in WebIntelligence. If you convert it into a Webservice you are able to consume SOMs in Dashboards also. With the provided JAVA SOM toolbox you are also able to integrate SOMs into your BEX Web Templates. In the JAVA world, everything is fine. But what about ABAP? The ABAP side is not so bright. As far as I know, there is no neural network in ABAP available. For my understanding right now to setup a SOM in ABAP is a research project.

My conclusion

The integration of structured and non-structured information into an enterprise datawarehouse is the next important step. You can gain lot of new information from your unstructured data. SAP provides a toolset to accomplish that. 

SOMs look really promising and could be next cool and sexy thing in BI Reporting.