Putting HANA’s Multiple Engines to the test

Have you ever wanted to combine HANA’s many engines but didn’t know how to?
Perhaps you’re still waiting for the right use case?

Here we wanted to identify people or places within unstructured documents, PDF and Word documents. Surface some intelligence, perhaps the relationships between the people.

For the purpose of this blog, we have taken some public documents – company annual reports, UK political manifestos and SAP documentation.

Could this be achieved within SAP HANA? The answer is of course yes 🙂
The process went like this.

1. Acquire the unstructured documents (Smart Data Integration)
2. Identify people with in the documents (Text Analysis)
3. Clean the people identified (Smart Data Quality)
4. Identify linked people (HANA Graph)
5. Search for relationships between documents and people (Text Search & Text Mining)
6. Expose data for consumption (Calculation Views)
7. Combine the Output (SAP Analytics Cloud, Analytics Designer)

1. Acquire the unstructured documents (HANA Smart Data Integration)

Opening the virtual table shows that we have three useful columns, the directory path, the document name and the document itself in a binary format.

The SDI flowgraph below creates a physical table from the virtual table and adds 3 further columns (CATEGORY, MIME, LANG) that will be useful later.

2. Identify people in the documents (HANA Text Analysis)

Using HANA Text Analysis we can turn the unstructured documents into a structured form.  The structured form of the document identifies many different types of entities including people.  To do this we need to create the text index which will create the $TA table for us.  Below we created a .hdbfulltextindex

Looking at the data shows that Text Analysis has found a large number of people (13,996)

We should inspect some of the PERSON entities.

In the result below, we can see that some of the people are actual people – Bill McDermott, Gerhard Oswald, Bernd Leukert, Hasso Plattner and more familiar names, but there are some that do not appear to be people IDE, tion, ing, ment.

3. Clean the people identified (HANA Smart Data Quality)

As the above output shows, correctly extracting the people from the documents is not easy.  There are some entities identified as people that are not really people. It was recommended to me by a colleague Remi ASTIER to perform some Data Quality checks to clean up these names. This will give significantly better quality output.

Additionally, we can include further processing, specify some quality rules what is acceptable. e.g. requires first name and last name.

To do this we created the flowgraph below.

The output of the flowgraph is a table containing the unique people cleaned up. We have added the number of occurrences of each person and the number of documents they appear in.

4. Identify linked people (HANA Graph)

5. Search for relationships between documents and people (Text Search & Text Mining)