The Variety Challenge of Big Data

In the Analytics space, the challenge of query speed is the one the BI tools were most frequently confronted with. The objective is always to get short enough response time to go with the analytical flow users are trying to accomplish. And sometimes running their analysis on very large tables, or on Big Data, could turn into a patience test, patience being a virtue we definitely seem to be short of in this mobile-I-need-it-now world… But focusing on this issue assumed already some work had been done to get those tables up and ready. I am not dismissing the topic of volume of data – this will be the topic of another blog post very shortly- but I would like to get back to the basic: how to access the data and nowadays we cannot assume the format of the data is always tabular.

MongoDB plugin for SAP Lumira

MongoDB is a good example of a non-traditional database: MongoDB stores data in documents, giving organizations more flexibility as they develop and iterate on their applications. Today there are already SQL connectors for MongoDB, the fastest-growing non-relational database as this db-ranking page is showing. For SAP Lumira specifically, you can download a Simba plugin from the SAP Marketplace and start using it to query the data stored in your MongoDB documents with SQL. These connectors do already enable organizations to use BI tools on MongoDB, but they require a flattening step that we are looked into to see if they could be improved and therefore enable more end-users. Indeed, the flattening of the document structure upfront forces the end-user to work with SQL and tables from the get-go while with a proper UI, this necessary step could be instead done only once the user knows what to acquire.


A dedicated data acquisition experience

By developing a process that is better adapted to the documents in MongoDB, we can improve the user experience, improve response time, and make BI more accessible to users by removing the need to know SQL entirely. Our idea was to present our users with a native interface that reflects the structure of MongoDB documents, one that highlights accessible data that they can begin analyzing. We’ve built a nodal interface that allows users to pick and choose fields they would like to bring into Lumira:

Lumira mongo extension.jpg


We have changed the acquisition experience so that the flattening of the source that traditionally would happen even before the user would have defined the query, is now done only once the user has decided what to acquire. This lets the analyst truly benefit from the richness of collections of nested data. This video shows a demo of how this prototype is working:

Try it out for yourself by simply downloading the prototype, and installing it in SAP Lumira through the brand new extension manager. And do let us know about your experience, we are curious to hear from you.


Best,

Paul

To report this post you need to login first.

8 Comments

You must be Logged on to comment or reply to a post.

  1. Alok Soni

    Hi Paul,

     

    This is really great and helpful for our use case with mongo however do you know what authentication mechanism are supported by this extension? Does it work with Plain/LDAP authentication mechanism?

    (0) 
    1. Paul Ekeland Post author

      Hi Alok,

       

      Glad to see you find it useful! For the moment no authentication mechanisms are being supported with this plugin unfortunately.

      Thanks,

       

      Paul

      (0) 
  2. Alper Derici

    Hi Paul,

     

    This looks like a very functional and useful extension. Do you think it could be used for reporting on “Json” and “XML” files?

     

    Best regards,

    (0) 
    1. Paul Ekeland Post author

      Hi Alper,

       

      I think this is an approach that could be used for many semi-structured sources, to help improve the experience. JSon, XML, or any other document-store are good candidates for this kind of dedicated experience.

      Thanks,

       

      Paul

      (0) 
  3. Mike Howles

    Hey Paul,

     

    This looks really impressive.  I just tried to use this in Lumira 1.31 against a local MongoDB 3.4.  While I can connect and see my databases, I cannot see any Collections.  Is this DAE only compatible with a certain version of Lumira and/or MongoDB, or do my Collections have to be structured in a certain way?

     

    Thanks!

    (0) 

Leave a Reply