Skip to Content

I worked on a project recently where one of the requirements was to perform sentiment analysis based on Twitter data and to surface that in meaningful fashion in a dashboard. We wanted to isolate the text tokens that were identified and assign measure values to them based on the number of times they occurred in the data (i.e. a count on each ID), as well as the overall sentiment score that we calculated inside of HANA. We also wanted it to be possible to link back to the source tweets based on a selection (i.e. summary to details level navigation).

Once the idea was baked, I needed a visualization to support the model we had created. I settled on a Word cloud (or Tag cloud if you prefer) using D3 and the D3 word cloud layout by Jason Davies, because, although there’s some argument about the value of Word clouds, for text analysis and isolation of twitter tokens, it seemed to be a good fit. I also noticed that the word cloud was available in Lumira but I couldn’t find a readily acceptable replacement for Design Studio. Primarily this was because I needed to handle a click event to retrieve a list of tweets based on the originating token.

Here is a picture of the Word cloud layout from the site above in case you aren’t familiar:

D3_wordcloud.jpg

It took me a little while to create the layout properly, as some of the examples were incomplete, but eventually I came out with a word cloud that avoided collisions pretty well and that I could attach up to 2 measures to. The size of the tags is derived from a measure of your choice, but I used occurrences of a given text token. If you don’t have a 2nd measure, this will cause the word cloud to render the words in dark gray:

D3_wordcloud_gray.jpg

I pulled the Twitter data from the feed of the JW Marriott Marquis in Dubai, not the customer we were working with, just a really cool hotel that generates some positive buzz on Twitter. In my case, I wanted to use HANA text analysis (I couldn’t find a version of this guide for SPS09) to also derive sentiment from the tags, and color them either green or red depending on whether or not the sentiment is positive or negative (neutral sentiments remain gray). The text analysis tables in HANA contain information about whether a given token is associated with strong positive, weak positive, neutral, weak negative, or strong negative sentiment. The contents look like this:

D3_wordcloud_TA_Table.jpg

By building a calculation view that converted TA_TYPE into a numeric column ranging from -2 (strong negative) to 2 (strong positive). This gave me a chance to use a poly linear color scale in D3:


var colorScale = d3.scale.linear()
      .domain([-2, 0, 2])
      .range(["red", "#5D6770", "green"]);



Once a measure is attached for color, the word cloud looks like this:

D3_wordcloud_color.jpg

As I said before, the buzz is pretty positive about this particular hotel (looking at the pictures I can imagine why!). I borrowed Manfred Schwarz‘s D3 on-click event to link back to the details behind a token (i.e. specific tweets) and made it available in a BIAL on-click event:

D3_wordcloud_details.jpg

Finally, I added some Display properties you can set to rotate text tags < 5 characters 90 degrees (as above), randomly, or not at all, as well as a few fonts for the text itself.

I was also able to make the component responsive, using Mike’s solution from the Progress Set example: Resize issue with custom Chart component (SDK)

I tried to make the rotation and text available through BIAL methods but I could only manage it if I deleted all the text elements first. Since the keys weren’t really changing, just the property, the enter() method wasn’t picking it up any other way. This worked but it had the additional (undesired) effect that the cloud would re-compute with each click. It looked cool but was too distracting for the final product. I removed the BIAL methods from the final commit, but you can always use the .wordcloud selector in CSS to change font-family beyond the 3 I provided as predefined options. Check the animated GIF below (click if it doesn’t start) for what it did look like and if anyone has a better idea for the rotation I would take it!

cLx8fA.gif

So there you have it, my first extension contribution! While I work out how to contribute to the development community (needing to read this post: SDK Development Community Git Repository (sdkpackage)) I put the source here:

DesignStudio/com.sap.sample.tagcloud at master · jmsrpp/DesignStudio · GitHub

I’ll update the post once it’s there.

To report this post you need to login first.

29 Comments

You must be Logged on to comment or reply to a post.

  1. Karol Kalisz

    Hi James,

    another beautiful SDK extension – you can make a try to follow the technical blog on how to contribute, add it first to “prototypes”, then I can help you on moving it into databound package. I have made you as contributor in the repository.

    Karol

    (0) 
      1. James Rapp Post author

        That would be great Mike … I have a couple other ideas that I hope to get together once 1.5 is released and can get familiar with the contribution process in the meantime using this extension.

        (0) 
    1. James Rapp Post author

      Hi Karol,

      I followed your blog and video and I’ve been able to add the extension to prototypes on my local machine. I pushed that commit and it shows up:

      sdkpackage/src/org.scn.community.prototypes/res/TagCloud at master · org-scn-design-studio-community/sdkpackage · GitHub

      I guess I also need to commit the contribution.ztl and contribution.xml that were changed by the build.cmd but I am also picking up build.cmd and sdkinstall as unstaged changes as well. Which other files do I need to commit to finish the process?

      Thanks,

      Jim

      (0) 
      1. Karol Kalisz

        Hi James,

        thanks for contribution – your component is integrated in prototypes. You have checked in all component related files, the generated files and also the updated build script were missing, but I have updated them.

        here is the docu: Tag Cloud – SCN Design Studio Community

        you can now finalize it and then I can help you to move it out from prototypes.

        Karol

        (0) 
  2. Team Delfi

    Hi James,

    Great component!

    However I was wondering if it takes into account filters applied to the datasource it is based on. I have tried several things but so far it does not refresh when its datasource is changed.

    Kr,

    Lise

    (0) 
    1. James Rapp Post author

      Hi Lise,

      It should do … filters are applied to the Design Studio data source and I would expect it to trigger an afterUpdate() so that the tag cloud gets redrawn. However, that isn’t happening when I tested it. I’m not sure why. I will look into it and get back to you.

      Thanks,

      Jim

      (0) 
    1. James Rapp Post author

      Hi,

      It’s because I configured it to use the key for some reason 🙂

      cloudLayout = d3.layout.cloud().size([width, height])

               .words(tags)

               .padding(2)

               //.rotate(function(d) { return d.text.length > 5 ? 0 : 90; })

               .rotate(function(d) {

               if (d.tagRotation === “90 degrees”) { return d.text.length > 5 ? 0 : 90}

               else if (d.tagRotation === “Random”) { return d.text.length > 5 ? 0 : ~~(Math.random() * 5) * 30 – 60}

               else { return 0 }

               ;})

               .text(function(d) { return d.key; })

               .font(function(d) { return d.tagFont; })

               .fontSize(function(d) { return wordScale(+d.frequency); })

               .on(“end”, draw)

               .start();

      I should really change this to d.text. Will try to get that done over the weekend.

      (0) 
  3. Team Delfi

    Hi James,

    Thank you for your last correction. However, the tagcloud seem to encounter issues in Chrome and Firefox. I cannot open the component in those two browsers. Any idea how to resolve that issue?

    Kr,

    Josephine

    (0) 
    1. James Rapp Post author

      Hi Josephine,

      Not sure … I used it in Chrome for a demo less than a month ago and haven’t made any other changes to the component since then. What’s the exception in the Chrome console?

      Jim

      (0) 
      1. Mike Howles

        Hey Jim,

        I took the liberty of porting your tag cloud over to new Design Studio 1.6 pack.  Your code logic and properties are all still in tact, and I added a few extras, along with our APS support.

        A preview, I hope you like it:

        /wp-content/uploads/2015/12/tagcloud_849040.png

        (0) 
        1. Fabien AUBERT

          Hi Mike,

          I’m trying to use the tag cloud in 1.6 but I get a weird result. Values are displayed but outside the component itself in the design mode ?? Also nothing is displayed at runtime in Chrome. I have to use IE ?

          Fabien

          Tagcloud.PNG

          (0) 
              1. James Rapp Post author

                This is my component after all … so I should probably look into it. Looks like Mike tested it successfully above so not sure what’s going on.

                Can you check the Chrome console to see if there is an exception being logged? I just got a new PC so need to get Eclipse connected to the community repo again from scratch. Let’s troubleshoot a little bit in the meantime.

                (0) 
                1. Mike Howles

                  Thanks, Jim 🙂 – Hopefully I’ve not lobotomized the component too bad 😉

                  Also, Fabien – I’m wondering if this may be because it looks like it’s in a Panel container?  Panel and Grid containers are notorious when it comes to messing with widths and heights of DIV handlers and giving them a 0 width/heights that the Tag Cloud may not be accounting for.  Can you try again without a Panel component, just to verify?

                  (0) 
                  1. James Rapp Post author

                    Should be ok … I’m using your detectSize() to re-process inside a panel. I’ve used it responsively in the past. That being said, it does look like something that would happen with auto dimensions so will be interesting to see what Fabien comes back with.

                    (0) 
  4. Fabien AUBERT

    I aslo did the test in the M mode. Positioning is better in Design Time but only 1 value is displayed (“WEST”). Compare to classical mode it works in Chrome. Funny is that the position in Chrome and IE are different !! It seems that the introduction of the M will aslo introduce additionnal problems for extensions. Do you plan to support all extensions for both mode or you will prefer to focus on the M mode which is the future ?

    Capture.PNG

    (0) 
    1. Mike Howles

      I am avoiding M mode like the plague until I can verify some critical bug fixes that I hope make it into SP1. Otherwise, I’ll continue to avoid M mode until SP2 or beyond. I think M Mode is the future direction based on what SAP has shared, but it’s too darn buggy in 1.6 SP0 to be quite frank

      For now, yes I think between Jim and I, we can triage this issue in Classic mode, and if it happens to fix itself in M mode, even better!  🙂

      (0) 
  5. Ömer Akgül

    Hi,

    I tried to used tag component. In design time it looks perfect but when run app in browser it gives error like below.

    it gives error borth in local run and portal run.

    Why it gives error ?

    Thanks alot.

    Error message: localhost is not responding due to a long-sunning script.

    Capture.PNG

    (0) 
    1. James Rapp Post author

      Hi Ömer,


      I’m not sure offhand … can you answer a couple of questions?


      • What version of Design Studio are you using?
      • Can you check the F12 developer console to see if there’s an exception occurring?

      Thanks,

      Jim

      (0) 
      1. Ömer Akgül

        Hi James,

        I am using Design Studio 1.6 which is last version and component verion 3.0.  I checked developer console there wasn’t any exception.

        But When I try to reopen design studio it also gives error like below.

        1.PNG

        (0) 

Leave a Reply