Background

In part 1 of this blog series, we saw how –

  • HCI Uses the Twitter Component of Apache Camel
  • Apache Camel examples can be utilized to understand the inner workings of HCI
  • sinceId will enable us to capture only new Tweets from a Twitter hanle
  • twitter4j.status is the format of search results of Twitter adapter

In this blog we will now use the leanings above to update our Integration flow to enable us to Query the timeline of a twitter user for any Delta Tweets.

Integration Flow

/wp-content/uploads/2016/04/46_938232.png

High Level Process

  • As it is important to persist the since_Id, we will use Global Variables of HCI to retrieve and update the since_Id.
  • The Keywords CamelTwitterKeywords and CamelTwitterSinceId are used to set the Search Keywords and SinceId ( Refer to camel documentation for Twitter )
  • The conversion of the Java Array List twitter4j.status into XML is a custom process that uses the twitter4.status class and its various methods.
  • The code is just a snippet on how to use the Twitter4j.status. Not all methods have been used and this can be modified to suit your needs.

Detailed Steps

Step
Configuration and Usage
Timer Start
  • Trigger the Integration Flow with a Timer Start Event
Content Modifier
  • Read the Global Variable since_id persisted in the HCI Data store

/wp-content/uploads/2016/04/47_938233.png

Script
  • Custom Groovy Script
  • Sets sinceID into the messageHeader using key – CamelTwitterSinceId
  • Sets keywords into the messageHeader using key – CamelTwitterKeywords

import com.sap.gateway.ip.core.customdev.util.Message;
import java.util.HashMap;
def Message processData(Message message) {
    def propertyMap = message.getProperties()
    def messageLog = messageLogFactory.getMessageLog(message);
    //Read since_id from the Properties
    String strsince_id = propertyMap.get("since_id")
    messageLog.setStringProperty("strsince_id", strsince_id);
    Long sinceid;
    if (strsince_id==null || strsince_id == "" ){
      sinceid = 1;
      def map = message.getHeaders();
    }
    else{
     sinceid =Long.parseLong(strsince_id);
    }
    def map = message.getHeaders()
    message.setHeader("CamelTwitterKeywords", "from:saphci_integ");
    message.setHeader("CamelTwitterSinceId",sinceid);
    def keywords = map.get("CamelTwitterKeywords");
    return message;
}











Request-Reply
  • Makes call to the Twitter Adapter
  • The Keywords and sinceId will be read from the Message Header set in the  Groovy Script

/wp-content/uploads/2016/04/48_938246.png

Script
  • Custom Script to convert twitter4j.status to a Custom XML Message
  • Loops through each Tweet and then reads corresponding properties to form XML
  • Persists latest twitter ID as since_id into the message property

import com.sap.gateway.ip.core.customdev.util.Message;
import java.util.HashMap;
import twitter4j.StatusJSONImpl;
import java.text.DateFormat;
import java.text.SimpleDateFormat;
import java.util.List;
import java.util.Date;
import twitter4j.Status;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.apache.commons.lang.StringEscapeUtils;
def Message processData(Message message) {
    List<Status> list = message.getBody();
    def messageLog = messageLogFactory.getMessageLog(message);
    DateFormat df = new SimpleDateFormat("yyyy-MM-dd'T'HH:mm:ss");
    df.setTimeZone(TimeZone.getTimeZone("UTC"));
    def propertyMap = message.getProperties()
    String xmlHeader = "<?xml version=\"1.0\" encoding=\"UTF-8\"?><TwitterSearchResult><Resultset>";
    String listOfTweets= "";
    //Read each Tweet and form XML
    for (body in list) {
       String street = " ";
       String countrycode = " ";
       String country = " ";
       String latitude = " ";
       String longitude = " ";
       if (body.getPlace() != null){
           street = body.getPlace().getFullName();
           countrycode = body.getPlace().getCountryCode();
           country = body.getPlace().getCountry();
        }
        if (body.getGeoLocation() != null){
            latitude = body.getGeoLocation().getLatitude();
            longitude = body.getGeoLocation().getLongitude();
        }
        String tweetText = (body.getText());
        String userName = (body.getUser().getName());
        String postText = (body.getText());
        String tweet = "<result><tweetText>"+postText+"</tweetText><tweetID>"+body.getId()+"</tweetID><TweetHandle>"+body.getUser().getScreenName()+"</TweetHandle><userID>"+body.getUser().getId()+"</userID><tweetUser>"+userName+"</tweetUser><tweetDate>"+df.format(body.getCreatedAt())+"</tweetDate><tweetStreet>"+street+"</tweetStreet><tweetCountryCode>"+countrycode+"</tweetCountryCode><tweetCountry>"+country+"</tweetCountry><Longitude>"+longitude+"</Longitude><Latitude>"+latitude+"</Latitude></result>"
        listOfTweets = listOfTweets + tweet;
     }
    String xmlFooter = "</Resultset></TwitterSearchResult>";
    String outputMessage = xmlHeader + listOfTweets +  xmlFooter;
    String id = "";
    //Read tweetID for since_id
    for (body in list) {
        id = body.getId();
        break;
     }
    //Set tweetID into message property
    if(id == null || id == ""){
        String strsince_id = propertyMap.get("since_id")
        messageLog.setStringProperty("strsince_id", strsince_id);
        message.setProperty("since_id",strsince_id);
    }else{
        messageLog.setStringProperty("sinceID", id);
        message.setProperty("since_id",id);
    }
    message.setBody(outputMessage);
    return message;
}










Write Variables
  • Updates the Global Variable since_id with value from the Message Property as set in the Groovy script

/wp-content/uploads/2016/04/49_938247.png

Receiver
  • Pushes Content into a SFTP Channel

Execution of Flow

Scenario Results

Run#1

  • sinceId not set in Global variable.
  • Defaulted to “1” and all tweets are read as per page size set

/wp-content/uploads/2016/04/50_938267.png

/wp-content/uploads/2016/04/51_938269.png

/wp-content/uploads/2016/04/52_938274.png

Run#2

  • sinceId is read from Global Store
  • No new tweets since last read and hence no tweet in output
  • sinceId is updated back into the Global Store with the same value as there is no new Tweet and hence no new ID.

/wp-content/uploads/2016/04/54_938283.png

Run#3

  • User has a new Tweet
  • Tweet is read
  • SinceID is updated into Global Store

/wp-content/uploads/2016/04/55_938307.png

/wp-content/uploads/2016/04/56_938308.png

Run#4

  • No new Tweets
  • sinceID points to the TweetID of previous run Tweet
/wp-content/uploads/2016/04/57_938309.png

Final Note

Through this use case, we had the opportunity to explore Apache Camel and get an insight into the working of Hana Cloud Integration. While this blog focuses on the Twitter example from Apache Camel, there are multiple other examples which which Apache Camel provides which are equally helpful and can help you on your journey of HCI as you start building complex integration patterns.

It is always good to understand how the underlying platform of execution works and I hope this blog inspires you to explore Apache Camel Framework and  use it within HCI.

To report this post you need to login first.

Be the first to leave a comment

You must be Logged on to comment or reply to a post.

Leave a Reply