Background
In part 1 of this blog series, we saw how -
- HCI Uses the Twitter Component of Apache Camel
- Apache Camel examples can be utilized to understand the inner workings of HCI
- sinceId will enable us to capture only new Tweets from a Twitter hanle
- twitter4j.status is the format of search results of Twitter adapter
In this blog we will now use the leanings above to update our Integration flow to enable us to Query the timeline of a twitter user for any Delta Tweets.
Integration Flow
High Level Process
- As it is important to persist the since_Id, we will use Global Variables of HCI to retrieve and update the since_Id.
- The Keywords CamelTwitterKeywords and CamelTwitterSinceId are used to set the Search Keywords and SinceId ( Refer to camel documentation for Twitter )
- The conversion of the Java Array List twitter4j.status into XML is a custom process that uses the twitter4.status class and its various methods.
- The code is just a snippet on how to use the Twitter4j.status. Not all methods have been used and this can be modified to suit your needs.
Detailed Steps
Step
| Configuration and Usage |
|---|
| Timer Start | - Trigger the Integration Flow with a Timer Start Event
|
| Content Modifier | - Read the Global Variable since_id persisted in the HCI Data store
|
| Script | - Custom Groovy Script
- Sets sinceID into the messageHeader using key - CamelTwitterSinceId
- Sets keywords into the messageHeader using key - CamelTwitterKeywords
import com.sap.gateway.ip.core.customdev.util.Message;
import java.util.HashMap;
def Message processData(Message message) {
def propertyMap = message.getProperties()
def messageLog = messageLogFactory.getMessageLog(message);
//Read since_id from the Properties
String strsince_id = propertyMap.get("since_id")
messageLog.setStringProperty("strsince_id", strsince_id);
Long sinceid;
if (strsince_id==null || strsince_id == "" ){
sinceid = 1;
def map = message.getHeaders();
}
else{
sinceid =Long.parseLong(strsince_id);
}
def map = message.getHeaders()
message.setHeader("CamelTwitterKeywords", "from:saphci_integ");
message.setHeader("CamelTwitterSinceId",sinceid);
def keywords = map.get("CamelTwitterKeywords");
return message;
}
|
| Request-Reply | - Makes call to the Twitter Adapter
- The Keywords and sinceId will be read from the Message Header set in the Groovy Script
|
| Script | - Custom Script to convert twitter4j.status to a Custom XML Message
- Loops through each Tweet and then reads corresponding properties to form XML
- Persists latest twitter ID as since_id into the message property
import com.sap.gateway.ip.core.customdev.util.Message;
import java.util.HashMap;
import twitter4j.StatusJSONImpl;
import java.text.DateFormat;
import java.text.SimpleDateFormat;
import java.util.List;
import java.util.Date;
import twitter4j.Status;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.apache.commons.lang.StringEscapeUtils;
def Message processData(Message message) {
List<Status> list = message.getBody();
def messageLog = messageLogFactory.getMessageLog(message);
DateFormat df = new SimpleDateFormat("yyyy-MM-dd'T'HH:mm:ss");
df.setTimeZone(TimeZone.getTimeZone("UTC"));
def propertyMap = message.getProperties()
String xmlHeader = "<?xml version=\"1.0\" encoding=\"UTF-8\"?><TwitterSearchResult><Resultset>";
String listOfTweets= "";
//Read each Tweet and form XML
for (body in list) {
String street = " ";
String countrycode = " ";
String country = " ";
String latitude = " ";
String longitude = " ";
if (body.getPlace() != null){
street = body.getPlace().getFullName();
countrycode = body.getPlace().getCountryCode();
country = body.getPlace().getCountry();
}
if (body.getGeoLocation() != null){
latitude = body.getGeoLocation().getLatitude();
longitude = body.getGeoLocation().getLongitude();
}
String tweetText = (body.getText());
String userName = (body.getUser().getName());
String postText = (body.getText());
String tweet = "<result><tweetText>"+postText+"</tweetText><tweetID>"+body.getId()+"</tweetID><TweetHandle>"+body.getUser().getScreenName()+"</TweetHandle><userID>"+body.getUser().getId()+"</userID><tweetUser>"+userName+"</tweetUser><tweetDate>"+df.format(body.getCreatedAt())+"</tweetDate><tweetStreet>"+street+"</tweetStreet><tweetCountryCode>"+countrycode+"</tweetCountryCode><tweetCountry>"+country+"</tweetCountry><Longitude>"+longitude+"</Longitude><Latitude>"+latitude+"</Latitude></result>"
listOfTweets = listOfTweets + tweet;
}
String xmlFooter = "</Resultset></TwitterSearchResult>";
String outputMessage = xmlHeader + listOfTweets + xmlFooter;
String id = "";
//Read tweetID for since_id
for (body in list) {
id = body.getId();
break;
}
//Set tweetID into message property
if(id == null || id == ""){
String strsince_id = propertyMap.get("since_id")
messageLog.setStringProperty("strsince_id", strsince_id);
message.setProperty("since_id",strsince_id);
}else{
messageLog.setStringProperty("sinceID", id);
message.setProperty("since_id",id);
}
message.setBody(outputMessage);
return message;
}
|
| Write Variables | - Updates the Global Variable since_id with value from the Message Property as set in the Groovy script
|
| Receiver | - Pushes Content into a SFTP Channel
|
Execution of Flow
| Scenario | Results |
|---|
Run#1 - sinceId not set in Global variable.
- Defaulted to "1" and all tweets are read as per page size set
| 
|
Run#2 - sinceId is read from Global Store
- No new tweets since last read and hence no tweet in output
- sinceId is updated back into the Global Store with the same value as there is no new Tweet and hence no new ID.
| 
|
Run#3 - User has a new Tweet
- Tweet is read
- SinceID is updated into Global Store
| 
|
Run#4 - No new Tweets
- sinceID points to the TweetID of previous run Tweet
|  |
Final Note
Through this use case, we had the opportunity to explore Apache Camel and get an insight into the working of Hana Cloud Integration. While this blog focuses on the Twitter example from Apache Camel, there are multiple other examples which which Apache Camel provides which are equally helpful and can help you on your journey of HCI as you start building complex integration patterns.
It is always good to understand how the underlying platform of execution works and I hope this blog inspires you to explore Apache Camel Framework and use it within HCI.
