How to use the OData-API for exchanging Messages from Logs and the Datastore in Integration Suite

[JoergHopmann]

In this blog I will show the basic steps and components to access the MessageProcessingLog and the attached messages, wether they are stored in the log itself, in the MessageStore or in the DataStore.

Later in further blogs I will go into details and show some specific implementations and will compare the different types of message-stores. So please be patient and check back for further information.

The starting point is in the SAP Business Accelerator Hub. There is an API package SAP Cloud Integration which offers some OData APIs in Version 2. We are focussing on the Message Processing Logs and download the definition of, what is called here a vdm - virtual data model. I don't want to go deeper in the edmx, only so far, if you search on your fav. search-engine you may get the following statement:

edmx file is an XML file that defines an Entity Data Model (EDM), describes the target database schema, and defines the mapping between the EDM and the database.

On the Business Accelerator Hub you can find tons of documentation about how to use the source code generator in maven, on how to build up (in this case) java-classes, test the API online, with a sandbox or with your own tenant and even code-snippets in many different languages can be downloaded.

In our scenario we build a client in the Netweaver Development Studio. Then maven-support is initially not part of it but can be added. Search for the keyword M2E. A good starting point on how to generate the client classes (vdm - virtual data model) and how to generate a project from a Maven archetype is on sap.github.io .

The overall framework for the implementation here is an OnPremise fat-client, but everything can run deployed on a BTP tenant, only the GUI frontend needs to be changed.

What you get when generating the sources for your client is the following helper-packages and classes and a MessageProcessingLogsService:

There are several steps to consider for enhancing your pom-file: Dependencies and plugins. E.g. here: SAP Cloud Application Programming Model: Deep Insert (5) Consume Remote Service with SAP Cloud SDK 

The starting point here is the method getAllMessageProcessingLogs()

Depending on wether the code runs on your local laptop or in the cloud, you have to create a destination first:

import com.sap.cloud.sdk.cloudplatform.connectivity.AuthenticationType;
import com.sap.cloud.sdk.cloudplatform.connectivity.DefaultHttpDestination;
import com.sap.cloud.sdk.cloudplatform.connectivity.HttpDestination;
import com.sap.cloud.sdk.cloudplatform.security.BasicCredentials;

(...)
public DefaultHttpDestination getDestination() throws IllegalArgumentException	{
	if (destination == null)	{
		if (this instanceof BasicAuthentication)	{
			destination = DefaultHttpDestination.builder(tenant.getURL())
				.authenticationType(AuthenticationType.BASIC_AUTHENTICATION)
	   	    	.basicCredentials(new BasicCredentials(((BasicAuthentication)this).getUser(), ((BasicAuthentication)this).getPassword()))
	   	    	.keyStore(keystore)
	   	    	.build();
		}
	}
	return destination;
}

(On your tenant you can use the DestinationService)

With the destination described above we can request all packages and artifacts. As for now the request is sent by getAllIntegrationRuntimeArtifacts(). This includes iFlows but also Rest-APIs and more.

Unfortunately

• runtime-artifacts do not point to any package
• designtime-artifacts are called from a package-perspective but do not include some types like Rest-APIs. 

(Status on 2024-05-24 is a response of HTTP 501 - not yet implemented. the dev is seems still working on putting everything in place)

Hence we have a gap of links between packages and artifacts, in case they are not iFlows (e.g. Rest-Api). The following code puts "everything" together as complete as possible and attaches the artifacts without a link to a package into a virtual package named "<unknown>"

//create Hashtable for storing our version of packages
designPackages = new Hashtable<String, DesignPackage>();

if (authentication != null)	{
	//get url
	String tenantURL = authentication.getTenant().getRuntimeURL();
	
	//create destination
	DefaultHttpDestination destination = authentication.getDestination();
	
	//create the services generated in the vdm
	IntegrationContentService iCS = new DefaultIntegrationContentService().withServicePath("/api/v1/");
	
	//create Hashtable for storing our version of runtime artifacts
	Hashtable<String, IntegrationRuntimeArtifact> ira_table = new Hashtable<String, IntegrationRuntimeArtifact>();
	
	//retrieve all packages, execute the network-request
	try {
		final List<IntegrationPackage> ip = iCS.getAllIntegrationPackages()
				.executeRequest(destination);
		
		//retrieve all runtime artifacts
		final List<IntegrationRuntimeArtifact> ira = iCS.getAllIntegrationRuntimeArtifacts()
				.executeRequest(destination);
		
		//and put them in the table
		for (IntegrationRuntimeArtifact ira_iter : ira)
			ira_table.put(ira_iter.getId(), ira_iter);
		
		//iterate over all packages
		for (IntegrationPackage ip_iter : ip) {
			
			//store it in our Hashtable
			DesignPackage dp = new DesignPackage(ip_iter);
			designPackages.put(dp.getName(), dp);
			
			//get artifacts for this package
			final List<IntegrationDesigntimeArtifact> ida = ip_iter.getIntegrationDesigntimeArtifactsOrFetch();
			
			//set artifacts as child to the package
			for (IntegrationDesigntimeArtifact ida_iter : ida) {
				String url = tenantURL + "/http/" + ida_iter.getId();
				Artifact a = new Artifact(ida_iter, url);
				dp.addArtifact(a);

				//remove from the list of runtime artifacts
				ira_table.remove(a.getId());
			}
		}
		
		//add remaining artifacts not linked to any package
		if (ira_table.size() > 0)	{
			
			//default package
			IntegrationPackage integrationPackage = new IntegrationPackage();
			integrationPackage.setName("<unknown>");
			defaultDesignPackage = new DesignPackage(integrationPackage);
			designPackages.put(defaultDesignPackage.getName(), defaultDesignPackage);
			
			Enumeration<String> t = ira_table.keys();
			while (t.hasMoreElements())	{
				String iraId = t.nextElement();
				Artifact a = new Artifact((IntegrationRuntimeArtifact)ira_table.get(iraId), tenantURL + "/http/" + iraId + "/*");
				defaultDesignPackage.addArtifact(a);
			}
		}
	} catch (com.sap.cloud.sdk.datamodel.odata.client.exception.ODataConnectionException e) {
	} catch (ODataException e) {
	} catch (Exception e) {
	}
}

Having all packages and artifacts in place, we can get a (filtered) set of log-entries:

/**
* Returns an ArrayList of LogEntries. 
* The list is based on the filtering.
* <p>
* This methods processing-time depends on the amount of data requested from BTP.
* The status should be set on the GUI for user-information.
* In future releases this method will work in a background-task.
*
*  authentication object relevant for login and serving a destination based on this creds
*  logListFilter the object for filtering the requested logs, i.e. datetime and number of entries
* @return ArrayList of LogEntries. 
*/
public ArrayList<LogEntry> updateMessageProcessingLog(Authentication authentication, LogListFilter logListFilter)	{
	
	ArrayList<LogEntry> logEntries = new ArrayList<LogEntry>();
	
	if (authentication != null)	{
		MessageProcessingLogFluentHelper mPLFH = mPLS.getAllMessageProcessingLogs();
		
		//logs should be filtered always by artifact-id (which is unique)
		if (logListFilter.getArtifact() != null)
			mPLFH = mPLFH.filter(MessageProcessingLog.INTEGRATION_FLOW_NAME.eq(logListFilter.getArtifact().getId()));
		
		//number of results limited by number?
		if (logListFilter.getCountFilterEnabled())	{
			mPLFH = mPLFH
			.skip(logListFilter.getCountFilterFrom() - 1)
			.top(logListFilter.getCountFilterTo());
		}
		
		//filtered by date?
		if (logListFilter.getDateFilterEnabled())	{
			//set start and end, logListFilter returns a @LocalDateTime 
			mPLFH = mPLFH.filter(MessageProcessingLog.LOG_START.gt(logListFilter.getDateTimeFilterStart()));
			mPLFH = mPLFH.filter(MessageProcessingLog.LOG_END.lt(logListFilter.getDateTimeFilterEnd()));
		}
		
		//the result is always ordered ascending by the LogEnd
		mPLFH = mPLFH.orderBy(MessageProcessingLog.LOG_END, Order.ASC);
		final List<MessageProcessingLog> mpl = mPLFH.executeRequest(authentication.getDestination());
		
		for (MessageProcessingLog mpl_iter : mpl)	{
			logEntries.add(new LogEntry(mpl_iter));
			
			try	{
				List<MessageStoreEntry> messageStoreEntries = mpl_iter.fetchMessageStoreEntries();
				if (messageStoreEntries != null)
					for (MessageStoreEntry mse : messageStoreEntries)	{
						List<MessageStoreEntryAttachment> mSEAs = mse.fetchAttachments();
						//in future releases <do_something> with the MessageStoreEntries and their attachments
					}
			}
			catch(com.sap.cloud.sdk.datamodel.odata.client.exception.ODataResponseException e)	{
				System.out.println("com.sap.cloud.sdk.datamodel.odata.client.exception.ODataResponseException");
			}
		}
	}
	
	return logEntries;
}

The result, shown as a Java-GUI is as following:

The java-object MessageProcessingLog and corresponding ..Attachment is based on the classes automatically created within the generator-plugin:

 

With that code we can retrieve the payload on any message starting from object MessageProcessingLog retrieve a MessageProcessingLogAttachment with the following code:

import com.sap.cloud.sdk.datamodel.namespaces.messageprocessinglogs.MessageProcessingLog;
import com.sap.cloud.sdk.datamodel.namespaces.messageprocessinglogs.MessageProcessingLogAttachment;

(...)

//the messageprocessinglog returns a java-List of attachments
List<MessageProcessingLogAttachment> mplAttachments = null;
try {
	mplAttachments = messageProcessingLog.fetchAttachments();
	
	//create a Hashtable which we use to store a local object version of attachment, keys derived from the attachment-name
	attachments = new Hashtable<String, LogAttachment>();
	for (MessageProcessingLogAttachment mpla_iter : mplAttachments) {
		LogAttachment logAttachment = new LogAttachment(mpla_iter);
		//use the name as key and put the attachment in the hashtable
		attachments.put(mpla_iter.getName(), logAttachment);
		
		//this logEntry has an attachment
		hasAttachment = true;
	}
	isPrepared = true;

} catch (Exception e) {
	// TODO Auto-generated catch block
	e.printStackTrace();
}

Different to the previous approaches, the attachment itself (payload) is not provided as a java-object, but by an InputStream. It will be stored in a String and returned by a getter:

//the constructor gets the 'mpla' when instantiated
public LogAttachment(MessageProcessingLogAttachment mpla)	{
   this.mpla = mpla;
}

//prepare the payload seperate from the getter (for caching purpose)	
public void preparePayload()	{

    InputStream is;
    try {
	is = mpla.fetchMediaStream();
		
	if ( is instanceof ByteArrayInputStream)	{
	  int n;
	  try {
	    n = is.available();
	    byte[] bytes = new byte[n];
	    is.read(bytes, 0, n);
	    payload = new String(bytes, StandardCharsets.UTF_8);
	  } catch (IOException e) {
	    e.printStackTrace();
	  }
        }
    } catch (ODataException e) {
      // TODO Auto-generated catch block
      e.printStackTrace();
    }
}
//getter for returning the payload as a String 
//Todo for a future release: what happens if a payload is 'big'? Let's say > 10mb. 
//We don't want to store that in memory in a String!?
public String getPayload()	{
	
    if (payload == null)
      preparePayload();
	
    return payload;
}

Based on the code shown in this blog any app can download, show and also re-send the payload to it's origin:

Stay tuned for further blogs describing details. Don't hesitate to request details in which you are interested.

Best regards Jörg Hopmann