Resumable Upload of Large Files to Microsoft OneDrive via SAP PO

[PriyankaAnagani]

The scope for improvement is infinite. Whatever requirement we fulfil, there will be a scope for improvement!

Earlier, I’ve posted an article about uploading files to OneDrive through SAP PO
https://blogs.sap.com/2020/01/22/sap-ecc-proxy-onedrive-integration-using-microsoft-graph-api-in-sap...
I’ve recently got an opportunity to improve what I had done before, and I wanted to share those details with the SAP community.

Why is this improvement?

                  In my previous article, I’ve demonstrated uploading of files using one of the Graph APIs. It was a simple item upload of Graph API which is to place the files less than 4Mb of content. If the file size exceeds more than 4Mb, the API returns HTTP 413 error i.e., the request size exceeds the maximum limit. I got the requirement to transmit file size ranging from 10 to 20Mb.

What is the alternative to upload large files?

                  There are 4 different APIs that Microsoft Graph provides to upload files. For uploading of larger files, Microsoft recommends using “Resumable Item Upload” which establishes an upload session and uploads ranges of the file in sequential API requests.

How it Actually Works?

                  To upload a large file, first we should establish a new upload session which creates a temporary storage location where the bytes of the files will be stored. The upload session response returns a unique upload session URL. We can then make PUT file request to the upload session URL.

                  The entire file can be uploaded at once or split the file into multiple byte ranges. In case of splitting into fragments, the fragments of the file must be uploaded sequentially, and each fragment size must be multiple of 320Kb. Also, after writing the last byte of the file, it’s possible to receive a conflict which requires explicitly committing the upload session. Here are the steps.

• Step1: Make a POST call to Graph API with the name of the file and get the UploadSessionURL as a response.


• Step2: Make one or more PUT calls to the UploadSessionURL with file content in payload and Content Range & Content Length in HTTP Headers.


• Step3: Make a PUT call to the drive item with actual file name and UploadSessionURL in the payload.

 

 

The Design in SAP PO:

                  As it requires making 2 or more API calls, an orchestration is needed to ensure the session is created, file bytes are uploaded in sequential order and commit the upload session. Does it mean, it requires a BPM? Well, as we know, it’s not a good practice to design a BPM just to upload a file. But, because of the security guidelines in our organization, this data transmission should go via SAP PO. That pushed me to implement the below trick.

• Step1: In the request mapping, perform a REST lookup to create the upload session and store the UploadSessionURL from the lookup response in Dynamic configuration.


• Step2: Write the entire large file to the UploadSessionURL (as the API allows file up to 60MB).


• Step3: In the response mapping, perform another REST lookup to commit the upload session.

Mapping & Configuration in SAP PO:

                  As this article is an enhancement to previous one, the prerequisite steps are not covered here and below are the configurations for uploading large files.

Java Mapping has been used to read the attachment, perform a REST lookup to create upload session and to transform the attachment to main payload. Here is the snapshot of mapping.

public void transform(TransformationInput input, TransformationOutput output) throws StreamTransformationException {

		/**

	 * @Description	:	This Java mapping to create upload session via lookup & then load the attachment file using dynamic upload URL

	 * @Author		: 	Priyanka Anagani

	 * @Date		:	25-Jul-2021

	 **/

		AbstractTrace trace = getTrace();

		trace.addDebugMessage("JavaMapping Started....!");



		String str = "", tmpFileName = "", fileName ="", uploadSessionReq="", uploadURL="", fileSize="", attId = "",attName ="", contentType = "", contentRange="", driveItem="";

		int startIndex=0, endIndex=0;

		Attachment attachment = null;

		long range = 0;

		Object[] arrayObj = null;

		byte[] attBytes = null;

    	SystemAccessor accessor = null;



		InputAttachments inputAttachments = input.getInputAttachments();

		InputStream inputstream = input.getInputPayload().getInputStream();

		OutputStream outputstream = output.getOutputPayload().getOutputStream();



		try{

				//Get the name fo the file from source payload

				DocumentBuilder dBuilder = DocumentBuilderFactory.newInstance().newDocumentBuilder();

				org.w3c.dom.Document document = dBuilder.parse(inputstream);//source doc

				document.getDocumentElement().normalize();

				NodeList nList =  document.getElementsByTagName("FileName");

				fileName = nList.item(0).getTextContent();



				NodeList nList2 =  document.getElementsByTagName("FileType");

				fileName = fileName+ "." +nList2.item(0).getTextContent();

				

				NodeList nList3 =  document.getElementsByTagName("FileSize");

				fileSize = fileSize+nList3.item(0).getTextContent();

				fileSize = fileSize.trim();



				//set the file name in dynamic config

				Map map = (Map) input.getInputHeader().getAll();

				DynamicConfiguration conf = (DynamicConfiguration) input.getDynamicConfiguration();

				DynamicConfigurationKey key1 = DynamicConfigurationKey.create("http:/"+"/sap.com/xi/XI/System/REST","fileName");

				conf.put(key1, fileName);



				DynamicConfigurationKey key2 = DynamicConfigurationKey.create("http:/"+"/sap.com/xi/XI/System/REST","filesize");

				conf.put(key2, fileSize);



				uploadSessionReq = "<?xml version=\"1.0\" encoding=\"UTF-8\"?><root microsoft.graph.conflictBehavior=\"replace\"><description>FileUpload</description><name>" + fileName + "</name><fileSize>"+fileSize+"</fileSize></root>";



				Channel channel = LookupService.getChannel("BC_OneDrive", "REST_REC_CreateUpoadSession"); 	//Determine Channel

				accessor = LookupService.getSystemAccessor(channel); //Get SystemAccessor object	

				InputStream iStream = new ByteArrayInputStream(uploadSessionReq.getBytes());	

				XmlPayload payload = LookupService.getXmlPayload(iStream);

		   

				Payload restOutput = null;

				restOutput = accessor.call(payload);//call the RESTservice through lookup



				//Parse the result	and get Upload URL		  

				InputStream in = restOutput.getContent();

				ByteArrayOutputStream resp = new ByteArrayOutputStream();

				byte[] buffer = new byte[1024];

				int bufLength;

				while ((bufLength = in.read(buffer)) != -1) {

					resp.write(buffer, 0, bufLength);

				}

				trace.addDebugMessage("Received Response:"+resp.toString());

				JSONObject json = new JSONObject(resp.toString());

				uploadURL = (String) json.get("uploadUrl");

				

				startIndex =uploadURL.indexOf("items/")+6;

				endIndex = uploadURL.indexOf('/', startIndex+5);



				driveItem = uploadURL.substring(startIndex,endIndex);

				trace.addDebugMessage("Drive Item is :"+driveItem);

				//set upload URL in the dynamic config

				DynamicConfigurationKey key3 = DynamicConfigurationKey.create("http:/"+"/sap.com/xi/XI/System/REST","uploadUrl");

				conf.put(key3, uploadURL);



				DynamicConfigurationKey key4 = DynamicConfigurationKey.create("http:/"+"/sap.com/xi/XI/System/REST","driveItem");

				conf.put(key4, driveItem);

				



				//convert the attachment to mainpayload

				if(inputAttachments.areAttachmentsAvailable()){

						trace.addInfo("Attachments Found");

						Collection<String> collectionIDs = inputAttachments.getAllContentIds(true);

						arrayObj = collectionIDs.toArray();

 						for(int i =0;i<arrayObj.length;i++){

							attId = (String)arrayObj[i];

							attachment = inputAttachments.getAttachment(attId);

							contentType = attachment.getContentType();

							attBytes = attachment.getContent();

						}//end of for

				}//end of if

				outputstream.write(attBytes);//write mainayload file

			

				range =Long.parseLong(fileSize);

				range=range-1;

				trace.addInfo("Range is:"+Long.toString(range));

				contentRange = "0-"+range;

				DynamicConfigurationKey key5 = DynamicConfigurationKey.create("http:/"+"/sap.com/xi/XI/System/REST","range");

			  conf.put(key5, contentRange);



		}//end of try

		catch(Exception ex){

			trace.addWarning("Exception Raised...! "+ex);

		}

}//end of transform

 

Step1- Using the REST lookup channel, a HTTP POST call is performed to get the UploadSessionURL as a response. The createUploadSession URL is like below.The lookup channel configuration is shown below.

 

The request & response payloads of createUploadSession call looks like below.

 

Step2 - A HTTP PUT call is performed to the uploadSesionURL received in step1. Below is the channel config.

 

After uploading the last byte, it's possible that Graph API returns a conflict wile committing and renaming the file. Hence, the same has been handled in channel Error Handling.

 

Whenever Microsoft API return a conflict, the file will not be named as it was requested in the createUploadSession. We can only see a temporary file created in the folder as shown below.

 

Step3 - In the response mapping, create a UDF for REST lookup & make a HTTP PUT call to the OneDrive item with uploadSessionURL to complete file writing.

 

Here is the sample code.

public String commitRename(String var1, Container container) throws StreamTransformationException{

	String str = "", fileName="", uploadURL ="", req ="", driveItem="";

	AbstractTrace trace = container.getTrace();

	SystemAccessor accessor = null;



	try{

		DynamicConfiguration conf = (DynamicConfiguration) container.getTransformationParameters().get(StreamTransformationConstants.DYNAMIC_CONFIGURATION);

		DynamicConfigurationKey key1 = DynamicConfigurationKey.create("http:/"+"/sap.com/xi/XI/System/REST","fileName");

		fileName = conf.get(key1);



		DynamicConfigurationKey key2 = DynamicConfigurationKey.create("http:/"+"/sap.com/xi/XI/System/REST","uploadUrl");

		uploadURL = conf.get(key2);



		DynamicConfigurationKey key3 = DynamicConfigurationKey.create("http:/"+"/sap.com/xi/XI/System/REST","driveItem");

		driveItem = conf.get(key3);



		uploadURL = uploadURL.replace("&", "&");

		req = "<?xml version=\"1.0\" encoding=\"UTF-8\"?><root microsoft.graph.sourceUrl=\"" + uploadURL+ "\"><name>"+fileName+"</name><driveItem>"+driveItem+"</driveItem></root>";





		Channel channel = LookupService.getChannel("BC_OneDrive", "REST_REC_CoomitRenameFile"); 	//Determine Channel

		accessor = LookupService.getSystemAccessor(channel); //Get SystemAccessor object	

		InputStream iStream = new ByteArrayInputStream(req.getBytes());	

		XmlPayload payload = LookupService.getXmlPayload(iStream);

			   

		Payload restOutput = null;

		restOutput = accessor.call(payload);//call the RESTservice through lookup



		//Parse the result	and get response		  

		InputStream in = restOutput.getContent();

		ByteArrayOutputStream resp = new ByteArrayOutputStream();

		byte[] buffer = new byte[1024];

		int bufLength = 0;

		while ((bufLength = in.read(buffer)) != -1){

			resp.write(buffer, 0, bufLength);

		}

		//trace.addDebugMessage("Received Response:"+resp.toString());



	}//end of try

	catch(Exception ex)

	{

		trace.addWarning("Exception Raised...! "+ex);

	}

	return str;

}

Here is the sample payload for commit rename request.

Rename Lookup Channel configuration is as below.

Successful message processed.

With this trick, large files can be transmitted easily to OneDrive. I Hope this will be helpful.

Reference: https://docs.microsoft.com/en-us/graph/api/driveitem-createuploadsession?view=graph-rest-1.0

Please share your feedback in the comments section, Happy Learning!!