Using Job Scheduler in SAP Cloud Platform [5]: Long-Running (Async) Jobs

This tutorial is part of a little series about SAP Cloud Platform Job Scheduler

Quicklinks:
Intro Blog
Tutorial 1
Sample Code (Trial)
Sample Code (Prod)
Quick Guide

In previous tutorials, we’ve learned how to configure Jobscheduler and how to write apps which are protected with OAuth 2.0

Basically, what we’ve done was to create a web app with a REST endpoint and ask Jobscheduler to call that endpoint.
That endpoint hopefully returns a success response and Jobscheduler is happy to mark the jobrun with green letters

With other words:
Jobscheduler WAITS for the response.
It is like a patient cab driver…. but his patience is not endless:
To be concrete: patience ends after 15 seconds
Yes:
Jobscheduler has a timeout: 15 seconds

But what can we do, if we have operations that require more time?

In this blog, let’s discuss an important scenario: Long running jobs
They are required in scenarios like e.g. data replication running overnight, etc
In such case, Jobscheduler does NOT wait for the response of the called endpoint.
Such scenario has to be executed in ASYNCHRONOUS way
This means:
Jobscheduler triggers an app’s endpoint
Once the app is done, it calls sort of callback to inform Jobscheduler about the finished job

Let’s learn how it is done

Prerequisites

* You should have had a look at the previous tutorials
* You should be familiar with Node.js (otherwise check here)
* You need an account in SAP Cloud Platform
As usual, our tutorial is based on the Trial account (Jobscheduler ‘lite’), such that everybody can follw it.
In productive account (Jobscheduler ‘standard’), the code looks a bit different, but this is explicitly described in a section below

Overview:

1. See the timeout
2. Implement the required app behavior
2.1. Endpoint
2.2. Backend operation
2.3. Update status
2.4. Update status prod
3. Test it

1. Normal synch job

Let‘s start with the negative scenario, then we repair it in a second step
We create a little node application which represents a long-running operation.
To simulate it, we just wait for 18 seconds

1.1. Create Jobscheduler instance

See description here

1.2. Create XSUAA instance

See description here

1.3. Create app

const express = require('express');
const app = express();

app.get('/runjob', function(req, res){
   setTimeout(() => {
      res.send('Finished job after 18 seconds waiting');
   }, 18000);
});

app.listen(process.env.PORT, function(){})

1.4. Deploy

You know how to deply your app

1.5. Try it in browser

Invoke the endpoint in a browser window,e.g.
https://longrun.cfapps.eu10.hana.ondemand.com/runjob
after 18 seconds, you get the response with the message as coded above

1.6. Try JobScheduler

Create job, check logs. There you’ll find the error message:

Error: ESOCKETTIMEDOUT – Request timeout after 15 seconds

OK, this was expected because in our app, we wait 18 seconds, which is longer than the allowed 15 seconds
Yes, our wait-18-seconds is our long-running operation
Let’s see the next chapter to learn how to repair it

Summary:
Jobscheduler has a built-in timeout of 15 seconds

 

2. The solution: Async Job

What needs to be done?

1. Our endpoint has to respond immediately with success status 202
2. We have to trigger our long-running operation
3. We have to manually set the result in Jobscheduler

OK, let’s see in detail

2.1. The endpoint

The Jobscheduler doesn’t know much about us:
It just knows the endpoint which it is supposed to call
So we have to inform Jobscheduler about our plans of running long:
for that purpose we use our endpoint
Concrete: Our endpoint has to send a response with status code 202
Like that, we tell Jobscheduler not to wait for us

Jobscheduler interprets status codes as follows:
* Error code: endpoint operation had an error -> job is completed and will be marked with error
* 200 : endpoint operation finished successfully -> job is completed with success
* 202: endpoint has been triggered and has responded with 202 -> job is not completed, it is on hold until further notice

As per definition, status code 202 means “Accepted”
See appendix for explanation of status code 202

For our endpoint implementation this means:
as soon as our endpoint is invoked, we respond with 202

app.get('/runjob', function(req, res){       
   res.status(202).send('Accepted async job, but long-running operation still running.')

As you’ve already noticed, I’m distinguishing:
– The term “job” belongs to the Jobscheduler, it is the trigger in the cloud
– The term “operation” indicates the actual work, which is done e.g. in a backend

2.2. The backend-operation

After responding to the Jobscheduler, our endpoint will trigger the long-running backend-operation
In our little sample app, we just wait for 3 seconds:

const doLongRunningOperation = function (doFail) {
   return new Promise((resolve, reject)=>{
      const letItFail = doFail
      setTimeout(() => {     
         if(letItFail === 'true'){
            reject({message: 'Backend operation failed with error code 123'});
         }else{
            resolve({message: 'Successfully finished long-running backend operation'})   
         }
      }, 3000); // wait... wait...
   })
}

Note:
To make testing easier, our little sample app can be configured:
Depending on a URL parameter (which is set by us), the long-running backend operation will fail or will succeed

2.3. The Status

Now comes the interesting part:
How to inform the Jobscheduler about the result of our backend operation?
We cannot expect that Jobscheduler calls us again to ask us
So now it is vice versa: WE have to call Jobscheduler

How do we call Jobscheduler?
Jobscheduler provides a REST API for remote control
This is a big feature and here we cover a small subset:
How to use the REST API to set the status of a job execution
See the REST API documentation for an overview of other features (create jobs, view results, etc)

In order to access a particular job execution and set the status, we need some information:
1. We need the internal info about the job and the URL to call
2. We need authentication info
3. We need to know the structure of the endpoint and the data which we have to send

And this is how we get the required info:

2.3.1. How to get the internal info of concrete job to update?

This information is sent to us by Jobscheduler when it calls our endpoint.
It is hidden in the headers
And here are the headers which we need:

Example values:

2.3.2. How to authenticate?

There are credentials for us, we only need to find them:
As usual in the cloud platform, our app gets credentials in the application environment, when it is bound to a service instance.
We can see it either in the cloud cockpit, or in the command line (as described in previous blog)
Now we need to distinguish:
Authentication mechanism is different in Trial and productive landscape, or with other words, it is different depending on the service plan used to create the instance of Jobscheduler service

Authentication in Trial

How does the environment look like?

{
  "VCAP_SERVICES": {
    "jobscheduler": [
    {
      "credentials": {
        "user": "sbss_gkokz3ayc90hyyyvqat3chwzvkhzkwrqrvby2zw6pjapxxmecgtrcifezrfzjm294xc=",
        "password": "aa_rr/YIrbkHGVneWSchRfbnFSJjc8=",

To access the values, we can traverse through the JSON

const VCAP_SERVICES = JSON.parse(process.env.VCAP_SERVICES)
const CREDENTIALS = VCAP_SERVICES.jobscheduler[0].credentials
const USER = CREDENTIALS.user
const PWD = CREDENTIALS.password

Note:
You can use a little node module to access the environment variables: @sap/xsenv (see here)

2.3.3. How to call the udpateStatus endpoint?

Now that we have all the required info, we can go ahead with composing the request to the REST API (see the documentation)

How to build the URL?
This is the template

{host}/scheduler/jobs/{jobId}/schedules/{scheduleId}/runs/{runId}

Which HTTP Verb?
For update the status, we use PUT

Which payload?
As stated in the documentation, the payload has to be a (stringified) json object with mandatory parameter “success” and optional “message”

{
  "success": true,
  "message": "Successful finished long running operation"
}

Which code?

Now, to programmatically execute that request to the REST endpoint of Jobscheduler, I’ve decided to use the native node module, but you can use any other convenient module of your choice

The code shows what we learned above:
1. how we access the information from headers
2. how we compose the authorization from the env variables
3. how we compose the URL with the information extracted from the headers

And at the end we execute the request

const doUpdateStatus = function(headers, success, message){
      jobId = headers['x-sap-job-id']
      scheduleId = headers['x-sap-job-schedule-id']
      runId = headers['x-sap-job-run-id']
      host = headers['x-sap-scheduler-host']
   
      const data = JSON.stringify({success: success, message: message}) 
      const options = {
         host:  host.replace('https://', ''),
         path:  `/scheduler/jobs/${jobId}/schedules/${scheduleId}/runs/${runId}`,
         method: 'PUT',
         headers: {
            Authorization: "Basic " + Buffer.from(USER + ':' + PWD).toString("base64")
         }
      };
      
      const req = https.request(options, (res) => {
      ...
      });     
      ...   
      req.write(data)
      req.end()   
   })
}

Note:
Above code works only in SAP Cloud Platform Trial account, where the jobscheduler service instance can only be created with service plan lite

Please refer to the appendix section for the full application code

2.4. Update status in prod, service plan ‘standard’

In productive landscape, where the jobscheduler service instance is created with service plan ‘standard’, the REST API is protected with OAuth 2.0
(remember: above code sample was based on Trial landscape, where only basic authentication is required)

As such, updating the status takes some more lines of code
In order to use the REST endpoint, we have to send a valid JWT token
So we need to obtain a token before we execute the REST request

2.4.1. Fetch JWT token

We get the token from XSUAA
The credentials required for that call are again provided by jobscheduler in the environment, but this time we don’t get user and password, but instead we get the clientid and clientsecret.
Furthermore we need to know the URL which we have to call in order to get the token.
It is the oauth authorization server URL provided by xsuaa. We get that url in the environment as well
More info about OAuth can be found here
So we can go ahead and create a helper method for fetching the JWT token

const fetchJwtToken = function() {
   return new Promise ((resolve, reject) => {

      // VCAP variables containing the oauth credentials
      const UAA = CREDENTIALS.uaa
      const OA_CLIENTID = UAA.clientid; 
      const OA_SECRET = UAA.clientsecret;
      const OA_ENDPOINT = UAA.url;
      
      const options = {
         host:  OA_ENDPOINT.replace('https://', ''),
         path: '/oauth/token?grant_type=client_credentials&response_type=token',
         headers: {
            Authorization: "Basic " + Buffer.from(OA_CLIENTID + ':' + OA_SECRET).toString("base64")
         }
      }

      https.get(options, res => {
      . . .

The response is a JSON-structured string containing several properties, so we can get the JWT token as follows:

const responseAsJson = JSON.parse(response)
const jwtToken = responseAsJson.access_token            

So here it is, the token
Now can call the REST api to update the status

2.4.2. Update status

Calling the REST endpoint to change the status of a job run is all the same like realized above, just one small difference:
The authentication has to be done with OAuth 2.0 instead of Basic Authentication
We have to send the JWT token in the “Authorization” header as follows:

headers: {
   Authorization: 'Bearer ' + jwtToken

See Appendix for the full code

3. Test

Deploy the application to your account in SAP Cloud Platform
Create a job and enter the action URL similar like this:

https://longrun.cfapps….hana.ondemand.com/runjob

Let it run and quickly navigate to view the run log.
You can see that the endpoint has responded and that Jobscheduler has set a yellow status to indicate that the endpoint has responded but the real status of the long-running operation is not clear yet
Note:
If you don’t see the yellow status, you have to be more quick

After our application has invoked the updateStatus-endpoint of jobschedulers REST api, you can see that the yellow status in the run log has changed accordingly.

To test a negative result, create a job and enter the following action URL;
https://longrun.cfapps…..hana.ondemand.com/runjob?doFail=true
It will show the expected error in the log (like coded by us):

Note:
But what if the long-running operation NEVER ends?
What if the endpoint fails while trying to set the status?
Jobscheduler needs to be aware of that, there must be a fallback.
And yes, in such cases, Jobscheduler has (another) built-in timeout for async jobs

Summary

Any job which runs more than 15 seconds is considered long-running
The endpoint of your application has to respond immediately with status 202
Once the long-running operation has finished, your app has to update the status of the jobscheduler
To update the status, you have to do REST request with PUT and required info
The required info can be found in headers and app-environment
Authentication for REST api: basic auth in lite, oauth in prod

The configuration of a job in Jobscheduler is same as usual

Appendix 1: HTTP Status Code 202

For your convenience, I’ve copied the definition from here:
https://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html

10.2.3 202 Accepted

The request has been accepted for processing, but the processing has not been completed. The request might or might not eventually be acted upon, as it might be disallowed when processing actually takes place. There is no facility for re-sending a status code from an asynchronous operation such as this.

The 202 response is intentionally non-committal. Its purpose is to allow a server to accept a request for some other process (perhaps a batch-oriented process that is only run once per day) without requiring that the user agent’s connection to the server persist until the process is completed. The entity returned with this response SHOULD include an indication of the request’s current status and either a pointer to a status monitor or some estimate of when the user can expect the request to be fulfilled.

Appendix 2: Full sample code for Trial

This project can be deployed to Trial account, it works with Jobscheduler instance created with service plan ‘lite’

manifest.yml

---
applications:
- name: longrun
  host: longrun
  memory: 128M
  buildpacks:
    - nodejs_buildpack
  services:
    - xsuaainstance
    - jobschedulerinstance

package.json

{
  "name": "myapp",
  "main": "server.js",
  "dependencies": {
    "express": "^4.16.3"
  }
}

server.js

const express = require('express')
const app = express()
const https = require('https');


// access credentials from environment variable
const VCAP_SERVICES = JSON.parse(process.env.VCAP_SERVICES);
const CREDENTIALS = VCAP_SERVICES.jobscheduler[0].credentials;
const USER = CREDENTIALS.user; 
const PWD = CREDENTIALS.password;

// our endpoint which is called by Jobscheduler 
app.get('/runjob', function(req, res){       
   // always return success status for async job
   res.status(202).send('Accepted async job, but long-running operation still running.');

   // afterwards the actual processing
   handleAsyncJob(req.headers, req.query.doFail)
});

// our server
app.listen(process.env.PORT || 3000, ()=>{})

// helper
const handleAsyncJob = function (headers, doFail) {
   doLongRunningOperation(doFail)
   .then((result) => {
      doUpdateStatus(headers, true, result.message)
   })
   .catch((error) => {
      doUpdateStatus(headers, false, error.message)
      .then(()=>{
         console.log("Successfully called REST api of Jobscheduler")
      }).catch((error)=>{
         console.log('Error occurred while calling REST api of Jobscheduler ' + error)
      })
   })
}

// our backend operation
const doLongRunningOperation = function (doFail) {
   return new Promise((resolve, reject)=>{
      const letItFail = doFail
      setTimeout(() => {     
         if(letItFail === 'true'){
            reject({message: 'Backend operation failed with error code 123'});
         }else{
            resolve({message: 'Successfully finished long-running backend operation'})   
         }
      }, 3000); // for testing purpose, 3 seconds are long-running enough
   })
}

// our helper method to set the status in Jobscheduler
const doUpdateStatus = function(headers, success, message){
   return new Promise((resolve, reject) => {
      jobId = headers['x-sap-job-id']
      scheduleId = headers['x-sap-job-schedule-id']
      runId = headers['x-sap-job-run-id']
      host = headers['x-sap-scheduler-host']
   
      const data = JSON.stringify({success: success, message: message}) 
      const options = {
         host:  host.replace('https://', ''),
         path:  `/scheduler/jobs/${jobId}/schedules/${scheduleId}/runs/${runId}`,
         method: 'PUT',
         headers: {
            'Content-Type': 'application/json',
            'Content-Length': data.length,
            Authorization: "Basic " + Buffer.from(USER + ':' + PWD).toString("base64")
         }
      };
      
      const req = https.request(options, (res) => {
         res.setEncoding('utf8')
         if (res.statusCode !== 200 && res.statusCode !== 201) {
           return reject(new Error(`Failed to update status of job ${jobId}`))
         }
   
         res.on('data', () => {
            resolve()
         })
      });
      
      req.on('error', (error) => {
         return reject({error: error})
      });
   
      req.write(data)
      req.end()   
   })
}

Appendix 3: Sample code for Prod

Use this javascript file for scenarios in productive account, where Jobscheduler instance is created with service plan ‘standard’
The difference to previous app is only in the code:
Calling the REST api with OAuth flow

server.js

const express = require('express')
const app = express()
const https = require('https');

// access credentials from environment variable (alternatively use xsenv)
const VCAP_SERVICES = JSON.parse(process.env.VCAP_SERVICES)
const CREDENTIALS = VCAP_SERVICES.jobscheduler[0].credentials
//oauth
const UAA = CREDENTIALS.uaa
const OA_CLIENTID = UAA.clientid; 
const OA_SECRET = UAA.clientsecret;
const OA_ENDPOINT = UAA.url;

// our endpoint which is called by Jobscheduler 
app.get('/runjob', function(req, res){       
   // always return success status for async job
   res.status(202).send('Accepted async job, but long-running operation still running.');

   // afterwards the actual processing
   handleAsyncJob(req.headers, req.query.doFail)
});

// our server
app.listen(process.env.PORT || 3000, ()=>{})

// helper
const handleAsyncJob = function (headers, doFail) { 
   doLongRunningOperation(doFail)
   .then((result) => {
      doUpdateStatus(headers, true, result.message)
   })
   .catch((error) => {
      doUpdateStatus(headers, false, error.message)
      .then(()=>{
         console.log("Successfully called REST api of Jobscheduler")
      }).catch((error)=>{
         console.log('Error occurred while calling REST api of Jobscheduler ' + error)
      })
   })
}

// our backend operation
const doLongRunningOperation = function (doFail) {
   return new Promise((resolve, reject)=>{
      const letItFail = doFail
      setTimeout(() => {     
         if(letItFail === 'true'){
            reject({message: 'Backend operation failed with error code 123'});
         }else{
            resolve({message: 'Successfully finished long-running backend operation'})   
         }
      }, 3000); // for testing purpose, 3 seconds are long-running enough
   })
}

// jwt token required for calling REST api
const fetchJwtToken = function() {
   return new Promise ((resolve, reject) => {
      const options = {
         host:  OA_ENDPOINT.replace('https://', ''),
         path: '/oauth/token?grant_type=client_credentials&response_type=token',
         headers: {
            Authorization: "Basic " + Buffer.from(OA_CLIENTID + ':' + OA_SECRET).toString("base64")
         }
      }

      https.get(options, res => {
         res.setEncoding('utf8')
         let response = ''
         res.on('data', chunk => {
           response += chunk
         })

         res.on('end', () => {
            try {
               const responseAsJson = JSON.parse(response)
               const jwtToken = responseAsJson.access_token            
               if (!jwtToken) {
                  return reject(new Error('Error while fetching JWT token'))
               }
               resolve(jwtToken)
            } catch (error) {
               return reject(new Error('Error while fetching JWT token'))               
            }
         })
      })
      .on("error", (error) => {
         console.log("Error: " + error.message);
         return reject({error: error})
      });
   })   
}

// our helper method to set the status in Jobscheduler
const doUpdateStatus = function(headers, success, message){
   return new Promise((resolve, reject) => {
      return fetchJwtToken()
         .then((jwtToken) => {

            const jobId = headers['x-sap-job-id']
            const scheduleId = headers['x-sap-job-schedule-id']
            const runId = headers['x-sap-job-run-id']
            const host = headers['x-sap-scheduler-host']
         
            const data = JSON.stringify({success: success, message: message})             
            const options = {
               host:  host.replace('https://', ''),
               path:  `/scheduler/jobs/${jobId}/schedules/${scheduleId}/runs/${runId}`,
               method: 'PUT',
               headers: {
                  'Content-Type': 'application/json',
                  'Content-Length': data.length,
                  Authorization: 'Bearer ' + jwtToken
               }
            }
            
            const req = https.request(options, (res) => {
               res.setEncoding('utf8')
               const status = res.statusCode 
               if (status !== 200 && status !== 201) {
                  return reject(new Error(`Failed to update status of job ${jobId}. Error: ${status} - ${res.statusMessage}`))
               }
         
               res.on('data', () => {
                  resolve()
               })
            });
            
            req.on('error', (error) => {
               return reject({error: error})
            });
         
            req.write(data)
            req.end()   
      })
      .catch((error) => {
         console.log('ERROR: failed to fetch JWT token: ' + error)
         reject(error)
      })
   })
}