Skip to Content
Technical Articles

Using Job Scheduler in SAP Cloud Platform [6]: Troublemaking

This blog is part of a little series about SAP Cloud Platform Job Scheduler

Have you configured a job to call your oauth-protected endpoint?
And does it fail?
And do you get an error like 401 or 403?
And you hate it?

If yes, you should go should give this blog a chance and hopefully it will solve your problem…!
I’ve listed possible reasons for failure and possible ways how to find the error and solve the problem.
I’m not explaining much, you can find detailed descriptions in the blogs of this series

And please:
If you have another tip how you solved a problem, please add it in the comment section

1. Jobscheduler instance creation

When creating the instance of jobscheduler service, you have to pass the following parameter:

{
   "enable-xsuaa-support": true 
}

See here for details

2. Jobscheduler instance creation

When creating the instance of jobscheduler service, you have to choose service plan standard
Otherwise, OAuth flow is not fully supported for your endpoint, and authorization will always fail

3. XSUAA instance creation

When creating the instance of “Authorization and Trust Management” (XSUAA), you have to choose service plan as application or broker
See here for details

4. XSUAA scope contains ‘grant…’

Background:
You create an application and you define “scopes” to control the access to it.
Means, a user who calls your app needs to have that “scope”
Thus Jobscheduler must have it as well
But Jobscheduler is not a user, so the scope must be assigned via “GRANT” in the xs-security.json file

"scopes": [{
  "name": "$XSAPPNAME.scopeformyapp",
  "grant-as-authority-to-apps": ["$XSSERVICENAME(my_jobscheduler_instance_name)"]
}]

This is essential!
When Jobscheduler calls your app, it sends a JWT token, and this token will contain the required scope –  but only if it has been granted
See here for details

5. xs.security.json

In case of changes to xs-security.json:
The instance of XSUAA needs to be updated, re-deploy of app is not sufficient.
If you’ve modified the xs-security.json file, then you need to exeucte the following command to make the changes available in the instance of xsuaa:

cf update-service yourXsuaa -c xs-security.json

Note that updating a service instance can only be done on command line, it is not supported in the cockpit.
Note:
After changing the xsuaa instance, you need to bind it again to your app.
See section 8

6. SAP_JWT_TRUST_ACL

Background:
Jobscheduler cannot read the credentials of your app (I mean the OAuth credentials which are provided to your app, when you bind it to xsuaa
So it cannot send the expected clientid
Instead, jobscheduler sends its own clientid
But, that clientid is wrong, so it is rejected, unless your app declares an exlicit TRUST to it
To specify that trust, your app has to provide the following environment variable, which specifies the trusted clientid.
You can specify that clientid (you can find it in the credentials section of the jobscheduler, bound to your app), or just an asterisk (obviously, not recommended in productive scenarios)
The mentioned environment variable can be set in the manifest.yml file:

  env:
    SAP_JWT_TRUST_ACL: >
      [
        {"clientid":"*", "identityzone":"*"}
      ]

Environment variables can also be set via command line or in the cloud cockpit
Note: make sure not to forget all the brackets.
Note: there must be the square brackets as well
See here for details

7. Check the TRUST

After deployment of your app, it might make sense to check if the TRUST property has properly reached the cloud
It can be checked in the cloud cockpit, in the details screen of your app, in the section user-provided variables
Or you can use command line
cf env yourAppName
Result in user-provided section:

SAP_JWT_TRUST_ACL: [
{“clientid”:”*”, “identityzone”:”*”}
]

Note: make sure that the quotation marks are there

8. Laundry program for binding

In tutorial no 3, we‘ve learned that the order of creation and binding of service instances is crucial, because otherwise the jobscheduler doesn’t get the required scope assignment. And as a consequence, calling your endpoint will fail

Remember:
Create Jobs first, but
Bind xsusa first again

Instead of going through the process of creation and binding again, you can just execute I(what I call) the laundry program.
I mean, just unbind the servce instances, then bind again in the required order

These are the commands to be executed on command line
Make sure to adapt your app name and  your service instance name

First unbind both instances:

cf unbind-service yourApp yourXsuaa

cf unbind-service yourApp yourJobscheduler

cf restage yourApp

Then bind xsuaa:

cf bind-service yourApp yourXsuaa

cf restage yourApp

Finally bind jobscheduler:

cf bind-service yourApp yourJobscheduler

cf restage yourApp

(I confess that the amount of commands it a bit paranoid…)

9. Analyze the error

In the Jobscheduler run log, check the error message:
The error status is 401 or 403?
In case of 403, the authentication is fine, but authorization is missing. This would be an indicator that the jobscheduler doesn’t have the required scope

10. Call your endpoint manually

Make sure that your endpoint behaves as expected.
I mean, it is possible that the problem is not with Jobscheduler, but with the endpoint
To verify it, invoke the endpoint URL with a REST client (e.g. Postman)
You need to follow the OAuth flow, it is described here or more detailed here

 

11. View the content of the token

When calling the endpoint manually, like described in previous section, you fetch the JWT token manually.
Once you have it, you can decode it.
Then check out the scopes section: how does the required scope look like?
Remember it and compare with the token and the scope which is sent by Jobscheduler (see next section)
Find the required description here

 

12. View the content of the token sent by Jobscheduler

This is the essential check:
Does the JWT token which is sent by Jobscheduler contain the required scope?
This might be not easy to identify.
To verify it, hook into your app, before your endpoint fails.
You need to access the “request” object and get the “authorization” header.
The value of this header contains the token, you can decode it
Check if the required scope is there
See here how it is done in Node.js

13. Other attempt to view the content of the token

If previous way is not possible:
Add another endpoint to your app.
No frameworks.
No protection, no security checks.
The endpoint should just read the jwt token

In addition, in the code, you can also access the TRUST environment variable, and check if it contains valid JSON.
I mean, e.g. if the quotation marks were lost, then parsing the content would fail, because not valid JSON

14. One more attempt to view the content of the token

If previous way is not possible:
Increase the log level of the involved components, like xssec, or app router

xssec:

To increase the log level, add this variable to your environment:

DEBUG=xssec:*

App router:

In case you have an application router involved, you can also configure it to write millions of log entries.
E.g. run this command

cf set-env yourAppName XS_APP_LOG_LEVEL DEBUG

In addition, more environment variables to add:

REQUEST_TRACE = TRUE

DEBUG=*

15. Really the last attempt to view the content of the token

If there’s no other option, you can write a separate app, which does nothing than read the JWT token.
Similar like above attempt, but this time a separate app
Here’s why it is the last attempt: a separate app is not the same app, so it might not be really helpful.
Nevertheless, you bind it to the same instance of xsuaa and jobscheduler (remember the order) like your productive app

1. Hopefully you’re little familiar with Node.js, so you can create a meaningful app with below files
The code of the app can be found in the appendix
2. Deploy
3. Create a Job
4. Then check the log (jobscheduler run log and also Cloud Platform log).
There you can find the information about the JWT token which was sent by Jobscheduler.

If you find a bug or have added enhancements: please let us know

Appendix: Troublemaker app files

Please don’t expect too much from this app:
It only prints the content of the JWT token which it receives
And it writes the value of the TRUST param to the log
That’s already all.
But it might be helpful….
Has already be useful for Werner (thanks, Werner…)
Once you see the scope contained in the token, you might be surprised, maybe the scope-check was not properly implemented, who knows…
The tiny app is written in node.js, so you should get familiar with node if you aren’t yet 😉

You have to create a folder in your file system
Then create the 3 files inside
Adapt the service names in the manifest file
Enhance the implementation as desired
Make sure to write a comment to share your enhancements with the community
To deploy, navigate into the project directory and run cf push.

Finally, here are the required files:

manifest.yml

To deploy the app to the SAP Cloud Platform, we need a manifest file
Make sure to adapt the names of the service instances.
Ideally, just use the manifest of your productive app

---
applications:
- name: troublemaker
  host: troublemaker
  memory: 64M
  buildpacks:
    - nodejs_buildpack
  services:
    - your_xsuaa_instance
    - your_jobscheduler_instance
  env:
    SAP_JWT_TRUST_ACL: >
      [
        {"clientid":"*", "identityzone":"*"}
      ]

package.json

{
  "main": "server.js",
  "dependencies": {
    "express": "^4.16.3"
  }
}

server.js

const express = require('express');
const app = express();

/* The endpoint */
app.get('/job', function(req, res){
   const authHeader = req.headers.authorization;
   const acl = process.env.SAP_JWT_TRUST_ACL

   const jwtCheckResult = _checkAuthorization(authHeader)

   const jobscheduler_client = jwtCheckResult.clientid
   const jobscheduler_identityzone = jwtCheckResult.identityzone

   const trustCheckResult = _checkTrust(acl, jobscheduler_client, jobscheduler_identityzone)

   if(jwtCheckResult.isValid && trustCheckResult.isValid){
      res.status(200).send(`Successfully checked JWT token and TRUST. \n Token contains scopes: ${jwtCheckResult.scopes} \n for jobscheduler-client: ${jobscheduler_client}` )
   }else{
      res.status(403).send(`Validation failed. Invalid JWT and/or TRUST. See log for details`)      
   }
});

/* The server */
app.listen(process.env.PORT, function(){})

/* The helpers */
const _checkAuthorization = function(authHeader){
   console.log(`===> [Check JWT] Checking the authorization...`)
   if (! authHeader){
       console.log('===> [Check JWT]: Endpoint was called with no authorization header at all')
       return {isValid: false, message: 'Request does not contain authorization header'}
   }

   if(! authHeader.includes('earer')){
       console.log(`===> [Check JWT]: Endpoint was called with authorization header, but it does not contain bearer token: ${authHeader}`)
       return {isValid: false, message: 'Authorization header of request does not contain bearer token'}
   }

   var theJwtToken = authHeader.substring(7);
   if(! theJwtToken){
       console.log(`===> [Check JWT]: Endpoint was called with bearer-token header, but the token seems to be invalid: ${authHeader}`)
       return {isValid: false, message: 'Authorization header contains invalid JWT token'}
   }
   
   console.log(`===> [Check JWT] the received JWT token: \n${theJwtToken}`)
   let jwtBase64Encoded = theJwtToken.split('.')[1];
   if(! jwtBase64Encoded){
       console.log(`===> [Check JWT]: Endpoint was called with bearer-token header, but the token seems to have invalid structure: Must have 3 segments separated with dot.`)
       return {isValid: false, message: 'Authorization header contains invalid JWT token'}    
   }

   let jwtDecodedAsString = Buffer.from(jwtBase64Encoded, 'base64').toString('ascii');
   console.log(`===> [Check JWT] the decoded JWT token: ${jwtDecodedAsString}`)
   let jwtDecodedJson = undefined
   console.log(`===> [Check JWT] parsing the JWT token...`)
   try {
       jwtDecodedJson = JSON.parse(jwtDecodedAsString);            
   } catch (error) {
       console.log(`Endpoint was called with bearer-token header, but parsing the token caused error: ${error}`)
       return {isValid: false, message: 'Error while parsing JWT token. See log for details.'}                
   }

   // JWT token is valid, however, checking the scopes has to be done manually
   console.log(`===> [Check JWT]: JWT: scopes: ${jwtDecodedJson.scope}`);
   console.log(`===> [Check JWT]: JWT: client_id: ${jwtDecodedJson.client_id}`);
   console.log(`===> [Check JWT]: JWT: identity_zone: ${jwtDecodedJson.zid}`);
   console.log(`===> [Check JWT]: JWT: user: ${jwtDecodedJson.user_name}`);

   return {
      isValid: true, 
      message: `Successfully parsed JWT token. See log for more details. \n Scopes: ${jwtDecodedJson.scope} \n client: ${jwtDecodedJson.client_id} \n user: ${jwtDecodedJson.user_name}`,
      clientid: jwtDecodedJson.client_id,
      identityzone: jwtDecodedJson.zid,
      scopes: jwtDecodedJson.scope
   }                
}

const _checkTrust = function (aclstring, clientSentByJobScheduler, identitySentByJobscheduler) {
   console.log('===> [Check TRUST] Checking the The variable SAP_JWT_TRUST_ACL...')
   
   if(! aclstring){
      console.log('===> [Check TRUST] The variable SAP_JWT_TRUST_ACL was not found in the app environment')
      return {isValid: false, message: 'Missing environment variable SAP_JWT_TRUST_ACL'}                
   } 

   let aclJson = undefined
   try {
       aclJson = JSON.parse(aclstring)
   } catch (error) {
      console.log(`===> [Check TRUST]: Error while parsing env var: SAP_JWT_TRUST_ACL. Value is not valid JSON: ${error}`);
      return {isValid: false, message: 'Error while parsing SAP_JWT_TRUST_ACL. Value is not valid JSON. See log for details'}                
   }
   console.log(`===> [Check TRUST]: value of SAP_JWT_TRUST_ACL is valid JSON`);
   
   // check correct format
   const aclIsArray = Array.isArray(aclJson)
   console.log(`===> [Check TRUST]: value of SAP_JWT_TRUST_ACL is valid Array: ${aclIsArray}`);     
   if (! aclIsArray){
      console.log(`===> [Check TRUST]: Incorrect format of SAP_JWT_TRUST_ACL: Value is NOT array`);
      return {isValid: false, message: 'Incorrect format of SAP_JWT_TRUST_ACL: Value is NOT array. See log for details'}                       
   }

   // check if jobscheduler (foreign) clientid is trusted   
   var foundMatch = false;
   for (var aclEntry in aclJson) {
      let clientidInAcl = aclJson[aclEntry].clientid
      let identityzoneInAcl = aclJson[aclEntry].identityzone
      console.log(`===> [Check TRUST]: value of SAP_JWT_TRUST_ACL[i].clientid: ${clientidInAcl} and identityzone: ${identityzoneInAcl}`);

       if (((clientidInAcl === clientSentByJobScheduler) || (clientidInAcl === '*'))
                && ((identityzoneInAcl === identitySentByJobscheduler) || (identityzoneInAcl === '*'))) {
           foundMatch = true;
           break;
       }
   }
   if (foundMatch) {
      console.log(`===> [Check TRUST]: the jobscheduler clientid and identityzone are contained in TRUST. All fine ;-)`);
      return {isValid: true, message: 'The jobscheduler clientid and identityzone are contained in TRUST'}                       
   } else {
      console.log(`===> [Check TRUST]: Error: the jobscheduler clientid is NOT contained in TRUST. Result: 403 - Forbidden`);
      return {isValid: false, message: 'The jobscheduler clientid and identityzone NOT contained in TRUST_ACL'}                       
   }
}
4 Comments
You must be Logged on to comment or reply to a post.
  • Dear Roggan

    thanks for your sharing, by following your instruction,  I checked my configuration many times, I still get error code 401 in the job execution log, i think the token is not passed from jobscheduler to my backend servcie.

    for my case, I have already backend service(CAP java) and uaa, uaa has been bind to the backend service, I need to call one of my service endpoint with jobscheduler.

    here is my steps:

    1. create scheduler with plan “standard” and parameters:{“enable-xsuaa-support”: true }​
    2. update my existing xs-security.json th additional new scope:
       {
            “name”: “$XSAPPNAME.TrustExt”,
            “description”: “Enable Calls from trusted external source”,
            “grant-as-authority-to-apps”: [
              “$XSSERVICENAME(myschedulername)”
            ]
          },

      then use cf update-service..

    3. update my existing backend service with binding to scheduler instance, specified after binding of uaa, and propertt SAP_JWT_TRUST_ACL: [{“clientid”:”*”, “identityzone”:”*”}]
    4. deploy updated backend service

    after that, I create a job and execute it immediately, it get error 401.

     

    then. I unbind both uaa and scheduler for back service,restage, bind uaa, restatge, bind scheduler, restaget. the job still gets error 401.

     

    do you have any idea what’s wrong with my case?

     

    Regards,

    Eric

     

    • Hi Carlos,

      These blog series are much informative. Thanks.

      In all these blog post the REST endpoints resides in CF. Is it possible to configure external API endpoint? if so how authentication will be done?

      Thanks,

      • Hi Akash S
        Thanks for your feedback, Thanks !
        No, Jobscheduler has been designed to facilitate development in CF
        It is nice, that Jobscheduler can automatically do the OAuth flow when calling an endpoint
        To facilitate that, an application needs to be BOUND to Jobscheduler.
        That can only be done inside CF
        If required, as a workaround, you can deploy and bind a proxy app, which will then call your external endpoint
        Hope this is helpful
        Kind Regards,
        Carlos