SAP Data Intelligence Cloud – Local development of Node.JS Operators

You are working on a Pipeline where you need to build a Node.JS operator to implement some complex business logic or control flow.

What you would be normally doing, while developing the code inside the Node.JS Operator, would be to write a couple of JavaScript lines, then save the Pipeline and finaly running it. The last step, would schedule the graph for execution, and would take some time until all processes are up and running and the pipeline is in running status.

If the code you need to write is somehow complex and you need to write many lines of code, the above described steps would consume a lot of time, and the overall development experience will not be the best.

Thanks to the Node.JS Subengine SDK [here] it is possible to take locally the development process of this operator.

Let me share the steps

For simplicity, I will create a Pipeline that uses a Terminal Operator to send some text to a Node.JS Operator, the text will be changed and sent back to the Terminal Operator.

In the local development, I will read the user input in the command terminal, to emulate the Terminal Operator, and i am going interact ( even debug ) with the Node.JS Operator.

Pipeline

Step 1

Create a new Pipeline and place a Terminal Operator and a Node Base Operator.

Add an Input port to the Node.JS Operator named input of type string. 

Add an Output port to the Node.JS Operator named output of type string.

Keep in mind that inside Node.JS you use Javascript types and data structures, visit the help documentation for the correct mapping between DI types and JavaScript types [here]

If you open the script attached to the Node Operator, you will find somethig like this

At this point, you would normally do the flow described at the beginning of this post.

Step 2

Right click on the Node.JS Operator and select Save As Operator.

Enter a full qualified name (F.Q.N)  and display name

This step will save the Opetator metadata (operator.json) and script (script.js) in the DI repository.

Step 3

Open the Repositoty view and navigate to the path

subengines –> com –> sap –> node –> operators –> <FQN>

Right click on the latest folder of your operator’s fqn and select Export File

Additionally export the vflow-sub-node-sdk.tar.gz file located in subengines –> com –> sap –> node

Step 4

For the local development you will need to install the Node.js runtime.

Copy and extract both files to an empty folder and finally move the vflow-sub-node-sdk.tar.gz into the localNode folder (this is the extacted file from the vflow-sub-node-sdk.tgz exported file in step 3)

The local folder should look like below

The important files here are

1. operator.json, that contains operator metadata, like input and output ports
2. script.js, that contains the javascript code to run
3. vflow-sub-node.sdk.tar.gz that contains the SDK module

Step 5

I will open with VS Code the localNode folder to do the development part.

First, run npm init -y to initialize an empty node.js project

Then, install the sdk module by running npm i -s vflow-sub-node-sdk.tar.gz

Just for the purpose of this blog, I will not use any 3rd party module that requires installation. As you may already know, for installing 3rd party modules on DI you should either build a custom Docker image with the modules [here] or upload to the repository [here]

Step 6

Write the logic for the operator in the script.js file.

In this example the changeText function receives a string and alternates each character Uppercase and lowercase.

const SDK = require("@sap/vflow-sub-node-sdk");
const operator = SDK.Operator.getInstance();

const changeText = (inputText) => {
    let outputText = Array.from(inputText)
    
    outputText.forEach((v,i,a) => {
      a[i] = (i%2) ? v.toUpperCase() : v.toLowerCase()
    })
    
    return outputText.join('') 
  }


operator.getInPort("input").onMessage((message) => {
    //Call the function and pass the input message
    let newMessage = changeText(message)

    //Send newMessage to output port
    operator.getOutPort("output").send(changeText(newMessage));
     
});

/**
 * A keep alive hook for the node process.
 * @param tick length of a heart beat of the operator
 */
 function keepAlive(tick) {
    setTimeout(() => {
      keepAlive(tick);
    }, tick);
  }
  

// keep the operator alive in 1sec ticks
keepAlive(1000);

Step 7

The last step, in order to run the operator script, is to wrap this as a subprocess.

Create a parent.js file like below

Notice the nodeOperator.on() function used when receiving a message from an Operator output port

and nodeOperator.send() to send a message to an Operator input port

const  cp  = require('child_process');
const rl = require('readline');

//First, spawn the subprocess
const nodeOperator = cp.spawn('node', ['script.js'], {
    stdio: ['inherit', 'inherit', 'inherit', 'ipc'],
})

//When message received from the operator, log to console and read a new line
//The first time, it will receive the Initializing operator message that will trigger
//the first readline fx execution
nodeOperator.on('message', function(data) {
    console.log(data);
    readLine();
});

//Create readline interface
let interface = rl.createInterface({
    input: process.stdin,
    output: process.stdout
  });


const readLine = function () {
    interface.question('Input Text: ', function (input) {
      // exit text
      if (input == 'exit') {
            interface.close();         
            process.exit(0)
        }
        
        //Log the input text
        console.log(`Received: ${input}`);
        //Send the text to the operator input port
        nodeOperator.send({name: 'input', value : input})
      
    });
  };

Step 8

Run the parent.js and interact with the operator.

As you can see, the input text read from the terminal is sent to the input port of the operator, and the response is log on to the console.

The response, is a JSON object in the form

{
name: <port_name>,
value: <value>
}

At this point, you can even place breakpoints and debug the code like any normal nodejs project.

Step 9

Once you are done with the operator’s script, simply copy and paste the code from the script.js into the operator’s script in the Data Intelligence Pipeline Modeler and run it from there.

You should keep the parent.js file on your local folder as this is just used to instantiate and interact with the operator

Conclusion

it’s possible to locally develop the Node.JS operator logic with no much effort.

More complex scenarios could be achieved by adding more logic to the parent.js code.

If you require to use 3rd party modules, you can install the modules as you would do in a normal node project, by using npm or yarn. Just be sure to run the operator using a Docker image you created that includes this packages while migrating the code to SAP Data Intelligence.

Martin