SAP Data Intelligence - Custom Node.JS Operator with 3rd Party libraries

[martin_donadio]

SAP Data Intelligence lets you create custom Operators to implement different functionalities not covered out of the box by the predefined ones.

This functionalities that could be for example some hashing algorithms, zipping files, performing some encryption algorithms, could be easily implemented by importing 3rd party library using different Subengines (Node.JS, Python, C++). You can find more information here.

In this blog post. I will focus on Node.JS Subengine and share the steps performed to create a Custom Operator that implements bcrypt .

There might be multiple ways to achieve this, and I will share just the one I followed that worked for me (but sure it's not the only one).

One approach is to create everything in your local environment, assemble the solution and use the repository view in the Modeler to import that solution containing the custom(s) Operator(s).

The hardest part of this approach is to know exactly how the folder structure should looks like and how the solution metadata descriptor should be formed.

On the other hand, using the Modeler, the approach should be like this

Step 1

Open the Pipeline Modeler, in the Operators view, click on Create Operator

In the popup window complete the Name, Display Name, Base Operator and Category.

The Name could be a FQDM, this will be reflected later in the folder structure.

Chose "Node Base Operator" to use the Node.JS Subengine.

Select a predefined Category, or just type a new Category Name if you prefer to group all custom Operators. Note that by default view, you won't see the new category

 

Step 2

Click on Customize Visible Categories and select the new category in case you created a new one in the previous step

 

 

Step 3

Change the Icon and complete the Input and Output ports with name and type. In this example, I will use one input String port and one output String port.

 

 

Step 4

Click on Repository tab to switch the view and export the operator as a Solution. This will download a Zip file containing all necessary folder structure and solution metadata so we can import later the Operator with the required modules.

In the Repository view, navigate to Subengines -> com -> sap -> node -> operators -> com -> acme -> operatorz.

Right click on operatorz and select Export as Solution

Complete name and version as X.Y.Z

Finally, click on Export Solution to download the file

 

Step 5

At this point, we will work on the downloaded file to include the Node.JS modules and the Script file with the code for the custom operator.

Extract the file and navigate to the lowermost folder ( ...com/acme/operatorz )

We need to include the "node_modules" folder with all the packages we want to use in the operator.

The easy way to do this, is to run "npm install -save <package>"

 

Step 6

Modify the script.js file and write the source code for the Custom Operator. In this example operator I will use bcrypt to hash an input string using 10 salt rounds.

 

const SDK = require("@sap/vflow-sub-node-sdk");

const operator = SDK.Operator.getInstance();



const bcrypt = require('bcrypt');

const saltRounds = 10;



/**

 * This operator receives messages on port "in1",

 * generates salt using 10 rounds and hash the 

 * message text using that salt

 * to port "out1".

 */

operator.getInPort("in1").onMessage(async (msg) => {

 

  const salt = bcrypt.genSaltSync(saltRounds);

  const hash = bcrypt.hashSync(msg, salt);



  operator.getOutPort("out1").send(hash );

});



/**

 * A keep alive hook for the node process.

 * @param tick length of a heart beat of the operator

 */

function keepAlive(tick) {

  setTimeout(() => {

    keepAlive(tick);

  }, tick);

}



// keep the operator alive in 1sec ticks

keepAlive(1000);

 

Step 7

Zip together the content and manifest.json (don't zip the root folder OperatorZ-0.0.1, because this will create that folder as the root one in the zip file)

Step 8

Import the solution into the Repository. Delete the operator folder (operatorz) to avoid conflicts and Import the zip file generated in the step before (containing node_modules and script)

 

Test the Custom Operator

Create a simple Pipeline with two operators, a Terminal and the Custom Operator.

The custom operator will receive some text and reply with the hashed text using bcrypt

 

Run the Pipeline and write some text

 

 

The terminal sends the texts to the custom operator that returns the result of hashing the text with 10 salt rounds.

 

Hash: $2b$10$Xa0NzV6Py1McmaGWbs0K/O.lQHOuj3OrHe7acKNCifZx0eZCVBg.u

Text: doesthisworked?

 

Not sure if this really worked?

 

 

Conclusion

With not much effort, it's possible to generate a custom nodejs based operator and include 3rd party modules.

 

* One important information to share, is that there are some modules that contains platform specific files so if you perform the steps using Windows, you might face troubles when importing the Operator to Data Intelligence.

I found this situation with "bcrypt" module. After the first attempt to run the pipeline I got this error message in the Custom Operator

Error: /vrep/vflow/subengines/com/sap/node/operators/com/acme/operatorZ/node_modules/bcrypt/lib/binding/napi-v3/bcrypt_lib.node: invalid ELF header

The solution I found was to perform the npm i -save bcrypt command in Ubuntu and export the node_module from there.

For other modules, like nodemailer I found no issue working in Windows.

 

I hope you find this blog post useful for your custom operator implementation

 

Martin