SAP Data Intelligence: Get notifications about your pipelines
I’m so happy to find time for writing my next blog post 😊 Today’s topic is alert notification, and I’m going to keep it short and simple.
I had to provide notification pipelines by each DI project that I delivered. There are some cases when customers want to be notified about relevant events from SAP Data Intelligence, for example:
- a pipeline failed, and a customer wants to receive the notification about it
- as soon a pipeline is completed, a customer want to receive the notification with the result.
For the first case SAP Data Intelligence offers a template for email notification. To send an email a SMTP connection must be established in DI. After that you can use standard template “Example Notification”:
The workflow is following: The “Pipeline” operator triggers your pipeline. If there is no error, then an event will be sent to the “Workflow Terminator”. In this case no email will be sent. If there is an error, then an event will be sent to the “Notification” operator (“Send Email” operator can be used too). I’m going to use the pipeline (PL_Alert) with the Custom Python Operator (ZPython) that consists division by zero. This will produce an error. The email message contains in this case the raw output:
Obviously, customers will be annoyed about such email. To make the email more user friendly, you can customize the email using a custom operator. I’m going to use Python. Sometimes customers don’t want to receive unnecessary details, just the information that a pipeline failed:
#processing the error message def on_input(error): email_body = "Dear User,\n\nyour pipeline failed." api.send("output", email_body) api.set_port_callback("input", on_input)
The email received by a customer:
Of course, you can add more information into the email, e.g. error message, timestamp, pipeline name, etc. To add the error message, you should parse JSON of the “Pipeline” operator’s raw output. The email could be like:
This is only an example. Sure, you can make it more informative and official.
Using Custom Operators, you can also send notifications to Microsoft Teams, Slack, etc. It is up to you to decide what messenger should be used.
As demonstrated, you can create your notification pipeline smoothly. Depending on your logic, you can choose a standard template pipeline or customized pipeline. You can also build the “Notification” or “Send Email” in your pipeline directly.
Feel free to share your thoughts/ ideas in the comments 😊
Dear Yuliya, great blog post! Thanks for sharing 🙂
Hi Tim. Thanks! I'm waiting for your next DI post 😉
Great Blog Post Yuliya. Is it possible for you to share SMTP settings? I have tried this earlier and got the following error. :
"Group messages: Group: default; Messages: Graph failure: tls: first record does not look like a TLS handshake Process(es) terminated with error(s). restartOnFailure==false"
I wasn't able to resolve it yet. Any help is highly appreciated.
please, try to add "insecureSkipVerify": true into pipeline's JSON for the Send Email/ Notification operator in the config part.
Thanks for your response, Yuliya.
I have added "insecureSkipVerify": true to the config part of the graph's JSON, I am still getting the same error message. Any idea what could be the issue?
in this case you have to look into your SMTP server. For this particular example, I used my private email in public cloud. I had a customers that used their SMTP by setting VPN, and customers that used relay server. My suggestions: check network settings, firewalls, port (maybe it is blocked).
Thanks Yuliya. I will try checking that. 🙂
I am facing the same problem with the error message "tls: first record does not look like a TLS handshake".
Are there already any known solutions?
Sending the mail without the use of TLS works fine.
I am not sure but it might be some docker specific problem like https://stackoverflow.com/questions/61907287/docker-first-record-does-not-look-like-a-tls-handshake-error-when-pull-image
Thanks and best
Great blog. Thanks for sharing.
I was trying to implement the same in DI 3.1 on-premise system but getting a error as such "Graph failure: dial tcp 126.96.36.199:587: connect: connection refused".
Here are the steps I have followed;
Could you please share some feedback on the error .Have I missed anything in the steps mentioned above.
It would be very kind of you if you can help me to understand the error and resolve as I am trying to create a simple email notifcation pipeline for a customer using SAP DI
as I tried it with a gmail account for sender, it didn't work for me. For this blog post I used another account, not gmail (I didn't want to lose time for investigation gmail settings for that). You can test your pipeline with another email domain.
I did try with other email providers but it threw same error "Graph failure: dial tcp <diff ip for email providers>:587: connect: connection refused".
Unable to figure out what could the reason behind the error . There could be the possibility of firewall issue as I am using SAP DI within VPN . I am checking that well too .Hope this problem gets resolved.
Thanks for the reply .It is an firewall issue .I even tried sending notification using python code in jupyter notebook other than using the operator and found that it is due to firewall issue as SAP DI is connection using some VPN
While trying to use the notification operator we are getting below errors:
With TLS: we are getting handshake error.
Without TLS: unrecognized authentication type.
Could you please suggest the approach?
please, read comments above, and check your firewall.
Hi Yuliya Reich
Does this suggest we create another graph to which we delegate the trigger control and add the notification capability?
If yes, is there a roadmap item, to make this a platform feature? This can be a configuration belonging to a graph where we can e.g. choose notification on error/success to be sent, and a notification profile. The notification profile is configured external to the graphs at the platform level and points to an e-mail template and other settings for notification.
A much better alternative, out-of-the-box integration with the SAP Alert Notification Service for SAP BTP would be great.
thanks for your thoughts.
You can use the pipeline above or e.g. predefined operator (e.g. "Send Email") in your pipeline, see the message form Mohammad. In the last case you don't need to create an additional pipeline.
Regarding the second part of your message, I don't have much to add here, because I'm not part of the Product Management team. I checked the roadmap, but couldn't find anything that can answer your question. I haven't worked before with the Alert Notification Service, so I have to check if it's possible to connect these two services.
Hi Yuliya Reich
Thanks for your response. Having an operator within the pipeline itself wouldn't help because when the pipeline is cancelled due to a platform problem (which is generally the case for alerting rather than exception management), the "Send E-mail" operator will not be executed. So, it should be the platform that is taking care of that; or as in your example another pipeline checking the health of the original pipeline. The latter is a bit of an inconvenience for users especially considering the SAP DI platform can handle that as a feature, ideally with out-of-the-box integration (like with SAP HANA Cloud, SAP HANA Database) with the SAP Alert Notification Service for SAP BTP.
Thanks Yuliya for an approach for capturing notifications from pipelines in DI, however it was interesting to know DI's suggested approach for any system alerts / configs / in fact platform related notifications. Will that be custom pipelines with diagnostic logs captured and notified using operators? Or any other possibility in DI engine?
Thanks Yuliya for a great blog, really helpful!
Hello Yuliya Reich,
Very good useful post. Thanks for sharing it. I tried to use Example notification graph and able to send email.
But when I try to make changes to it using your custom python script, I am not able to send email.
Could you tell me what would be the output port of python operator. Is it possible to provide the graph code.