Skip to Content
Technical Articles
Author's profile photo Yuliya Reich

SAP Data Intelligence: Get notifications about your pipelines

I’m so happy to find time for writing my next blog post 😊 Today’s topic is alert notification, and I’m going to keep it short and simple.

I had to provide notification pipelines by each DI project that I delivered. There are some cases when customers want to be notified about relevant events from SAP Data Intelligence, for example:

  • a pipeline failed, and a customer wants to receive the notification about it
  • as soon a pipeline is completed, a customer want to receive the notification with the result.

For the first case SAP Data Intelligence offers a template for email notification. To send an email a SMTP connection must be established in DI. After that you can use standard template “Example Notification”:

The workflow is following: The “Pipeline” operator triggers your pipeline. If there is no error, then an event will be sent to the “Workflow Terminator”. In this case no email will be sent. If there is an error, then an event will be sent to the “Notification” operator (“Send Email” operator can be used too). I’m going to use the pipeline (PL_Alert) with the Custom Python Operator (ZPython) that consists division by zero. This will produce an error. The email message contains in this case the raw output:

Obviously, customers will be annoyed about such email. To make the email more user friendly, you can customize the email using a custom operator. I’m going to use Python. Sometimes customers don’t want to receive unnecessary details, just the information that a pipeline failed:

#processing the error message
def on_input(error):
    email_body = "Dear User,\n\nyour pipeline failed."
    api.send("output", email_body)
api.set_port_callback("input", on_input)

The email received by a customer:

Of course, you can add more information into the email, e.g. error message, timestamp, pipeline name, etc. To add the error message, you should parse JSON of the “Pipeline” operator’s raw output. The email could be like:

This is only an example. Sure, you can make it more informative and official.

Using Custom Operators, you can also send notifications to Microsoft Teams, Slack, etc. It is up to you to decide what messenger should be used.

As demonstrated, you can create your notification pipeline smoothly. Depending on your logic, you can choose a standard template pipeline or customized pipeline. You can also build the “Notification” or “Send Email” in your pipeline directly.

Feel free to share your thoughts/ ideas in the comments 😊

Assigned Tags

      24 Comments
      You must be Logged on to comment or reply to a post.
      Author's profile photo Tim Huse
      Tim Huse

      Dear Yuliya, great blog post! Thanks for sharing 🙂

      Author's profile photo Yuliya Reich
      Yuliya Reich
      Blog Post Author

      Hi Tim. Thanks! I'm waiting for your next DI post 😉

      Author's profile photo Rajesh PS
      Rajesh PS

      Yuliya Reich

      Is it possible to send material number, purchase order numbers present inside payload via email in case of dead graphs?
      Author's profile photo Lohit Kumar
      Lohit Kumar

      Great Blog Post Yuliya. Is it possible for you to share SMTP settings? I have tried this earlier and got the following error. :

      "Group messages: Group: default; Messages: Graph failure: tls: first record does not look like a TLS handshake Process(es) terminated with error(s). restartOnFailure==false"

      I wasn't able to resolve it yet. Any help is highly appreciated.

      Thanks,

      Lohit.

      Author's profile photo Yuliya Reich
      Yuliya Reich
      Blog Post Author

      Hi Lohit,

      please, try to add "insecureSkipVerify": true into pipeline's JSON for the Send Email/ Notification operator in the config part.

      Regards,

      Yuliya

      Author's profile photo Lohit Kumar
      Lohit Kumar

      Thanks for your response, Yuliya.

      I have added "insecureSkipVerify": true to the config part of the graph's JSON, I am still getting the same error message. Any idea what could be the issue?

      Best Regards,

      Lohit.

      Author's profile photo Yuliya Reich
      Yuliya Reich
      Blog Post Author

      Hi Lohit,

      in this case you have to look into your SMTP server. For this particular example, I used my private email in public cloud. I had a customers that used their SMTP by setting VPN, and customers that used relay server. My suggestions: check network settings, firewalls, port (maybe it is blocked).

      Regards,

      Yuliya

      Author's profile photo Lohit Kumar
      Lohit Kumar

      Thanks Yuliya. I will try checking that. 🙂

      Author's profile photo Johannes EDER
      Johannes EDER

      Hi!

      I am facing the same problem with the error message "tls: first record does not look like a TLS handshake".

      Are there already any known solutions?

      Sending the mail without the use of TLS works fine.

      I am not sure but it might be some docker specific problem like https://stackoverflow.com/questions/61907287/docker-first-record-does-not-look-like-a-tls-handshake-error-when-pull-image

      Thanks and best

      Johannes

      Author's profile photo Mohammad Safiullah
      Mohammad Safiullah

      Dear Yulia,

      Great blog. Thanks for sharing.

      I was trying to implement the same in DI 3.1 on-premise system but getting a error as such "Graph failure: dial tcp 74.125.193.108:587: connect: connection refused".

      Here are the steps I have followed;

      1. set up personal Gmail account
      2. added 2 way authentication with app password
      3. written simple python code using the "python" operator as below
        1. df = "Hello World"
        2. api.send("output",df)
      4. selected manual in connection setting in "send email" operator
        1. host - smtp.gmail.com
        2. port - 587'
        3. user name - <gmail id>
        4. pwd - <app password obtained from step 2>
        5. tried setting insecure on and off as mentioned in the comments but in vain
        6. selected TLS as port entered 587
        7. deafult to - <recipient gmail id>
      5. saved and executed

      Could you please share some feedback on the error .Have I missed anything in the steps mentioned above.

      It would be very kind of you if you can help me to understand the error and resolve as I am trying to create a simple email notifcation pipeline for a customer using SAP DI

      Regards

      Safi

      Author's profile photo Yuliya Reich
      Yuliya Reich
      Blog Post Author

      Hi Safi,

      as I tried it with a gmail account for sender, it didn't work for me. For this blog post I used another account, not gmail (I didn't want to lose time for investigation gmail settings for that). You can test your pipeline with another email domain.

       

      Regards,

      Yuliya

      Author's profile photo Mohammad Safiullah
      Mohammad Safiullah

      Hi Yulia,

      I did try with other email providers but it threw same error "Graph failure: dial tcp <diff ip for email providers>:587: connect: connection refused".

      Unable to figure out what could the reason behind the error . There could be the possibility of firewall issue as I am using SAP DI within VPN .  I am checking that well too .Hope this problem gets resolved.

       

      Regards

      Safi

      Author's profile photo Mohammad Safiullah
      Mohammad Safiullah

      Hi ,

      Thanks for the reply .It is an firewall issue .I even tried sending notification using python code in jupyter notebook other than using the operator and found that it is due to firewall issue as SAP DI is connection using some VPN

       

      Regards

      Safi

      Author's profile photo Indu Khurana
      Indu Khurana

      Hello Yuliya,

      While trying to use the notification operator we are getting below errors:

      With TLS: we are getting handshake error.

      Without TLS: unrecognized authentication type.

      Could you please suggest the approach?

      Thanks.

      Author's profile photo Yuliya Reich
      Yuliya Reich
      Blog Post Author

      Hi Indru,

      please, read comments above, and check your firewall.

      Regards,

      Yuliya

      Author's profile photo Serdar Simsekler
      Serdar Simsekler

      Hi Yuliya Reich

      Does this suggest we create another graph to which we delegate the trigger control and add the notification capability?

      If yes, is there a roadmap item, to make this a platform feature? This can be a configuration belonging to a graph where we can e.g. choose notification on error/success to be sent, and a notification profile. The notification profile is configured external to the graphs at the platform level and points to an e-mail template and other settings for notification.

      A much better alternative, out-of-the-box integration with the SAP Alert Notification Service for SAP BTP would be great.

      Kind Regards

      Serdar Simsekler

      Author's profile photo Yuliya Reich
      Yuliya Reich
      Blog Post Author

      Hello Serdar,

      thanks for your thoughts.

      You can use the pipeline above or e.g. predefined operator (e.g. "Send Email") in your pipeline, see the message form Mohammad. In the last case you don't need to create an additional pipeline.

      Regarding the second part of your message, I don't have much to add here, because I'm not part of the Product Management team. I checked the roadmap, but couldn't find anything that can answer your question. I haven't worked before with the Alert Notification Service, so I have to check if it's possible to connect these two services.

      Regards,

      Yuliya

      Author's profile photo Serdar Simsekler
      Serdar Simsekler

      Hi Yuliya Reich

      Thanks for your response. Having an operator within the pipeline itself wouldn't help because when the pipeline is cancelled due to a platform problem (which is generally the case for alerting rather than exception management), the "Send E-mail" operator will not be executed. So, it should be the platform that is taking care of that; or as in your example another pipeline checking the health of the original pipeline. The latter is a bit of an inconvenience for users especially considering the SAP DI platform can handle that as a feature, ideally with out-of-the-box integration (like with SAP HANA Cloud, SAP HANA Database) with the SAP Alert Notification Service for SAP BTP.

      Kind Regards

      Serdar Simsekler

      Author's profile photo Rajesh PS
      Rajesh PS

      Serdar Simsekler Yuliya Reich

      Tim Huse

      Kai-Michael Roesner

      I'm having a scenario - kafka to HANA DB update in SDI, it is always been a Major challenge with respect to Error handling, reprocessing and notification with Payload.

       

      I haven't got any proper response from SDI community and corrective directions on this part and its seriously  been a Big lag and not sure what is the Roadmap on this. SAP PI/PO, CPI and BPM are excellent in this part.

       

      https://answers.sap.com/questions/13916111/retrieve-payload-for-dead-graphs.html?source=email-global-notification-qa-answer-published-to-question

       

      https://answers.sap.com/questions/13795166/manage-technical-and-business-exceptions-in-sap-da.html

      Author's profile photo Leena Gopinath
      Leena Gopinath

      Thanks Yuliya for an approach for capturing notifications from pipelines in DI, however it was interesting to know DI's suggested approach for any system alerts / configs / in fact platform related notifications. Will that be custom pipelines with diagnostic logs captured and notified using operators? Or any other possibility in DI engine?

      Author's profile photo Indu Khurana
      Indu Khurana

      Thanks Yuliya for a great blog, really helpful!

      Author's profile photo Sirisha Pappala
      Sirisha Pappala

      Hello Yuliya Reich,

      Very good useful post. Thanks for sharing it. I tried to use Example notification graph and able to send email.

      But when I try to make changes to it using your custom python script, I am not able to send email.

      Could you tell me what would be the output port of python operator. Is it possible to provide the graph code.

      Thanks,

      Sirisha

      Author's profile photo Namrata Kar
      Namrata Kar

      Hi Yuliya Reich

      Is it possible to send email body in html format? I am unable to find any material related to this.

      Thanks in advance.

      Author's profile photo Rajesh PS
      Rajesh PS

      Serdar Simsekler Yuliya Reich

      Tim Huse

      Kai-Michael Roesner

      I'm having a scenario - kafka to HANA DB update in SDI, it is always been a Major challenge with respect to Error handling, reprocessing and notification with Payload.

       

      I haven't got any proper response from SDI community and corrective directions on this part and its seriously  been a Big lag and not sure what is the Roadmap on this. SAP PI/PO, CPI and BPM are excellent in this part.

       

      https://answers.sap.com/questions/13916111/retrieve-payload-for-dead-graphs.html?source=email-global-notification-qa-answer-published-to-question

       

      https://answers.sap.com/questions/13795166/manage-technical-and-business-exceptions-in-sap-da.html