Skip to Content
Technical Articles
Author's profile photo Yuliya Reich

SAP Data Intelligence: Get notifications about your pipelines

I’m so happy to find time for writing my next blog post 😊 Today’s topic is alert notification, and I’m going to keep it short and simple.

I had to provide notification pipelines by each DI project that I delivered. There are some cases when customers want to be notified about relevant events from SAP Data Intelligence, for example:

  • a pipeline failed, and a customer wants to receive the notification about it
  • as soon a pipeline is completed, a customer want to receive the notification with the result.

For the first case SAP Data Intelligence offers a template for email notification. To send an email a SMTP connection must be established in DI. After that you can use standard template “Example Notification”:

The workflow is following: The “Pipeline” operator triggers your pipeline. If there is no error, then an event will be sent to the “Workflow Terminator”. In this case no email will be sent. If there is an error, then an event will be sent to the “Notification” operator (“Send Email” operator can be used too). I’m going to use the pipeline (PL_Alert) with the Custom Python Operator (ZPython) that consists division by zero. This will produce an error. The email message contains in this case the raw output:

Obviously, customers will be annoyed about such email. To make the email more user friendly, you can customize the email using a custom operator. I’m going to use Python. Sometimes customers don’t want to receive unnecessary details, just the information that a pipeline failed:

#processing the error message
def on_input(error):
    email_body = "Dear User,\n\nyour pipeline failed."
    api.send("output", email_body)
api.set_port_callback("input", on_input)

The email received by a customer:

Of course, you can add more information into the email, e.g. error message, timestamp, pipeline name, etc. To add the error message, you should parse JSON of the “Pipeline” operator’s raw output. The email could be like:

This is only an example. Sure, you can make it more informative and official.

Using Custom Operators, you can also send notifications to Microsoft Teams, Slack, etc. It is up to you to decide what messenger should be used.

As demonstrated, you can create your notification pipeline smoothly. Depending on your logic, you can choose a standard template pipeline or customized pipeline. You can also build the “Notification” or “Send Email” in your pipeline directly.

Feel free to share your thoughts/ ideas in the comments 😊

Assigned tags

      12 Comments
      You must be Logged on to comment or reply to a post.
      Author's profile photo Tim Huse
      Tim Huse

      Dear Yuliya, great blog post! Thanks for sharing 🙂

      Author's profile photo Yuliya Reich
      Yuliya Reich
      Blog Post Author

      Hi Tim. Thanks! I'm waiting for your next DI post 😉

      Author's profile photo Lohit Kumar
      Lohit Kumar

      Great Blog Post Yuliya. Is it possible for you to share SMTP settings? I have tried this earlier and got the following error. :

      "Group messages: Group: default; Messages: Graph failure: tls: first record does not look like a TLS handshake Process(es) terminated with error(s). restartOnFailure==false"

      I wasn't able to resolve it yet. Any help is highly appreciated.

      Thanks,

      Lohit.

      Author's profile photo Yuliya Reich
      Yuliya Reich
      Blog Post Author

      Hi Lohit,

      please, try to add "insecureSkipVerify": true into pipeline's JSON for the Send Email/ Notification operator in the config part.

      Regards,

      Yuliya

      Author's profile photo Lohit Kumar
      Lohit Kumar

      Thanks for your response, Yuliya.

      I have added "insecureSkipVerify": true to the config part of the graph's JSON, I am still getting the same error message. Any idea what could be the issue?

      Best Regards,

      Lohit.

      Author's profile photo Yuliya Reich
      Yuliya Reich
      Blog Post Author

      Hi Lohit,

      in this case you have to look into your SMTP server. For this particular example, I used my private email in public cloud. I had a customers that used their SMTP by setting VPN, and customers that used relay server. My suggestions: check network settings, firewalls, port (maybe it is blocked).

      Regards,

      Yuliya

      Author's profile photo Lohit Kumar
      Lohit Kumar

      Thanks Yuliya. I will try checking that. 🙂

      Author's profile photo Johannes EDER
      Johannes EDER

      Hi!

      I am facing the same problem with the error message "tls: first record does not look like a TLS handshake".

      Are there already any known solutions?

      Sending the mail without the use of TLS works fine.

      I am not sure but it might be some docker specific problem like https://stackoverflow.com/questions/61907287/docker-first-record-does-not-look-like-a-tls-handshake-error-when-pull-image

      Thanks and best

      Johannes

      Author's profile photo Mohammad Safiullah
      Mohammad Safiullah

      Dear Yulia,

      Great blog. Thanks for sharing.

      I was trying to implement the same in DI 3.1 on-premise system but getting a error as such "Graph failure: dial tcp 74.125.193.108:587: connect: connection refused".

      Here are the steps I have followed;

      1. set up personal Gmail account
      2. added 2 way authentication with app password
      3. written simple python code using the "python" operator as below
        1. df = "Hello World"
        2. api.send("output",df)
      4. selected manual in connection setting in "send email" operator
        1. host - smtp.gmail.com
        2. port - 587'
        3. user name - <gmail id>
        4. pwd - <app password obtained from step 2>
        5. tried setting insecure on and off as mentioned in the comments but in vain
        6. selected TLS as port entered 587
        7. deafult to - <recipient gmail id>
      5. saved and executed

      Could you please share some feedback on the error .Have I missed anything in the steps mentioned above.

      It would be very kind of you if you can help me to understand the error and resolve as I am trying to create a simple email notifcation pipeline for a customer using SAP DI

      Regards

      Safi

      Author's profile photo Yuliya Reich
      Yuliya Reich
      Blog Post Author

      Hi Safi,

      as I tried it with a gmail account for sender, it didn't work for me. For this blog post I used another account, not gmail (I didn't want to lose time for investigation gmail settings for that). You can test your pipeline with another email domain.

       

      Regards,

      Yuliya

      Author's profile photo Mohammad Safiullah
      Mohammad Safiullah

      Hi Yulia,

      I did try with other email providers but it threw same error "Graph failure: dial tcp <diff ip for email providers>:587: connect: connection refused".

      Unable to figure out what could the reason behind the error . There could be the possibility of firewall issue as I am using SAP DI within VPN .  I am checking that well too .Hope this problem gets resolved.

       

      Regards

      Safi

      Author's profile photo Mohammad Safiullah
      Mohammad Safiullah

      Hi ,

      Thanks for the reply .It is an firewall issue .I even tried sending notification using python code in jupyter notebook other than using the operator and found that it is due to firewall issue as SAP DI is connection using some VPN

       

      Regards

      Safi