Autonomous Self-Analysis System I: Get Email Notification with Issue Analysis Report on a HANA Issue

[former_member366190]

Working in an SAP HANA mission critical data center, the DBA colleagues are usually monitoring multiple SAP HANA systems. It has been quite challenging for them to react quickly and properly when the productive HANA system encounters critical issues. With the help of SAP HANA dump analyzer, the workflow in analyzing the critical issue can be automated now, and the DBA will get the analysis result much faster. This blog will introduce an enhanced solution, an autonomous self-analysis system: get notified via email with issue analysis report automatically on a HANA issue. With such a solution, instead of reacting in panic when someone reports the HANA system is not responding, the DBA can be prepared. He/she can tell what is wrong and take corresponding actions based on the email notification containing the issue analysis report.

To achieve such an autonomous self-analysis system, several other tools are used in conjunction with SAP HANA dump analyzer (in command line mode). For example, SAP HANAsitter is used for capturing specific HANA information automatically if the defined conditions are met. In this blog, we are using HANASitter to capture the SAP HANA runtime dump automatically under pre-defined conditions (e.g. the HANA system is hanging).

The autonomous self-analysis system describes the following scenario in Fig.1. Please be noted, this blog is only using a simple example to explain how the system/solution works, e.g. the system only sends an email to one recipient when the defined issue is detected etc. If you want to run it on your production system, you need to complete the implementation based on the requirements from your real-life scenario.

This blog provides the code snippet to make it easier to get started. If you are going to implement the solution on a production environment, you need to fully test it before running it productively as you will run it “at your own risk”.

Fig.1 Example Scenario of the Autonomous Self-Analysis System

Technically, when the email notification is generated, there are two options to provide the analysis reports.

1. Attach the analysis report to the email as attachment or


2. Copy the analysis report to a web server, provide the URL in the email notification to access the analysis report. In this blog, Tomcat is used as the web server.

Before implementing the solution, you need to contact your email server administrator to see how to configure the email sending. The following examples emphasize on the key steps of sending the email. There might be other steps required when implementing in your own environment, e.g. authentication steps for sending emails etc..

Send the Email Notification with Analysis Report as Attachment

There are many tools available to send email with attachment. In this blog, we will integrate the email function with SAP HANASitter using Python’s email module. The following Python code snippet shows how to email the target recipient with the analysis report as attachment.

from email.mime.application import MIMEApplication

from email.mime.multipart import MIMEMultipart

import smtplib



#Define the send from email, send to email, content of the email message, attachment file

fromMail = "fromMail@yourcompany.com"    

toMail = "toMail@yourcompany.com"

body = "Dear Recipient(s) !\n\nThere is a HANA issue detected, the issue is analyzed and please view the  

        analysis report in the attachment!\n\n\nKind Regards\nSAP HANA dump analyzer"

rteName = "runtime dump name"

attachmentFileName="/<directory>/" + rteName + "/analysis.html"



msg = MIMEMultipart()

msg['Subject'] = "HANA dump analyzer Report for " + rteName 

msg['From'] = fromMail

msg['To'] = toMail

attachment = MIMEApplication (file(attachmentFileName).read(), _subtype="html")

attachment.add_header('Content-Disposition','attachment',filename="analysis report" + rteName)

msg.attach(attachment)



# using smtp protocol to send email

s = smtplib.SMTP('mail.yourcompany.corp')

s.sendmail(fromMail, toMail, msg.as_string())

s.quit()

Send the Email Notification with an URL to the Analysis Report

A web server (e.g. Tomcat Server in our example) can be started and used to display the analysis report. The  steps are:

• Start the Tomcat on the target server.
  
  
  
  • Download the Tomcat and decompress it.
  
  
  • Start Tomcat, by taking the default configuration, 8080 will be used as the HTTP port if it’s not used yet.
    

    cd <Tomcat directory>/bin/
    
    ./catalina.sh start
    
    

    
    More documentation of setting up Tomcat can be found via the link.
  
  
  
  


• Go to Tomcat Web Application Manager Page, i.e. via
  

  HTTP://<servername:port>/manager/html​​

  
  Go to Deploy section to “Deploy directory or WAR file located on server”, then provide the target directory in the “Context path” and press “Deploy”.


• Once the analysis report is generated, copy the analysis report to the report directory. Create the URL that can access the analysis report.


• Send the email using Python email function, provide the URL to the analysis report in the message body.

The above steps describe how to send the email notification with the analysis report if it’s already available. To achieve the automation of the email notification when HANA is not responding, there are two steps missing:

1. Generate the analysis report using SAP HANA dump analyzer command line.


2. Use HANASitter to automate email notification with analysis report on a HANA issue.

Generate the Analysis Report Using SAP HANA dump analyzer Command Line

The SAP HANA dump analyzer can be executed via command line to analyze issue from a provided HANA runtime dump and return the analysis report. The help page of SAP HANA dump analyzer command line is available via:

java -jar HANADumpAnalyzer.jar -help

The SAP HANA dump analyzer provides different options to use the command line in different scenarios.

-v, -version: output the version information of SAP HANA dump analyzer

-s: HANA runtime dump file to be parsed. Currently analyzing only one runtime dump or its zip file is supported via the command line mode

-d: output directory to write the analysis report. If no value is provided for -d option, the analysis report will be opened via browser automatically if there is a browser available

The typical examples are:

Analyze rtedump1.trc, open the analysis report via the browser (on the same host where the command line is executed) directly:

java -jar HANADumpAnalyzer.jar -s rtedump1.trc

Analyze rtedump1.trc, write the analysis report to directory dir:

java -jar HANADumpAnalyzer.jar -s rtedump1.trc -d dir

In this post, you need to write the analysis report to a pre-defined directory.

Use HANASitter to Automate Email Notification with Analysis Report on a HANA Issue

SAP HANASitter is implemented as a python script. It can be used to configure reaction methods (e.g. creation of traces and dumps or collection of performance histories) when specific conditions (like high load) are met. In this blog, the HANASitter is configured to

• Check the HANA system every 10 minutes.


• If the HANA system is not responding “select * from dummy” request after 3 minutes, generates one HANA runtime dump.

We aim to enhance it in the following way:

• After the HANA runtime dump is generated, call the SAP HANA dump analyzer to generate the analysis report.


• Upload the analysis report to the report directory on the Tomcat server.


• Send email notification and provide the URL of the analysis report.

The steps are the following:

1. Start Tomcat server and deploy the report directory (i.e. to place the analysis reports) via Tomcat manager app page.


2. Create and send the analysis report in HANASitter.
   

   import smtplib
   
   from email.mime.text import MIMEText
   
   
   
   def notify_email(rte):
   
       rte_name = rte[:-4]
   
       target_dir = <target directory to place the analysis reports>
   
   
   
   #Call the SAP HANA dump analyzer to generate the analysis report under the target directory
   
       hda_str = "java -jar HANADumpAnalyzer.jar -s " + rte + " -d <target_dir>
   
       os.system(hda_str)
   
       print("hda: analyzed runtime dump using HANA dump analyzer")
   
   
   
   #Get the URL to access the analysis report just generated 
   
      url = "http://<server name:port>/" + rte_name + "/analysis.html"
   
   
   
   #Prepare the email, e.g. from, to, content body, subject
   
       fromMail = "fromMail@yourcompany.com"
   
       toMail = "toMail@yourcompany.com"
   
       body = "Dear Recipient(s) !<br><br>There is a HANA issue detected, the issue is analyzed and please view <a href=" + url +">the analysis report</a> <br><br><br>Kind Regards<br>SAP HANA dump analyzer"
   
       msg = MIMEText(body, 'html')
   
       msg['Subject'] = "HANA dump analyzer Report for " + rte_name
   
       msg['From'] = fromMail
   
       msg['To'] = ToMail
   
   
   
   #Send out the email
   
       s = smtplib.SMTP('mail.yourcompany.corp')
   
       s.sendmail(fromMail, you, msg.as_string())
   
       s.quit()
   
       
   
       return True 

   
   P.S.: Please be noted the above sample code is only explaining the key steps. There are also some steps e.g.user permission, error handling etc. you need to take care of. For example, most likely you will run the HANASitter with sidadm user. SAP HANA dump analyzer is called within HANASitter to generate the analysis report. The Tomcat server is using a different user for displaying the analysis report. If this is the case, the following pre-requisites must be fulfilled:
   
   
   
   • The sidadm should have the authorization to call SAP HANA dump analyzer.
   
   
   • The sidadm should have the authorization to write the analysis report to the target folder.
   
   
   • The analysis report file permission setting should allow the Tomcat server to display it.
   
   
   
   As mentioned in the beginning, you will need to complete your own code/set up to make it fully working.


3. Trigger the email notification when the HANA runtime dump is generated by HANASitter.
   In the example code, it calls the notification function, i.e. notify_email(rte) after the runtime dump is generated in function record_rtedump(rtedumps_interval, hdbcons, comman):
   

   def record_rtedump(rtedumps_interval, hdbcons, comman):
   
       total_printout = ""
   
       for hdbcon_string, host in zip(hdbcons.hdbcons_strings, hdbcons.hosts):
   
           tenantDBString = hdbcons.tenantDBName+"_" if hdbcons.is_tenant else ""
   
           start_time = datetime.now()
   
           if hdbcons.rte_mode == 0: # normal rte dump
   
               rte = "rtedump_normal_"+host+"_"+hdbcons.SID+"_"+tenantDBString+datetime.now().strftime("%Y-%m-%d_%H-%M-%S")+".trc"
   
               filename = (comman.out_dir.replace(".","_")+rte)
   
               os.system(hdbcon_string+'runtimedump dump -c" > '+filename)   # have to dump to std with -c and then to a file with >    since in case of scale-out  -f  does not work
   
               print('write rtedump: ' + filename)
   
   #Start to insert
   
   #Triggering the analysis and email notification
   
               time.sleep(5)
   
               print('start of notification')
   
               notify_email(rte)
   
   #Finish the insert
   
                …

   
    


4. Define the condition for HANASitter and how it should react in config file.
   In this example, we are simply setting the following rule: If HANA is not responding to “select * from dummy” request after 3 minutes, then generate one HANA runtime dump.
   With the above condition, the HANASitter.config file looks like:
   

   # if no response after 180s
   
   -pt 180
   
   # create 1 runtime dump
   
   -nr 1
   
   # check every 600 seconds
   
   -ci 600
   
   # using key
   
   -k <HANASITTERKEY>
   
   -od <target directory to place the HANA runtime dumps>

   
   


5. Start HANASitter using the configuration file.
   

   python hda_hanasitter_emailNotification.py -ff hanasitter.config

   
   

After finishing all the above steps, you will be able to notify the target recipient with the analysis report if your HANA system has been hung for over three minutes.