Kibana Based alerts for Non Performant API and Error Status Code

Problem Statement :

Introduce Alerts mechanism in Kibana and integrate with Teams channel

a. For HTTP Errors
b. For Performance

For HTTP Errors

Query Preparation for Error Extraction

1. Prepare the Query in “Query workbench and do the explain ,Copy this Json somewhere

Select count(*) from logs-json-* where logs.status >= 400 and logs.localServerName is "api.cuxuzunfzu-public.model-t.cc.commerce.ondemand.com" and logs.requestFirstLine LIKE "%electronics%"

Where electronics is Site ID for which you want to monitor errors  on api Node . You can customize the Query based on  your needs

Translated JSON looks like this

{
  "from": 0,
  "size": 0,
  "query": {
    "bool": {
      "filter": [
        {
          "bool": {
            "must": [
              {
                "bool": {
                  "must": [
                    {
                      "range": {
                        "logs.status": {
                          "from": 400,
                          "to": null,
                          "include_lower": true,
                          "include_upper": true,
                          "boost": 1
                        }
                      }
                    },
                    {
                      "term": {
                        "logs.localServerName.keyword": {
                          "value": "api.cuxuzunfzu-public.model-t.cc.commerce.ondemand.com",
                          "boost": 1
                        }
                      }
                    },
                    {
                      "wildcard": {
                        "logs.requestFirstLine": {
                          "wildcard": "*electronics*",
                          "boost": 1
                        }
                      }
                    },
                    {
                      "range": {
                        "time": {
                          "from": "now-10m",
                          "to": null,
                          "include_lower": true,
                          "include_upper": true,
                          "boost": 1
                        }
                      }
                    }
                  ],
                  "adjust_pure_negative": true,
                  "boost": 1
                }
              }
            ],
            "adjust_pure_negative": true,
            "boost": 1
          }
        }
      ],
      "adjust_pure_negative": true,
      "boost": 1
    }
  },
  "_source": {
    "includes": [
      "COUNT"
    ],
    "excludes": [
      
    ]
  },
  "aggregations": {
    "COUNT_0": {
      "value_count": {
        "field": "_index"
      }
    }
  }
}
 You can also change the wildcard from exact match Translated json like this

2. Please update the wildcard query with match-phrase query in above Json.

WildCard Query:

   "wildcard": {
    "logs.requestFirstLine": {
      "wildcard": "*electronics*",
      "boost": 1
     }
   }
 

Match Phrase Query:

  "match_phrase": {
   "logs.requestFirstLine": {
     "query": "electronics",
      "boost": 1
   }
  }

Note: If we have to use hyphen "-" is not recognized as a character in the LIKE/WILDCARD query, so you can use MATCH_PHRASE instead. If you want to monitor error occured in last 10m or nay frequency this is the code Snippet {
                      "range": {
                        "time": {
                          "from": "now-10m",
                          "to": null,
                          "include_lower": true,
                          "include_upper": true,
                          "boost": 1
                        }
                      }
                    } } The above Jso has to copied in Extraction Query after craeting the montor Create the Monitor like below  Create the Trigger condition You can define priorty based on your business needs like in 10 mins more that 5 erros are high alerts Here in below example any error is High 

Create the Action

Give mandatory details and Select Destination

Add teams destination 

Select the teams channel configuration

Teams specific task

1. You have to create the teams channel

2. Add the App name in there Incoming Webhook this will give you Webhook URL which you can use for the Teams alert integration.

You have to add new Destination

Add new Desination  and add the above webhookURL below

Create the message which you want to share on teams once alerts occur and Copy in Action Message steps

{
   "@type":"MessageCard",
   "@context":"https://schema.org/extensions",
   "summary":"Error for ELECTRONICS",
   "themeColor":"FF0000",
   "sections":[
      {
         "activityImage":"URL",
         "activityTitle": "Errors Produced for ELECTRONICS SITE in Last 10 minutes ",
        
         "facts":[
            {
               "name":"Reason: ",
               "value":"ERROR-CODE - More than 400. <b>Number of Occurrences: {{ctx.results.0.hits.total.value}}</b>""
            }
         ],
         "text":"Api Calls failed.",
         "potentialAction":[
            {
               "@type":"OpenUri",
               "name" : "Check Kibana",
               "targets":[
                  {
                     "os":"default",
                     "uri":"URL FOR KIBANA "
                  }
               ]
            }
           
         ]
      }
   ]
}

b. For Performance

All the steps are same except the Query extraction are same . Query extraction which has to be done very carefully

Create your Query For example I have created the Query which
a.Fetch API from ELECTORNICS which are taking more than 3 seconds

SELECT count(*)  FROM logs-json-* where logs.responseTime  is not null and logs.localServerName is "api.cuxuzunfzu-public.model-t.cc.commerce.ondemand.com" and logs.requestFirstLine like "%ELECTRONCIS%" 
and  CAST(logs.responseTime AS INTEGER)  > 3000 

{
  "from": 0,
  "size": 0,
  "query": {
    "bool": {
      "filter": [
        {
          "bool": {
            "filter": [
              {
                "bool": {
                  "filter": [
                    {
                      "script": {
                        "script": {
                          "source": "XXXXXXXXXXXXXcWwuZXhwcmVzc2lvbi5mdW5jdGlvbi5GdW5jdGlvbkRTTCQy7+eYHhdeXG4CAARMAA12YWwkYXJndW1lbnRzdAAQTGphdmEvdXRpbC9MaXN0O0wADHZhbCRmdW5jdGlvbnQAPUxvcmcvb3BlbnNlYXJjaC9zcWwvZXhwcmVzc2lvbi9mdW5jdGlvbi9TZXJpYWxpemFibGVGdW5jdGlvbjtMABB2YWwkZnVuY3Rpb25OYW1ldAA1TG9yZy9vcGVuc2VhcmNoL3NxbC9leHByZXNzaW9uL2Z1bmN0aW9uL0Z1bmN0aW9uTmFtZTtMAA52YWwkcmV0dXJuVHlwZXQAJ0xvcmcvb3BlbnNlYXJjaC9zcWwvZGF0YS90eXBlL0V4cHJUeXBlO3hyADBvcmcub3BlbnNlYXJjaC5zcWwuZXhwcmVzc2lvbi5GdW5jdGlvbkV4cHJlc3Npb26yKjDT3HVqewIAAkwACWFyZ3VtZW50c3EAfgABTAAMZnVuY3Rpb25OYW1lcQB+AAN4cHNyABNqYXZhLnV0aWwuQXJyYXlMaXN0eIHSHZnHYZ0DAAFJAARzaXpleHAAAAABdwQAAAABc3IAMW9yZy5vcGVuc2VhcmNoLnNxbC5leHByZXNzaW9uLlJlZmVyZW5jZUV4cHJlc3Npb274AO0rxWvMkAIAA0wABGF0dHJ0ABJMamF2YS9sYW5nL1N0cmluZztMAAVwYXRoc3EAfgABTAAEdHlwZXEAfgAEeHB0ABFsb2dzLnJlc3BvbnNlVGltZXNyABpqYXZhLnV0aWwuQXJyYXlzJEFycmF5TGlzdNmkPL7NiAbSAgABWwABYXQAE1tMamF2YS9sYW5nL09iamVjdDt4cHVyABNbTGphdmEubGFuZy5TdHJpbmc7rdJW5+kde0cCAAB4cAAAAAJ0AARsb2dzdAAMcmVzcG9uc2VUaW1lfnIAOm9yZy5vcGVuc2VhcmNoLnNxbC5vcGVuc2VhcmNoLmRhdGEudHlwZS5PcGVuU2VhcmNoRGF0YVR5cGUAAAAAAAAAABIAAHhyAA5qYXZhLmxhbmcuRW51bQAAAAAAAAAAEgAAeHB0ABdPUEVOU0VBUkNIX1RFWFRfS0VZV09SRHhzcgAzb3JnLm9wZW5zZWFyY2guc3FsLmV4cHJlc3Npb24uZnVuY3Rpb24uRnVuY3Rpb25OYW1lC6g4Tc72Z5cCAAFMAAxmdW5jdGlvbk5hbWVxAH4ACnhwdAALaXMgbm90IG51bGxxAH4ACHNyACFqYXZhLmxhbmcuaW52b2tlLlNlcmlhbGl6ZWRMYW1iZGFvYdCULCk2hQIACkkADmltcGxNZXRob2RLaW5kWwAMY2FwdHVyZWRBcmdzcQB+AA5MAA5jYXB0dXJpbmdDbGFzc3QAEUxqYXZhL2xhbmcvQ2xhc3M7TAAYZnVuY3Rpb25hbEludGVyZmFjZUNsYXNzcQB+AApMAB1mdW5jdGlvbmFsSW50ZXJmYWNlTWV0aG9kTmFtZXEAfgAKTAAiZnVuY3Rpb25hbEludGVyZmFjZU1ldGhvZFNpZ25hdHVyZXEAfgAKTAAJaW1wbENsYXNzcQB+AApMAA5pbXBsTWV0aG9kTmFtZXEAfgAKTAATaW1wbE1ldGhvZFNpZ25hdHVyZXEAfgAKTAAWaW5zdGFudGlhdGVkTWV0aG9kVHlwZXEAfgAKeHAAAAAGdXIAE1tMamF2YS5sYW5nLk9iamVjdDuQzlifEHMpbAIAAHhwAAAAAHZyAEdvcmcub3BlbnNlYXJjaC5zcWwuZXhwcmVzc2lvbi5vcGVyYXRvci5wcmVkaWNhdGUuVW5hcnlQcmVkaWNhdGVPcGVyYXRvcgAAAAAAAAAAAAAAeHB0ADtvcmcvb3BlbnNlYXJjaC9zcWwvZXhwcmVzc2lvbi9mdW5jdGlvbi9TZXJpYWxpemFibGVGdW5jdGlvbnQABWFwcGx5dAAmKExqYXZhL2xhbmcvT2JqZWN0OylMamF2YS9sYW5nL09iamVjdDt0AEdvcmcvb3BlbnNlYXJjaC9zcWwvZXhwcmVzc2lvbi9vcGVyYXRvci9wcmVkaWNhdGUvVW5hcnlQcmVkaWNhdGVPcGVyYXRvcnQAG2xhbWJkYSRpc05vdE51bGwkODc3ODgwNzckMXQAVChMb3JnL29wZW5zZWFyY2gvc3FsL2RhdGEvbW9kZWwvRXhwclZhbHVlOylMb3JnL29wZW5zZWFyY2gvc3FsL2RhdGEvbW9kZWwvRXhwclZhbHVlO3EAfgAncQB+ABl+cgApb3JnLm9wZW5zZWFyY2guc3FsLmRhdGEudHlwZS5FeHByQ29yZVR5cGUAAAAAAAAAABIAAHhxAH4AFXQAB0JPT0xFQU4=",
                          "lang": "opensearch_query_expression"
                        },
                        "boost": 1.0
                      }
                    },
                    {
                      "wildcard": {
                        "logs.requestFirstLine.keyword": {
                          "wildcard": "*ELECTRONICS*",
                          "boost": "1.0"
                        }
                      }
                    }
                  ],
                  "adjust_pure_negative:true,boost": "1.0"
                }
              },
              {
                "wildcard": {
                  "logs.requestFirstLine.keyword": {
                    "wildcard": "*locationId*",
                    "boost": "1.0"
                  }
                }
              },
              {
                "range": {
                  "time": {
                    "from": "now-10m",
                    "to": null,
                    "include_lower": true,
                    "include_upper": true,
                    "boost": 1
                  }
                }
              }
            ],
            "adjust_pure_negative": "true",
            "boost": "1.0"
          }
        },
        {
          "script": {
            "script": {
              "source": "XXXXXXXX/macCAARMAA12YWwkYXJndW1lbnRzdAAQTGphdmEvdXRpbC9MaXN0O0wADHZhbCRmdW5jdGlvbnQAP0xvcmcvb3BlbnNlYXJjaC9zcWwvZXhwcmVzc2lvbi9mdW5jdGlvbi9TZXJpYWxpemFibGVCaUZ1bmN0aW9uO0wAEHZhbCRmdW5jdGlvbk5hbWV0ADVMb3JnL29wZW5zZWFyY2gvc3FsL2V4cHJlc3Npb24vZnVuY3Rpb24vRnVuY3Rpb25OYW1lO0wADnZhbCRyZXR1cm5UeXBldAAnTG9yZy9vcGVuc2VhcmNoL3NxbC9kYXRhL3R5cGUvRXhwclR5cGU7eHIAMG9yZy5vcGVuc2VhcmNoLnNxbC5leHByZXNzaW9uLkZ1bmN0aW9uRXhwcmVzc2lvbrIqMNPcdWp7AgACTAAJYXJndW1lbnRzcQB+AAFMAAxmdW5jdGlvbk5hbWVxAH4AA3hwc3IAE2phdmEudXRpbC5BcnJheUxpc3R4gdIdmcdhnQMAAUkABHNpemV4cAAAAAJ3BAAAAAJzcgA0b3JnLm9wZW5zZWFyY2guc3FsLmV4cHJlc3Npb24uZnVuY3Rpb24uRnVuY3Rpb25EU0wkMu/nmB4XXlxuAgAETAANdmFsJGFyZ3VtZW50c3EAfgABTAAMdmFsJGZ1bmN0aW9udAA9TG9yZy9vcGVuc2VhcmNoL3NxbC9leHByZXNzaW9uL2Z1bmN0aW9uL1NlcmlhbGl6YWJsZUZ1bmN0aW9uO0wAEHZhbCRmdW5jdGlvbk5hbWVxAH4AA0wADnZhbCRyZXR1cm5UeXBlcQB+AAR4cQB+AAVzcQB+AAcAAAABdwQAAAABc3IAMW9yZy5vcGVuc2VhcmNoLnNxbC5leHByZXNzaW9uLlJlZmVyZW5jZUV4cHJlc3Npb274AO0rxWvMkAIAA0wABGF0dHJ0ABJMamF2YS9sYW5nL1N0cmluZztMAAVwYXRoc3EAfgABTAAEdHlwZXEAfgAEeHB0ABFsb2dzLnJlc3BvbnNlVGltZXNyABpqYXZhLnV0aWwuQXJyYXlzJEFycmF5TGlzdNmkPL7NiAbSAgABWwABYXQAE1tMamF2YS9sYW5nL09iamVjdDt4cHVyABNbTGphdmEubGFuZy5TdHJpbmc7rdJW5+kde0cCAAB4cAAAAAJ0AARsb2dzdAAMcmVzcG9uc2VUaW1lfnIAOm9yZy5vcGVuc2VhcmNoLnNxbC5vcGVuc2VhcmNoLmRhdGEudHlwZS5PcGVuU2VhcmNoRGF0YVR5cGUAAAAAAAAAABIAAHhyAA5qYXZhLmxhbmcuRW51bQAAAAAAAAAAEgAAeHB0ABdPUEVOU0VBUkNIX1RFWFRfS0VZV09SRHhzcgAzb3JnLm9wZW5zZWFyY2guc3FsLmV4cHJlc3Npb24uZnVuY3Rpb24uRnVuY3Rpb25OYW1lC6g4Tc72Z5cCAAFMAAxmdW5jdGlvbk5hbWVxAH4ADnhwdAALY2FzdF90b19pbnRxAH4ADHNyACFqYXZhLmxhbmcuaW52b2tlLlNlcmlhbGl6ZWRMYW1iZGFvYdCULCk2hQIACkkADmltcGxNZXRob2RLaW5kWwAMY2FwdHVyZWRBcmdzcQB+ABJMAA5jYXB0dXJpbmdDbGFzc3QAEUxqYXZhL2xhbmcvQ2xhc3M7TAAYZnVuY3Rpb25hbEludGVyZmFjZUNsYXNzcQB+AA5MAB1mdW5jdGlvbmFsSW50ZXJmYWNlTWV0aG9kTmFtZXEAfgAOTAAiZnVuY3Rpb25hbEludGVyZmFjZU1ldGhvZFNpZ25hdHVyZXEAfgAOTAAJaW1wbENsYXNzcQB+AA5MAA5pbXBsTWV0aG9kTmFtZXEAfgAOTAATaW1wbE1ldGhvZFNpZ25hdHVyZXEAfgAOTAAWaW5zdGFudGlhdGVkTWV0aG9kVHlwZXEAfgAOeHAAAAAGdXIAE1tMamF2YS5sYW5nLk9iamVjdDuQzlifEHMpbAIAAHhwAAAAAXNxAH4AHwAAAAZ1cQB+ACIAAAAAdnIAP29yZy5vcGVuc2VhcmNoLnNxbC5leHByZXNzaW9uLm9wZXJhdG9yLmNvbnZlcnQuVHlwZUNhc3RPcGVyYXRvcgAAAAAAAAAAAAAAeHB0ADtvcmcvb3BlbnNlYXJjaC9zcWwvZXhwcmVzc2lvbi9mdW5jdGlvbi9TZXJpYWxpemFibGVGdW5jdGlvbnQABWFwcGx5dAAmKExqYXZhL2xhbmcvT2JqZWN0OylMamF2YS9sYW5nL09iamVjdDt0AD9vcmcvb3BlbnNlYXJjaC9zcWwvZXhwcmVzc2lvbi9vcGVyYXRvci9jb252ZXJ0L1R5cGVDYXN0T3BlcmF0b3J0ABtsYW1iZGEkY2FzdFRvSW50JDg3Nzg4MDc3JDF0AFQoTG9yZy9vcGVuc2VhcmNoL3NxbC9kYXRhL21vZGVsL0V4cHJWYWx1ZTspTG9yZy9vcGVuc2VhcmNoL3NxbC9kYXRhL21vZGVsL0V4cHJWYWx1ZTtxAH4ALXZyADJvcmcub3BlbnNlYXJjaC5zcWwuZXhwcmVzc2lvbi5mdW5jdGlvbi5GdW5jdGlvbkRTTAAAAAAAAAAAAAAAeHBxAH4AKHEAfgApcQB+ACp0ADJvcmcvb3BlbnNlYXJjaC9zcWwvZXhwcmVzc2lvbi9mdW5jdGlvbi9GdW5jdGlvbkRTTHQAJWxhbWJkYSRudWxsTWlzc2luZ0hhbmRsaW5nJDg3ODA2OWM4JDF0AJEoTG9yZy9vcGVuc2VhcmNoL3NxbC9leHByZXNzaW9uL2Z1bmN0aW9uL1NlcmlhbGl6YWJsZUZ1bmN0aW9uO0xvcmcvb3BlbnNlYXJjaC9zcWwvZGF0YS9tb2RlbC9FeHByVmFsdWU7KUxvcmcvb3BlbnNlYXJjaC9zcWwvZGF0YS9tb2RlbC9FeHByVmFsdWU7cQB+AC1xAH4AHX5yAClvcmcub3BlbnNlYXJjaC5zcWwuZGF0YS50eXBlLkV4cHJDb3JlVHlwZQAAAAAAAAAAEgAAeHEAfgAZdAAHSU5URUdFUnNyAC9vcmcub3BlbnNlYXJjaC5zcWwuZXhwcmVzc2lvbi5MaXRlcmFsRXhwcmVzc2lvbkVCLfCMx4IkAgABTAAJZXhwclZhbHVldAApTG9yZy9vcGVuc2VhcmNoL3NxbC9kYXRhL21vZGVsL0V4cHJWYWx1ZTt4cHNyAC5vcmcub3BlbnNlYXJjaC5zcWwuZGF0YS5tb2RlbC5FeHBySW50ZWdlclZhbHVlpmwHmYm3bkMCAAB4cgA1b3JnLm9wZW5zZWFyY2guc3FsLmRhdGEubW9kZWwuQWJzdHJhY3RFeHByTnVtYmVyVmFsdWUb79eD5r/h5AIAAUwABXZhbHVldAASTGphdmEvbGFuZy9OdW1iZXI7eHIAL29yZy5vcGVuc2VhcmNoLnNxbC5kYXRhLm1vZGVsLkFic3RyYWN0RXhwclZhbHVl5d/SeNHJxJQCAAB4cHNyABFqYXZhLmxhbmcuSW50ZWdlchLioKT3gYc4AgABSQAFdmFsdWV4cgAQamF2YS5sYW5nLk51bWJlcoaslR0LlOCLAgAAeHAAAAu4eHNxAH4AHHQAAT5xAH4ACHNxAH4AHwAAAAZ1cQB+ACIAAAABc3EAfgAfAAAABnVxAH4AIgAAAAB2cgBIb3JnLm9wZW5zZWFyY2guc3FsLmV4cHJlc3Npb24ub3BlcmF0b3IucHJlZGljYXRlLkJpbmFyeVByZWRpY2F0ZU9wZXJhdG9yAAAAAAAAAAAAAAB4cHQAPW9yZy9vcGVuc2VhcmNoL3NxbC9leHByZXNzaW9uL2Z1bmN0aW9uL1NlcmlhbGl6YWJsZUJpRnVuY3Rpb25xAH4AKXQAOChMamF2YS9sYW5nL09iamVjdDtMamF2YS9sYW5nL09iamVjdDspTGphdmEvbGFuZy9PYmplY3Q7dABIb3JnL29wZW5zZWFyY2gvc3FsL2V4cHJlc3Npb24vb3BlcmF0b3IvcHJlZGljYXRlL0JpbmFyeVByZWRpY2F0ZU9wZXJhdG9ydAAZbGFtYmRhJGdyZWF0ZXIkYTAxY2U0MzAkMXQAfShMb3JnL29wZW5zZWFyY2gvc3FsL2RhdGEvbW9kZWwvRXhwclZhbHVlO0xvcmcvb3BlbnNlYXJjaC9zcWwvZGF0YS9tb2RlbC9FeHByVmFsdWU7KUxvcmcvb3BlbnNlYXJjaC9zcWwvZGF0YS9tb2RlbC9FeHByVmFsdWU7cQB+AE1xAH4AL3EAfgBJcQB+AClxAH4ASnEAfgAwdAAlbGFtYmRhJG51bGxNaXNzaW5nSGFuZGxpbmckYTUwMDUyODEkMXQAvChMb3JnL29wZW5zZWFyY2gvc3FsL2V4cHJlc3Npb24vZnVuY3Rpb24vU2VyaWFsaXphYmxlQmlGdW5jdGlvbjtMb3JnL29wZW5zZWFyY2gvc3FsL2RhdGEvbW9kZWwvRXhwclZhbHVlO0xvcmcvb3BlbnNlYXJjaC9zcWwvZGF0YS9tb2RlbC9FeHByVmFsdWU7KUxvcmcvb3BlbnNlYXJjaC9zcWwvZGF0YS9tb2RlbC9FeHByVmFsdWU7cQB+AE1xAH4AQX5xAH4AM3QAB0JPT0xFQU4=",
              "lang": "opensearch_query_expression"
            },
            "boost": "1.0"
          }
        }
      ],
      "adjust_pure_negative:true,boost": "1.0"
    }
  },
  "sort": [
    {
      "_doc": {
        "order": "asc"
      }
    }
  ],
  "aggregations": {
    "count(*)": {
      "value_count": {
        "field": "_index"
      }
    }
  }
}


DO the Explain and copy the inner translated json in New Performance Error monitor