Automation SAP Kernel Upgrade maintenance using LVM tool

Introduction

This blog post Simplify system landscape management tool which we can implement in SAP landscape to perform SAP standardize activities in day to day support operations like system restart Kernel upgrade. This SAP tool can be  used as add-on in SAP NetWeaver JAVA application to automate multiple activities

• Automate repetitive, time-consuming system administration tasks ( system restart, Kernel upgrade) and orchestrate to our specific needs using Custom Operations/Hooks option. 
• Centralize landscape management has greater importance and gain landscape-wide visibility to control across infrastructure layers.

• Key functionality does LVM standard edition implements in SAP landscape

• Dashboards and Pods:   Get a quick high-level overview of current landscape state.
• Single/Mass Operations:   Centralize operations (Application, database, restart,    kernel upgrade, validation) for entire landscape using a single console.
• Landscape Visualization:   Visualize systems and underlying infrastructure and identity relationships.
• Custom Operations/Hooks:  Integrated custom scripts to perform kernel upgrade
• Task Scheduling:   Schedule and execute mass or sequential tasks during planned maintenance.

Due to the multiple SAP systems maintenance restart, manually kernel patch upgrade, the following problems faced in the environment

• Many resource required to work during Monthly/Quarterly Maintenance activity on weekends to complete kernel upgrade and other maintenance activity which required system restart for SAP instances.
• Huge administrative tasks handled by the resources for multiple systems restart during schedule maintenance activity or on demand activity
• Huge effort and resource time to perform kernel upgrade manually on each application server
• Due to downtime restriction adding multiple systems in maintenance window and perform kernel upgrade not possible
• Resource needs to login in each system to perform restart, Kernel upgrade and validation manually

Install LVM standard edition tool add-on which is available to all SAP customers for free of cost in  service market place, Although this LVM tool (LaMa tool) does not has any feature to automate kernel upgrade, to overcome this we  can create custom scripts at OS level and integrated with  LVM tool using Custom Operations/Hooks so that kernel upgrade can also be automate.

Below are the step by step configuration  that I did to accomplish the for-

Application restart with cleanipc command

Database restart with listener

Kernel backup/upgrade using custom operations scripts

Technical Prerequisites

Below Minimum Versions required to use LVM tool Custom scripts

SAP Hostagent 25

SAP NetWeaver 7.4 JAVA application

SAP LVM add-on VCM LVM and VCM LVM CR 2.0 SP10

Note – Please check latest version available in service market place and if possible use latest versions.

Configuration

Install LVM Add-on and hostagents.  

• Download VCM LVM and VCM LVM CR add-ons tools from service market place https://launchpad.support.sap.com/#/softwarecenter/ and deploy on SAP NetWeaver 7.4 JAVA  using TELNET or SUM

To be able to start and schedule own scripts  in LVM (LaMa Tool), many tasks need to be perform in sequence.

Adding System in tool 

• Login to LVM  tool for adding systems http://<host>:<port>/lvm.
• Go to Configuration — Systems– click to add ( determine) for adding system

• Select  Discover using host and instance under source
• Provide host name , SAPADM, <sid>adm and oracle user for database

• Click next to complete the system addition
• Go to Operations  to check the system status which added

• This standard tool has only option to restart system but we integrated custom UNIX shell scripts to perform, cleanipc command, kernel upgrade, backup of exiting kernel executable directories,  listener restart.

Creation of custom script files at OS level

Name: Script name

User name : User which has full authorization to execute script, in my case its sapadm

Command: Command which will execute to perform task below is example for kernel backup and upgrade

• To create custom scripts go to manage host which added in this tool
• Go to /usr/sap/hostctrl/exe/operations.d
• In this file, location of the real script is configured. If possible there should be only one location for all scripts on an nfs-share. Configuration file must contain following lines and should look like
• Here I created two scripts  LVM_backup_kernel.conf and LVM_Kernel_Upgrade.conf

• LVM_backup_kernel.conf – This scripts has command to kill all processes running from sidadm , run cleanipc command and take backup of kernel directories for all instances running on particular host.
• LVM_Kernel_Upgrade.conf – This scripts has path of kernel SAR files from where it will be extracted to Kernel directory in to each instance

Add Custom operations and register scripts in LVM Tool

• Go to LVM tool — SETUP ( top on right)– Custom Operations
• Give Custom operation name and select appropriate option
• Register custom scripts created at OS level
• Click on “provider implementation definition” tab
• Provide Name, Select “script registered with host agent
• Jump to “Get registered scripts from host”  check required options
• Select host which have added in this tool and on which custom scripts created
• Click on “Get registered scripts”
• Select “Registered script” to add script in my case I added “LVM_backup_kernel” which was created.
• Save

Follow the same for other script LVM_kernel_upgrade to register.

Schedule Kernel upgrade, application, database restart, activity  from LVM tool

Scheduling

• Go to Automation tab Schedule kernel upgrade, click on ADD
•  Select options as per your requirement, below I select sequential operations for stopping application, database, backup of kernel directories on all instances, kernel upgrade, start database and application validation if needed.
• Provide date and time when you wants to perform activity
• Provide email ID for notification on successfully /failed  of activity.
• Below automatic execution sequence is scheduled for the system along with kernel upgrade
• While scheduling task, select dependency to perform start and stop sequence in case multiple application servers
• Below task scheduled  in sequence:-
•      Stop Central Instance
•      Stop Central service
•      Stop Database
•      Execute custom script om central instance host for kernel directory backup
•      Execute Kernel Upgrade
•      Start Database, Central service and Central instance then
• Check the status of automatic execution  under “monitoring” tab and logs
• Post execution notification on Kernel Upgrade status

In this blog post we are able to automate kernel upgrade along with application and database restart, we can add more custom scripts to automate many other activities on LVM (LaMA) Standard edition version.

• Custom Operations based on SAP Profile Change
• Custom Operations based on HANA Profile Change
• Enables to perform mass – operations on the entire pool of systems (stop/(re)start etc.
• Define a target state (e.g. not running) for any instance (e.g. App Server).
• Enable to execute Operating System Update and Hostagent patching
• Enable Application validation

Currently LVM has two Editions, I used Standard Edition which is available for all  SAP customers for free,  For Enterprise Edition Customer has to pay License fee.