Technical Articles
Use SLES 15.2 To Automate SAP HANA High Availability
4 years ago, I wrote a blog explaining how to manually setup and configure SLES 12.2 with SAP HANA to run an automated HA solution from performance-optimized to resource optimized “SAP Hana TDI setup – SAP Hana HSR 3 Tiers with SAPHanaSR”
As a strategic partner of SAP, in its latest revision “15.2”, now SLES provides a fully integrated process to accelerate the deployment of SAP HANA in a highly available environment. Awesome right!!!
In this first article of a series of four, I will talk to you about how SLES HA Automation process and integration work with SAP HANA to facilitate the management of these systems, but also how to get the best of this marriage by leveraging enhanced native SLES tools specifically for SAP platform.
Some references
Before starting, it is important to make sure that you have the right information in regard to the technology, tools, and components used to achieve and realize some of the scenarios that I will cover. Here is a collection of guides, references, and SAP Notes to be reviewed.
SAP Notes
- 2235581 – SAP HANA: Supported Operating Systems
- 2684254 – SAP HANA DB: Recommended OS settings for SLES 15 / SLES for SAP Application 15
- 1999880 – FAQ: SAP HANA System Replication
- 2407186 – How-To Guides & Whitepapers for SAP HANA High Availability
Guides
- SUSE Linux Enterprise Server for SAP Applications 15 SP2
- SAP Hana Administration Guide 2.0 SP5
- SAP HANA System Replication
- FreeNAS® 11.3-U5 User Guide
What and how to articulate it?
I know you want to go straight to the config part of the solution ?, but I have to be diligent with you and at least show some updates from the components used in my previous blog since those updates/upgrades are the essences for the rest of the documentation.
VMware Platform
vSphere 6.0 to 6.7 – vSphere 6.7 contains major improvements which provide such as 2x better performance, 3x reduction in memory usage as well as 3x faster operation relating to VDRS.
FreeNAS Platform
FreeNAS 9.0 to 11.3-U5 – Not just a virtual appliance, the latest version of FreeNAS offer the ability to add virtual machines, Docker, and cloud synchronization support, it includes a mobile-friendly Angular web interface, improved iocage plugin system, real-time APIv2, and synchronization with many more cloud storage providers.
Now that I have talked about the 2 main pieces, let’s dive into SLES For SAP 15 SP2, as I mentioned earlier this revision of SLES provides an embedded mechanism to set up and configure SAP HANA to run in HA mode with auto-failover (scale-up).
I will not cover how to install SLES for SAP step by step neither how to install SAP HANA, but will cover the detail to realize the setup.
Operating System Core Install
By design, SLES 15.2 media gives you the ability to install your OS according to a specific product in order to install the necessary extension and module
The Basesystem Module and Server Application Module are selected de facto, make sure to select SAP Application Module which will auto-select the SLES HA Extension
Your System Role is an important point; however, you can choose only one role.
A very important point, that people don’t think about often, is the NTP server. You need to make sure to sync your server time.
Get your system ready for HA
Because the environment needs to be configured as a cluster, to guaranty the consistency of the following an SBD device needs to be attached on both server parts of the cluster.
This is where I use FreeNAS to create and share an iSCSI bloc to be attached
Now in order to leverage the SLES feature for SAP HANA, specifics package needs to be deployed since they don’t come natively in the core install if you don’t explicitly select them. The yast2-sap-ha for the cluster setup, and the yast2-hana-update to maintain your SAP system within the cluster.
Ready to run the procedure
My DNS is updated with the virtual ip and hostname given for my cluster
Both SAP Hana is installed according to the recommendation, my primary system is backed up
On my primary Hana I have created a key for backup
The PKI SSF data and file has been copied from the primary to the secondary node
- scp -p /usr/sap/HB0/SYS/global/security/rsecssfs/data/SSFS_HB0.DAT hb0adm@hanadb03:/usr/sap/HB0/SYS/global/security/rsecssfs/data/
- scp -p /usr/sap/HB0/SYS/global/security/rsecssfs/key/SSFS_HB0.KEY hb0adm@hanadb03:/usr/sap/HB0/SYS/global/security/rsecssfs/key/
Now from the primary, as root, I invoke “yast2” and select “HA Setup for SAP Products”
Select the “Scale UP: Performance-optimized”
On the communication layer, provide the ip range from the nic card to be used.
Edit both server ips and hostname and also make sure to select “Append to /etc/hosts”, because of the root key exchange between server, the password will be asked
The NTP server configured earlier should appear
It the following screen, I need to add the SBD device attached to the server
How to check the iSCSI? run the command “fdisk -l”
For the Watchdog, I will choose “softdog”
In this last step, I provide the system information such, SID, instance number, VIP, the site name and the behavior for taking over and auto registration
Review the config and hit Install
During the process, some package will be required to be installed
One completed a review of the log to make sure no error encored and click finish
Validate the cluster and SAP HANA replication, to do so I will open the HAWK interface but using the virtual hostname of my cluster on port 7630
At the OS level if I run the “crm status” the status of the resources will show green
Finally, I will register my cluster in my Hana Cockpit
I can see that my hanadb02 node the primary used as per the Cockpit
Test the failover
I now test the failover process by killing the hb0adm process on the hanadb02 (node1) and check the takeover on Hana Cockpit and the OS level (because HAWK is only deployed on node1)
I my primary node (hanadb02) I run my killer command ? “pkill -9 HB0”
On the logical order, after the pkill command is invoked, I run the “crm status” command and I can see that hanadb02 resources are stopped
On the Hana Cockpit, I can see that the replication is broken on Site1
After a few second the takeover is completed and we can see the hanadb03 acting as the master now
And finally, the Hana Cockpit shows the hanadb03 as well the master after refreshing without changing anything
Conclusion
The new embedded process provided by SLES is very good to automate build and task for SAP HANA, very straight forward and efficient if you are not a Linux savvy person to deal with multiple file manipulation and OS command.
In the next portion of this of articles, I will talk about the maintenance part of this process for SAP HANA, Cost Optimized configuration (QA/PRD), and the Chained setup (3 tiers).
Series
Part 1: Current
Part 2: Use SLES 15.2 To Automate SAP HANA HA Maintenance
Awesome, indeed, Williams. Thanks for the post.
Hey Denys, it is a pleasure to share with others
Thank you Williams for this post!
You are very welcome Ronny.
If you are not separating the iscsi device on a separate physical channel. the whole setup doesnt make sense. Separating the fencing device channel is very important for your high availability cluster. Having sbd as the fencing device especially for HANA instances (Since they mostly will not be carrying a FC card) only to complete the architecture you are looking for is a bad idea.
Hello Phani,
Thanks for your input, if you pay attention to my current setup i used only one channel to simplify the process only.
I talk about the feature provided by SLES and about the component to be used, the focus was not the design here 😉
As you know the SLES HA support FC or Iscsi SAN, so your comment is valid from network segregation point of view.
However, i appreciate all the comment as long as they can help others in their reflection.
Thank you.
Iam just pointing out the issue. After seeing the same blunders being done in production setups. Cloud providers are suggesting to use this mechanism without understanding the repercussions of the same.
I have at least 3-4 customers who have used this approach of using the iscsi devices on the same channel as data for the fencing devices. Whenever i have discussions with them on the reason for using the iscsi devices as fencing devices, they point me to Cloud Provider Documents / such blogs .
Hi Williams,
I've followed your intructions but in my case I can't Add any watchdogs to configure because I can't see any wathcdog in the list. However if I execute the comand "lsmod | grep do" I see that softdog is running.
Could you help me, please ?
Thanks in advance
Regards
FSG
Hello Francesc,
Can you share the screen picture of the "watchdog setup" after that you clic on "Add", you must have a list that come up.
Also which version of SLES are you running ?
Williams
Hi,
We've installed SuSe15 SP4 over Intel X386. When I select "Add" I don't have any module to select.
I've the possibility to continue with the installation process but it crash with the next log error after select "Install". I'm sure the problem is that these script try to read the watchdog information and it's find a null instead of character.
2023-04-18 21:15:42 <1> agora25(34603) [Ruby] system/hana.rb(version):103 --- called SapHA::System::HanaClass.version(HPP) ---
2023-04-18 21:15:42 <1> agora25(34603) [Ruby] system/shell_commands.rb(su_exec_outerr_status):92 Executing ["HDB", "version"] as user hppadm
2023-04-18 21:16:24 <1> agora25(34603) [Ruby] wizard/overview_page.rb(block in main_loop):70 --- SapHA::Wizard::ConfigurationOverviewPage.main_loop: event={"EventReason"=>"Activated", "EventSerialNo"=>24, "EventType"=>"WidgetEvent", "ID"=>:next, "WidgetClass"=>:PushButton, "WidgetDebugLabel"=>"Install", "WidgetID"=>:next} ---
2023-04-18 21:16:24 <1> agora25(34603) [Ruby] modules/Misc.rb(SysconfigRead):182 ."/etc/os-release"."VERSION_ID": '15.4'
2023-04-18 21:16:24 <1> agora25(34603) [Ruby] system/hana.rb(version):103 --- called SapHA::System::HanaClass.version(HPP) ---
2023-04-18 21:16:24 <1> agora25(34603) [Ruby] system/shell_commands.rb(su_exec_outerr_status):92 Executing ["HDB", "version"] as user hppadm
2023-04-18 21:16:24 <1> agora25(34603) [Ruby] system/connectivity.rb(run_rpc_servers):222 --- SapHA::System::ConnectivityClass.run_rpc_servers ---
2023-04-18 21:16:24 <1> agora25(34603) [Ruby] system/connectivity.rb(run_rpc_server):72 --- SapHA::System::Host.run_rpc_server ---
2023-04-18 21:16:24 <1> agora25(34603) [Ruby] system/shell_commands.rb(exec_status):43 Executing command ["ssh", "-o", "StrictHostKeyChecking=no", "-f", "root@agora26", "systemd-cat /usr/bin/ruby /usr/share/YaST2/lib/sap_ha/rpc_server.rb"]
2023-04-18 21:16:24 <1> agora25(34603) [Ruby] modules/Progress.rb(New):340 Progress::New(SAP High-Availability Setup, 2, ["Configure local node", "Configure remote node [agora26]"])
2023-04-18 21:16:24 <1> agora25(34603) [Ruby] sap_ha/sap_ha_installation.rb(local_configuration):108 --- SapHA::SAPHAInstallation.local_configuration: configuring current node ---
2023-04-18 21:16:24 <1> agora25(34603) [Ruby] sap_ha/node_logger.rb(method_missing):161 Starting setup process on node agora25
2023-04-18 21:16:24 <1> agora25(34603) [Ruby] sap_ha/sap_ha_installation.rb(block in local_configuration):113 --- SapHA::SAPHAInstallation.local_configuration: configuring NTP Configuration ---
2023-04-18 21:16:24 <1> agora25(34603) [Ruby] modules/Misc.rb(SysconfigRead):182 ."/etc/os-release"."VERSION_ID": '15.4'
2023-04-18 21:16:24 <1> agora25(34603) [Ruby] sap_ha/node_logger.rb(method_missing):161 Appying NTP Configuration
2023-04-18 21:16:24 <1> agora25(34603) [Ruby] sap_ha/sap_ha_installation.rb(block in local_configuration):113 --- SapHA::SAPHAInstallation.local_configuration: configuring Watchdog Setup ---
2023-04-18 21:16:24 <1> agora25(34603) [Ruby] sap_ha/node_logger.rb(method_missing):161 Appying Watchdog Configuration
2023-04-18 21:16:24 <3> agora25(34603) [Ruby] system/watchdog.rb(watchdog?):98 Module '' is not a watchdog!
2023-04-18 21:16:24 <3> agora25(34603) [Ruby] system/watchdog.rb(watchdog?):98 Module '' is not a watchdog!
2023-04-18 21:16:24 <3> agora25(34603) [Ruby] sap_ha/node_logger.rb(method_missing):161 Cannot install a watchdog module
2023-04-18 21:16:24 <1> agora25(34603) [Ruby] system/shell_commands.rb(exec_outerr_status):68 Executing command ["/usr/sbin/modprobe", nil]
2023-04-18 21:16:24 <3> agora25(34603) [Ruby] clients/sap_ha.rb(rescue in run_installation):404 An error occurred during the installation
2023-04-18 21:16:24 <3> agora25(34603) [Ruby] clients/sap_ha.rb(rescue in run_installation):405 no implicit conversion of nil into String
2023-04-18 21:16:24 <3> agora25(34603) [Ruby] clients/sap_ha.rb(rescue in run_installation):406 ["/usr/lib64/ruby/2.5.0/open3.rb:199:in `spawn'", "/usr/lib64/ruby/2.5.0/open3.rb:199:in `popen_run'", "/usr/lib64/ruby/2.5.0/open3.rb:190:in `popen2e'", "/usr/lib64/ruby/2.5.0/open3.rb:349:in `capture2e'", "/usr/share/YaST2/lib/sap_ha/system/shell_commands.rb:69:in `exec_outerr_status'", "/usr/share/YaST2/lib/sap_ha/system/watchdog.rb:126:in `load'", "/usr/share/YaST2/lib/sap_ha/configuration/watchdog.rb:83:in `block in apply'", "/usr/share/YaST2/lib/sap_ha/configuration/watchdog.rb:81:in `each'", "/usr/share/YaST2/lib/sap_ha/configuration/watchdog.rb:81:in `apply'", "/usr/share/YaST2/lib/sap_ha/sap_ha_installation.rb:114:in `block in local_configuration'", "/usr/share/YaST2/lib/sap_ha/sap_ha_installation.rb:112:in `each'", "/usr/share/YaST2/lib/sap_ha/sap_ha_installation.rb:112:in `local_configuration'", "/usr/share/YaST2/lib/sap_ha/sap_ha_installation.rb:44:in `run'", "/usr/share/YaST2/clients/sap_ha.rb:402:in `run_installation'", "/usr/share/YaST2/clients/sap_ha.rb:227:in `block in initialize'", "/usr/lib64/ruby/vendor_ruby/2.5.0/yast/builtins.rb:546:in `eval'", "/usr/share/YaST2/modules/Sequencer.rb:247:in `WS_run'", "/usr/share/YaST2/modules/Sequencer.rb:318:in `block in Run'", "/usr/share/YaST2/modules/Sequencer.rb:310:in `loop'", "/usr/share/YaST2/modules/Sequencer.rb:310:in `Run'", "/usr/share/YaST2/clients/sap_ha.rb:246:in `main'", "/usr/share/YaST2/clients/sap_ha.rb:448:in `<module:Yast>'", "/usr/share/YaST2/clients/sap_ha.rb:41:in `<top (required)>'", "/usr/lib64/ruby/vendor_ruby/2.5.0/yast/wfm.rb:347:in `eval'", "/usr/lib64/ruby/vendor_ruby/2.5.0/yast/wfm.rb:347:in `run_client'", "/usr/lib64/ruby/vendor_ruby/2.5.0/yast/wfm.rb:207:in `call_builtin'", "/usr/lib64/ruby/vendor_ruby/2.5.0/yast/wfm.rb:207:in `call_builtin_wrapper'", "/usr/lib64/ruby/vendor_ruby/2.5.0/yast/wfm.rb:196:in `CallFunction'", "/usr/lib/YaST2/bin/y2start:60:in `<main>'"]
2023-04-18 21:16:24 <3> agora25(34603) [Ruby] clients/sap_ha.rb(rescue in main):254 Error occurred during the unattended installation: no implicit conversion of nil into String
2023-04-18 21:16:24 <1> agora25(34603) [ui] YPushButton.cc(setRole):170 Guessing function key F9 for YPushButton "Stop" at 0x7f7c64045a40 from button role YCancelButton
2023-04-18 21:16:24 <1> agora25(34603) [ui] YPushButton.cc(setFunctionKey):202 Guessing button role YOKButton for YPushButton "OK" at 0x7f7c64034b10 from function key F10
2023-04-18 21:16:34 <1> agora25(34603) [Interpreter] bin/y2start:60 Called YaST client returned.
2023-04-18 21:16:34 <1> agora25(34603) [ncurses] YNCursesUI.cc(~YNCursesUI):121 Stop YNCursesUI
2023-04-18 21:16:34 <1> agora25(34603) [ncurses] NCurses.cc(~NCurses):145 Shutdown NCurses...
2023-04-18 21:16:34 <1> agora25(34603) [ncurses] NCurses.cc(~NCurses):164 NCurses down
2023-04-18 21:16:34 <1> agora25(34603) [Y2Ruby] binary/YRuby.cc(~YRuby):117 Shutting down ruby interpreter.
2023-04-18 21:16:34 <1> agora25(34603) [Y2Perl] YPerl.cc(destroy):164 Shutting down embedded Perl interpreter.
2023-04-18 21:16:34 <1> agora25(34574) [ncurses] NCurses.cc(RedirectToLog):565 isatty(stderr)yes
2023-04-18 21:16:34 <1> agora25(34574) [ncurses] NCurses.cc(RedirectToLog):574 isatty(stdout)yes
2023-04-18 21:16:34 <1> agora25(34574) [Ruby] clients/menu.rb(Launch):307 Got 0 from /sbin/yast sap_ha >&2
Thanks in advance