Introduction
We will go through step by step hands-on scenario to build virtual appliance for SAP HANA on AWS in this hands-on. It includes how to install SAP HANA on AWS and we will go through the steps required to install and configure SAP HANA on AWS. I hope the information contained in this hands-on provide you basic idea about what AWS services are and how to configure SAP HANA on AWS environment.
The hands-on is composed of following posts:
- Prior knowledge
- EC2 instance creation
- EC2 instance configuration
- SAP HANA installation for the master
- Adding worker to SAP HANA landscape.
This post handles the third part of the hands-on.
We will try most steps in command line (awscli) but you should know that all tasks in this hands-on could be done on AWS console too.
3. EC2 instance (host) setup
It’s not the end to create EC2 instance. We have to install prerequisite packages and configure EC2 instance. All tasks depend on the OS type and release (that is, AMI). In this hands-on, we use SLES 12SP3.
3.1. Connecting to EC2 instance
Everything is ready and you can connect to EC2 instance if everything has gone fine so far.
3.1.1. Connecting from Linux
You remember that we kept keypair file somewhere that was created in "2.7 Create a keypair" in
https//blogs.sap.com/2018/07/04/hands-on-configure-sap-hana-on-aws-part2/. The keypair file is always required whenever you connect to EC2 instance.
yourLinux:~ # ssh -i KeyPair.pem ec2-user@52.78.39.220
Last login: Thu Jun 21 09:27:58 2018 from 220.85.106.67
SUSE Linux Enterprise Server 12 SP3 x86_64 (64-bit)
As "root" (sudo or sudo -i) use the:
- zypper command for package management
- yast command for configuration management
Management and Config: https://www.suse.com/suse-in-the-cloud-basics
Documentation: https://www.suse.com/documentation/sles-12/
Forum: https://forums.suse.com/forumdisplay.php?93-SUSE-Public-Cloud
Have a lot of fun...
ec2-user@ip-172-31-128-21:~> sudo su -
ip-172-31-128-21:~ #
Memo.
As mentioned earlier, the full path of key pair PEM should be specified in “-i" option if the file is not located in the current working directory.
In case of SLES (Suse Linux Enterprise Server), you can use root or ec2-user for EC2 connection. Normally we logon by ec2-user and switch user to root after logon.
3.1.2. Connecting from Windows
To connect from PuTTY in window, we need two tasks:
- Extract a private key (PPK, PuTTy Private Key) from PEM file.
We downloaded the PEM file at "2.7 Create a key pair" in https//blogs.sap.com/2018/07/04/hands-on-configure-sap-hana-on-aws-part2/
Run PuTTYgen from windows command line.
Press [Load] button and specify keypair PEM file path. At this time, use “*.pem” in File Name box because it displays PPK files only by default.
Press [Save private key] button and save private key.
- Connect to EC2 instance via PuTTy.
In PuTTY, specify the PPK file name in "Connection > SSH > Auth".
Open the connection.
Refer to
https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/putty.html for the details.
3.2. Change hostname
The first task after you connect to EC2 instance is updating the host name along with the domain name according to your requirement, you need to update /etc/hosts and /etc/HOSTNAME accordingly to persist the update across system reboot. Also, you need to check if “Set Hostname via DHCP” is set to “no” in YaST (Yet Another Setup Tool). This should be set to “no”.
You are on EC2 instance now.
ip-172-31-128-21:~ # hostname imdbmaster
ip-172-31-128-21:~ # echo "imdbmaster" > /etc/HOSTNAME
ip-172-31-128-21:~ # cp /etc/hosts /etc/hosts.bak
ip-172-31-128-21:~ # echo "172.31.128.21 imdbmaster imdbmaster.local" >> /etc/hosts
ip-172-31-128-21:~ # sed -i '/preserve_hostname/ c\preserve_hostname: true' /etc/cloud/cloud.cfg
ip-172-31-128-21:~ # cp /etc/defaultdomain /etc/defaultdomain.bak
ip-172-31-128-21:~ # echo "local" >> /etc/defaultdomain
Memo.
In some SLELS release, /etc/defaultdomain is not required to set as above. I specify the command here in case you meet an error “default domain is not set” in some SLES release while starting autofs service.
Before going to the next steps, stop/start the EC2 instance.
yourLinux:~ # aws ec2 stop-instances --instance-ids i-0a8eb78dbab3d9353
yourLinux:~ # # You need to wait till the instance is successfully stopped
yourLinux:~ # aws ec2 start-instances --instance-ids i-0a8eb78dbab3d9353
Memo.
There is no explicit reason for stopping/starting EC2 instance. I was in the situation that package repository was not enabled and not all repositories were visible. This symptom was resolved after restarting newly created EC2 instance.
3.3. Install prerequisite software packages
You need to install a few prerequisite packages on EC2 instance you just created. The installation task is different for each OS type and release because some rpms or packages are already installed depending on OS type and release.
In this hands-on, we use SLES 12SP3 (ami-e22b898c for suse-sles-12-sp3-v20180215-hvm-ssd-x86). So, we will run a sequence of commands as follows:
You are on EC2 instance now.
imdbmaster:~ # zypper -n install systemd
imdbmaster:~ # zypper -n install gtk2
imdbmaster:~ # zypper -n install java-1_6_0-ibm
imdbmaster:~ # zypper -n install libicu
imdbmaster:~ # zypper -n install mozilla-xulrunner*
imdbmaster:~ # zypper -n install ntp
imdbmaster:~ # zypper -n install sudo
imdbmaster:~ # zypper -n install syslog-ng
imdbmaster:~ # zypper -n install tcsh libssh2-1
imdbmaster:~ # zypper -n install autoyast2-installation
imdbmaster:~ # zypper -n install yast2-ncurses
imdbmaster:~ # zypper -n install cpupower
imdbmaster:~ # zypper -n install libopenssl0_9_8
imdbmaster:~ # zypper -n install libgcc_s1
imdbmaster:~ # zypper -n install libstdc++6
imdbmaster:~ # zypper -n install unrar
imdbmaster:~ # zypper -n install libnuma-devel
imdbmaster:~ # zypper -n remove ulimit
Memo.
You can ignore below warnings in some package installations:
- “already installed, nothing to do”
- “not found in package names. Trying capabilities”
3.4. EC2 instance (Host) configuration
There need some host configuration tasks.
You are on EC2 instance now.
Followings are used for adding and updating the kernel parameter:
imdbmaster:~ # cp /etc/sysctl.conf /etc/sysctl.conf.bak
imdbmaster:~ # echo "net.ipv4.tcp_slow_start_after_idle=0" >> /etc/sysctl.conf
imdbmaster:~ # echo "kernel.shmmni=65536" >> /etc/sysctl.conf
imdbmaster:~ # sysctl -p /etc/sysctl.conf
Following two lines are for disabling DHCP from setting hostname:
imdbmaster:~ # sed -i '/DHCLIENT_SET_HOSTNAME/ c\DHCLIENT_SET_HOSTNAME="no"' /etc/sysconfig/network/dhcp
imdbmaster:~ # service network restart
Following lines are for configuring NTP, AUTOFS and boot LVM services:
imdbmaster:~ # cp /etc/ntp.conf /etc/ntp.conf.bak
imdbmaster:~ # echo "server 0.pool.ntp.org" >> /etc/ntp.conf
imdbmaster:~ # echo "server 1.pool.ntp.org" >> /etc/ntp.conf
imdbmaster:~ # echo "server 2.pool.ntp.org" >> /etc/ntp.conf
imdbmaster:~ # echo "server 3.pool.ntp.org" >> /etc/ntp.conf
imdbmaster:~ # chkconfig ntpd on
imdbmaster:~ # chkconfig autofs on
imdbmaster:~ # chkconfig boot.lvm on
imdbmaster:~ # chkconfig kdump off
imdbmaster:~ # service ntpd start
imdbmaster:~ # service autofs start
Memo.
In my case, “chkconfig boot.lvm on” always failed with “unknown service” error.
Instead of the command, I checked if use_lvmetad is set to 1 from /etc/lvm/lvm.conf file.
That is, “use_lvmetad = 1” should be set in /etc/lvm/lvm.conf file so lvm be activated on reboot.
Memo.
Once “service autofs start” is done, you need to check the status by “service autofs status”. You may happen to meet an message, " automount[18498]: lookup_init:136: lookup(yp): map auto.direct: Local domain name not set". Then, You can try two actions in the master and the worker:
- # echo "local" >> /etc/defaultdomain
- Stop and start EC2 instances.
Following lines are for setting clocksource to TSC upon reboot:
imdbmaster:~ # cp /etc/init.d/boot.local /etc/init.d/boot.local.bak
imdbmaster:~ # echo "tsc" > /sys/devices/system/clocksource/*/current_clocksource
imdbmaster:~ # echo "echo tsc > /sys/devices/system/clocksource/*/current_clocksource" >> /etc/init.d/boot.local
Following commands are from SAP Note 2205917 and SAP Note 2292711:
imdbmaster:~ # echo never > /sys/kernel/mm/transparent_hugepage/enabled
imdbmaster:~ # echo 10 > /proc/sys/vm/swappiness
imdbmaster:~ # echo 0 > /sys/kernel/mm/ksm/run
imdbmaster:~ # echo 0 > /proc/sys/kernel/numa_balancing
imdbmaster:~ # echo "#BEGIN: This section inserted by AWS SAP HANA Quickstart" >> /etc/init.d/boot.local
imdbmaster:~ # echo "echo never > /sys/kernel/mm/transparent_hugepage/enabled" >> /etc/init.d/boot.local
imdbmaster:~ # echo "echo 10 > /proc/sys/vm/swappiness" >> /etc/init.d/boot.local
imdbmaster:~ # echo "echo 0 > /sys/kernel/mm/ksm/run" >> /etc/init.d/boot.local
imdbmaster:~ # echo "echo 0 > /proc/sys/kernel/numa_balancing" >> /etc/init.d/boot.local
You can look at below two SAP NOTEs for further reference:
- SAP Note 2205917 - SAP HANA DB: Recommended OS settings for SLES 12 / SLES for SAP Applications 12
- SAP Note 2292711 - SAP HANA DB: Recommended OS settings for SLES 12 SP1 / SLES for SAP Applications 12 SP1
Refer to below link for further information regarding Linux Operating System configuration:
https://wiki.scn.sap.com/wiki/display/SAPHANA/Linux+Operating+System+with+SAP+HANA+Reference+Guide
3.5. Volume creation
You can check the attached block devices (EBSes) by “lsblk” command. EBSes will be shown as below:
imdbmaster:~ # lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
xvda 202:0 0 100G 0 disk
└─xvda1 202:1 0 100G 0 part /
xvdf 202:80 0 667G 0 disk
xvdg 202:96 0 667G 0 disk
xvdh 202:112 0 667G 0 disk
xvdi 202:128 0 667G 0 disk
xvdj 202:144 0 50G 0 disk
xvdk 202:160 0 30G 0 disk
Names are a bit different than we specified during "2.5. Prepare block device mappings" in
https://blogs.sap.com/2018/07/04/hands-on-configure-sap-hana-on-aws-part2/
Refer to
https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/device_naming.html for block device naming.
These EBSes are used for a couple of file systems. There are two kinds of EBS usage patterns in this hands-on. One is using volume group, the other is on single block device. Thus, the command sets required for file system creation are two kinds.
Some EBSes are combined into a volume group. It is for HANA database related file systems, /hana/data, /hana/log, /hana/shared and /backup file systems.
We’ll create volume group with four physical volumes. For this, we execute following commands sequence in EC2 instance:
- Create physical volumes from EBSes.
imdbmaster:~ # pvcreate /dev/xvdf /dev/xvdg /dev/xvdh /dev/xvdi
- Set the I/O scheduling option to NOOP for the physical volumes.
imdbmaster:~ # echo "noop" > /sys/block/xvdf/queue/scheduler
imdbmaster:~ # echo "noop" > /sys/block/xvdg/queue/scheduler
imdbmaster:~ # echo "noop" > /sys/block/xvdh/queue/scheduler
imdbmaster:~ # echo "noop" > /sys/block/xvdi/queue/scheduler
Memo.
NOOP (noop) is the simplest I/O scheduler for the Linux kernel based upon the FIFO queue concept. The NOOP scheduler inserts all incoming I/O requests into a simple FIFO queue and implements request merging. The scheduler assumes I/O performance optimization will be handled at some other layer of the I/O hierarchy.
This file can also contain the string “none” meaning that I/O scheduling does not happen for this device. This is usually because the device uses multi-queue queuing mechanism. In this case above commands will not work.
(https://doc.opensuse.org/documentation/leap/tuning/html/book.sle.tuning/cha.tuning.io.html#cha.tunin...)
imdbmaster:~ # vgcreate vghana /dev/xvdf /dev/xvdg /dev/xvdh /dev/xvdi
imdbmaster:~ # lvcreate -n lvhanalog -i 4 -I 256 -L 200G vghana
imdbmaster:~ # lvcreate -n lvhanadata -i 4 -I 256 -L 800G vghana
imdbmaster:~ # lvcreate -n lvhanashared -i 4 -I 256 -L 200G vghana
imdbmaster:~ # lvcreate -n lvhanaback -i 4 -I 256 -L 1200G vghana
imdbmaster:~ # mkfs.xfs /dev/mapper/vghana-lvhanalog
imdbmaster:~ # mkfs.xfs /dev/mapper/vghana-lvhanadata
imdbmaster:~ # mkfs.xfs /dev/mapper/vghana-lvhanashared
imdbmaster:~ # mkfs.xfs /dev/mapper/vghana-lvhanaback
- Create directories for file system mount points.
imdbmaster:~ # mkdir /hana /hana/data /hana/log /hana/shared /backup
Some EBSes are used in single block device. It is for /usr/sap and /media file systems. “/usr/sap” is for SAP HANA executables and libraries and it includes links to directories in “/hana/shared” file system. “/media” is for downloading SAP HANA installation image files.
We’ll create each file system using a single block device.
For these file systems, we execute following commands:
imdbmaster:~ # mkfs.xfs -f /dev/xvdj
imdbmaster:~ # mkfs.xfs -f /dev/xvdk
- Create directories for file system mount points
imdbmaster:~ # mkdir /usr/sap /media
/etc/fstab file needs to be updated to make new file systems mounted across EC2 instance (re)start. We need to add below lines in /etc/fstab:
/dev/xvdj /usr/sap xfs nobarrier,noatime,nodiratime,logbsize=256k 0 0
/dev/xvdk /media xfs nobarrier,noatime,nodiratime,logbsize=256k 0 0
/dev/mapper/vghana-lvhanadata /hana/data xfs nobarrier,noatime,nodiratime,logbsize=256k 0 0
/dev/mapper/vghana-lvhanalog /hana/log xfs nobarrier,noatime,nodiratime,logbsize=256k 0 0
/dev/mapper/vghana-lvhanashared /hana/shared xfs nobarrier,noatime,nodiratime,logbsize=256k 0 0
/dev/mapper/vghana-lvhanaback /backup xfs nobarrier,noatime,nodiratime,logbsize=256k 0 0
Memo.
In case of, SLES11SP4, /etc/fstab should be changed as below (delaylog option is added):
/dev/xvdj /usr/sap xfs nobarrier,noatime,nodiratime,logbsize=256k,delaylog 0 0
/dev/xvdk /media xfs nobarrier,noatime,nodiratime,logbsize=256k,delaylog 0 0
/dev/mapper/vghana-lvhanadata /hana/data xfs nobarrier,noatime,nodiratime,logbsize=256k,delaylog 0 0
/dev/mapper/vghana-lvhanalog /hana/log xfs nobarrier,noatime,nodiratime,logbsize=256k,delaylog
/dev/mapper/vghana-lvhanashared /hana/shared xfs nobarrier,noatime,nodiratime,logbsize=256k,delaylog 0 0
/dev/mapper/vghana-lvhanaback /backup xfs nobarrier,noatime,nodiratime,logbsize=256k,delaylog 0 0
“mount -a” will mount all file systems listed in /etc/fstab file. You can check if all file systems are shown in “df -h" command with correct sizes.
imdbmaster:~ # mount -a
imdbmaster:~ # df -h
Filesystem Size Used Avail Use% Mounted on
devtmpfs 30G 8.0K 30G 1% /dev
tmpfs 30G 0 30G 0% /dev/shm
tmpfs 30G 23M 30G 1% /run
tmpfs 30G 0 30G 0% /sys/fs/cgroup
/dev/xvda1 9.8G 1.7G 7.6G 18% /
tmpfs 6.0G 0 6.0G 0% /run/user/1000
/dev/xvdj 50G 33M 50G 1% /usr/sap
/dev/xvdk 30G 33M 30G 1% /media
/dev/mapper/vghana-lvhanadata 800G 34M 800G 1% /hana/data
/dev/mapper/vghana-lvhanalog 200G 33M 200G 1% /hana/log
/dev/mapper/vghana-lvhanashared 200G 33M 200G 1% /hana/shared
/dev/mapper/vghana-lvhanaback 1.2T 34M 1.2T 1% /backup
We’ve done in “EC2 instance configuration”. It's time to install SAP HANA on EC2 instance (
https://blogs.sap.com/2018/07/04/hands-on-configure-sap-hana-on-aws-part4/).
