Technical Articles
How to configure HANA network communication channels – Part2. Internal network
Introduction
Although various materials and documents for HANA networks have been available to ease your implementations and re-configurations, you might have found it time-consuming and experienced a hard time to see a whole picture at a glance.
Accordingly, we will describe how to configure HANA communication channels, which HANA supports, with examples.
This blog contains 2 Parts.
1. Public communication channel configurations
2. Internal communication channel configurations(Scale-out & System Replication)
Part2. Internal communication channel configurations(Scale-out & System Replication)
This blog provides an overview of considerations and recommended configurations in order to manage internal communication channels among scale-out / system replications.
Before we get started, let me define the term of network used in HANA.
- external(public) network : Channels used for external access to SAP HANA functionality by end-user clients, administration clients, application servers, and for data provisioning via SQL or HTTP
- internal network : Channels used for SAP HANA internal communication within the database or, in a distributed scenario, for communication between hosts
For your information, having internal networks under scale-out / system replication is a mandatory configuration in your production sites.
Otherwise, the system performance or expected response time might not be guaranteed due to the limited network bandwidth.
Prerequisites:
* You have installed internal networks in each nodes.
* Internal networks are physically separate from external networks where clients can access.
* The “hostname” in below refers to “internal hostname” in Part1. Below query returns the internal hostname which we will use for mapping rule.
SELECT “HOST” as hostname FROM M_HOST_INFORMATION WHERE KEY = ‘net_hostnames’;
Internal Network Configurations in Scale-out :
There are configurations you can consider changing for internal networks.
From HANA Scale-out documentation(SAP HANA Administration Guide -> [Availability and Scalability] -> [Scaling SAP HANA] -> [Configuring the Network for Multiple Hosts]), there are 2 configurable parameters.
global.ini -> [communication] -> listeninterface : .global or .internal
global.ini -> [internal_hostname_resolution] :
mapping rule : internal_ip_address=hostname
- .global
- Binds the processes to all interfaces.
- This option does not require an internal network address entry.(Default)
- .internal
- Binds the processes to this address only and to all local host interfaces.
- This option requires an internal network address entry.
Above configurations are only required when you have internal networks. Otherwise, please ignore this section.
Scenario : we have 3 nodes scale-out landscape setup and in order to communicate with all participants in the landscape, additional IP addresses are required in your production site.
Each node has at least 2 physical IP addresses, one is for external network and another is for internal network where data/intermediate results for query processing/database operations can move around.
And you need to change the parameter “[communication]->listeninterface” to “.internal” and add internal network entries as followings.
For instance, you have 10.0.1.* as public network and 192.168.1.* as internal network as described below picture.
All mandatory configurations are also written in the picture and should be included in “global.ini”.
mapping rule : internal_ip_address=hostname
Internal Network Configurations in System Replication :
There are also configurations you can consider changing for system replications.
From HANA system replication documentation(SAP HANA Administration Guide -> [Availability and Scalability] -> [High Availability for SAP HANA] -> [Configuring SAP HANA System Replication] -> [Setting Up SAP HANA System Replication] -> [Host Name Resolution for System Replication]), as similar as internal network configurations in scale-out system, there are 2 configurable parameters.
global.ini -> [system_replication_communication] -> listeninterface : .global or .internal
global.ini -> [system_replication_hostname_resolution] :
mapping rule : system_replication_internal_ip_address=hostname
- .global
- if no mappings specified(Default), the default network route is used for system replication communication. This is normally the public network.
- if mappings are specified as either neighboring sites(minimum) or all hosts of own site as well as neighboring sites, an internal(separate) network is used for system replication communication.
- .internal
- need to specify all hosts of own site as well as neighboring sites
- A separate network is used for system replication communication. The primary hosts listen on the dedicated ports of the separate network only, and incoming requests on the public interfaces are rejected.
- 2 tier setup is supported but not for 3 tier
As you recognized, “.internal” setting is a subset of “.global” and “.global” is a default and “.global” supports both 2-tiers and 3-tiers. Therefore, I would highly recommend to stick with the default value “.global” in the parameter “[system_replication_communication]->listeninterface”
IMPORTANT : the parameters in the “global.ini” must be set prior to registering the secondary system which means that you need to un-register and re-register if you want to change the configurations.
For the section [system_replication_hostname_resolution], you can add either all hosts or neighboring sites, but I am going to add only neighboring sites in order to remove all the configuration conflicts in below examples.
mapping rule : system_replication_internal_ip_address=hostname
1. Single node and System Replication(2 tiers)
2. Single node and System Replication(3 tiers)
3. Scale-out and System Replication(2 tiers)
4. Scale-out and System Replication(3 tiers)
Usually, tertiary site is located geographically far away from secondary site. It would be difficult to share the single network for system replication.
Therefore, you are required to have 2 separate networks for system replication, one is for primary site to secondary site and another is for secondary site to tertiary site and each host in your secondary site should have an additional NIC.
Conclusion :
As mentioned earlier, having internal networks are essential in production system in order to get the expected response time and optimize the system performance. Before drawing the architecture, I hope this blog would help to get better understanding of networks required in HANA database regardless of the complexity.
Thanks DongKyun for sharing this through this nice post.
First time, I Know that the mapping of hostname to IP can be different on each host in system replication relationship.
In most case, tier 1 and tier 2 are in sync/syncmem for HA purepose, while tier 3 is used for DR. Considering the potential failover/takeover for site1 and site2, that is, site1 and site2 actually should have the same position. So I think each host, we need maintain two entries for "2. Single node and System Replication(3 tiers)", for example, is that right?
So for s1host1,
10.5.2.1=s2host1
10.4.3.1=s3host1
For s2host1
10.5.1.1=s1host1
10.4.3.1=s3host1
For s3host1
10.4.1.1=s1host1
10.4.2.1=s2host1
Best Regards,
Stephen Sun
Hi Stephen,
Thank you for your valuable question.
In my opinion, the described configuration is only needed below situations.
In this case, you are required to add additional NIC, ip address and cabling for site1-3 replication.
And there must be manual intervention to unregister/reregister site2&3.
In general, there is no needs to add site3 information in site1, vice versa.
For your information, I copy sap note
2386973 - Near Zero Downtime Upgrades for HANA Database 3-tier System Replication
This note well describes the sequence of (un)registering/(re)registering when operating replication and upgrade.
The bottom line is to make site3 always attached to site2 in any cases. So site1 & site3 won't meet except the case that I described.
Hope this explains your question.
Best Regards
DK.Kim
Hi DK.Kim,
Thanks for the further explanation. We are actually considering the following scenarios:
(1) site1 is broken and needs repair;
(2) site2 take over the primary role;
(3) site3 is still registered to the site2 (as it's not impacted, async only as remote DR);
(4) site1 is repaired and joined the replication as secondary (sync to site2, site3 need unregistered from site2 and re-registered to site1).
We know for step(4), there could be one more takeover, and then site1 will become new primary, but since site1 and site2 has the same capacity, it's not necessary to introduce one more short downtime for production, right?
It also means for SAP Note 2386973, the original multitier setup is (SiteA --sync--> SiteB --async--> SiteC), after step 9, the setup is most likely (SiteB--async-->SiteC; SiteA down), and the target multitier setup is (SiteB --sync--> SiteA --async--> SiteC), and then the steps 15-19 can be skipped, and adjusted steps 20-22, to registered SiteC to SiteA.
Are the above scenarios reasonable?
Best Regards,
Stephen
Changed the parameter so that I could connect to HANA using HANA Studio. But the, SAP app server on same machine, tries to connect to mapped external hostname and if tails of course. Any ideas?
Regards
Very helpful. Thanks
HI DongKyun Kim, thanks for explanation .
Do you have similar detailed blog for for Scale up with Redhat cluster. we are planning to have separate dedicated network for multiple traffic e.g. Application, Replication, host management , backup, Heartbeat.
Thanks is advance