Securing the SAP Instance Agent / SAP Start Servic...

JoeGoerlich · ‎02-22-2021

SAP Instance Agent, also known as SAP Start Service or SAPStartSrv, offers web methods as SOAP Web Services to control a SAP instance. By default some of these web methods can be executed without prior authentication.
This means every client with access to the ports of SAPStartSrv may be able to call several web methods without authentication! This may have impact on the availability, integrity of the system and could also be utilized by adversaries to gather information about a target system.

The latest blogpost i could find on the topic of protected webmethods was written by Benjamin Nenninger in 2018: https://blogs.sap.com/2018/10/24/protected-web-methods.

In the meantime almost all SAP products (SAP NW AS Java seems to be a special case here, see below) leveraging any of the web methods of SAPStartSrv have been updated and there should be no need to allow unauthenticated access for them any longer.
In other words in standard scenarios there should be no need for unprotected web methods anymore and the parameter service/protectedwebmethods = ALL could be set without any side-effects. But now lets go into details.....

I recommend to also take a look at my blogpost Protecting web methods offered by SAP Host Agent.

Updates:

2022-07-25: Added information about a bug when reading the System Logs in SM21
2021-06-17: Added section for SAP Landscape Management 3.0.
2021-05-11: Adjusted the title to a more bold one.
2021-04-30: Added infos regarding the web methods used by the replacement of the Flash-based "System Overview" of nwa.
2021-04-21: Added details regarding the Log Viewer of SAP NetWeaver AS Java
2021-04-16: Added section for Azure Monitor for SAP Solutions - SAP NetWeaver provider (AMS-NW) and SUSE SAP Host Exporter.
2021-03-17: Added section for encrypted communication. Rephrased some sections to be more precise.

SAP Instance Agent / SAP Start Service / SAPStartSrv

In the following i will do the question and answer game (as you may recognise from other blogposts of myself) to develop a basic understanding of the SAP Instance Agent and its web methods.

Where do we find the SAP Instance Agent?

Typically we can find it on SAP NetWeaver AS ABAP or AS Java, SAP WebDispatcher, SAP HANA DB, SAP Trex, Diagnostics Agent, but also on other SAP products.

SAP Instance Agent also known as SAPStartSrv reads its parameters from the the instance profile or the DEFAULT.PFL. When no values are found it falls back to the Kernel default values.

Which ports are used by SAPStartSrv?

By default it binds 5$$13 (HTTP) and 5$$14 (HTTPS), where $$ is the instance number of the instance the SAP Instance Agent belongs to.

On which ip addresses are these ports accessible?

By default sapstartsrv binds its ports on all available NICs (indicated by 0.0.0.0):

~> sudo ss -tlpn | grep -e 5..1[34]

LISTEN  0   20  0.0.0.0:50013    0.0.0.0:*  users:(("sapstatsrv",pid=13585,fd=10))

LISTEN  0   20  0.0.0.0:50014    0.0.0.0:*  users:(("sapstatsrv",pid=13585,fd=9))

LISTEN  0   20  0.0.0.0:50113    0.0.0.0:*  users:(("sapstatsrv",pid=2118,fd=12))

LISTEN  0   20  0.0.0.0:50114    0.0.0.0:*  users:(("sapstatsrv",pid=2118,fd=9))

This could be adjusted by parameters 'service/hostname', 'service/http/hostname', 'service/https/hostname'.

For example 'service/hostname' and 'service/http/hostname could be set to 127.0.0.1 and 'service/https/hostname' could be set to $(SAPLOCALHOST) to reduce the attack surface.

~> sudo ss -tlpn | grep -e 5..1[34]

LISTEN  0   20  127.0.0.1:50013    0.0.0.0:*  users:(("sapstatsrv",pid=13585,fd=10))

LISTEN  0   20  192.168.75.135:50014    0.0.0.0:*  users:(("sapstatsrv",pid=13585,fd=9))

LISTEN  0   20  127.0.0.1:50113    0.0.0.0:*  users:(("sapstatsrv",pid=2118,fd=12))

LISTEN  0   20  192.168.75.135:50114    0.0.0.0:*  users:(("sapstatsrv",pid=2118,fd=9))

Please note: At time of writing SAP NetWeaver AS Java does not support this security measure as there are on one side various components of the nwa sending requests to web methods of the SAPStartSrv by addressing the $(SAPLOCALHOST)-IP-address instead of localhost. And on the other side a secure communication channel between AS JAVA and sapstartsrv using HTTPS is not supported at all.

Please note: I discovered a bug, which makes it impossible to read the syslog of other application servers in SM21 when the sapstartsrv is not listening on the loopback adapter (localhost, 127.0.0.1, ::1). SAP will release a fix for this in SAP note 3222183.

Can access to these ports be secured by any ACLs?

Access to these ports can be controlled individually by an ACL defined in parameter 'service/http/acl_file' for http and 'service/https/acl_file' for https.

This should be considered if for example no proper network separation is in place.

What about encrypted communication?

As mentioned above SAPStartSrv is able to allocate an HTTPS port. Therefore the relevant certificates have to be present in the SSL Server PSE (SAPSSLS.pse).

As of SAP Kernel 7.42 SAPStartSrv automatically initializes a systemPKI. If a valid SAPSSLS.pse is additionally available SAPStartSrv will use this as the preferred one in addition to the systemPKI. It will then decide using SNI which certificate is to be presented during the handshake.

Please note: This is independent from the parameter 'system/secure_communication'.

How to configure the protocols and cryptographic algorithms?

When it comes to encryption SAPStartSrv relays on the CommonCryptoLib (CCL).

For incoming connections the protocol (TLS 1.0, TLS 1.1, TLS 1.2) and the available cryptographic algorithms can be adjusted using the profile parameter 'ssl/ciphersuites'.

For outgoing connections it is the profile parameter 'ssl/client_ciphersuites'.

Some of the tools coming with SAPStartSrv do not read the DEFAULT or instance profiles. Instead the environment variable 'CCL_PROFILE' should be set on OS level for the user <sapsid>adm pointing to the profile holding the CCL's configuration (e.g., the DEFAULT.PFL).

Please note: This can not be done via the SETENV_<xx>-method in the DEFAULT or instance profiles. Environment variables set by SETENV_<xx> would be valid for child processes of sapstartsrv only.

Details about how to determine suitable values for those parameters can be found at my blogpost CommonCryptoLib: TLS protocol versions and cipher suites.

Which clients are accessing these ports?

Typical clients are SAP MMC, SAP MC, sapcontrol, saphostctr. But there may be also custom developed scripts or 3rd party tools, e.g., for monitoring purposes or start/stop of systems. For testing or troubleshooting also postman or SoapUI may be used as a client.

What is served on these ports?

SAPStartSrv offers web methods as SOAP Web Services according to the so called Porttypes configured in profile parameter 'service/porttypes'.

A WSDL is available at https://<hostname>:5$$14/?wsdl

Which web methods can be accessed without authentication?

This depends on the setting of profile parameter 'service/protectedwebmethods'.

SAP has grouped some of the web methods in 'protection groups' (not an official term). Parameter 'service/protectedwebmethods' allows to specify one of the following groups (from least to most protective):

1. 'NONE'

2. 'DEFAULT'

3. 'SDEFAULT'

4. 'ALL'

In addition to that we can extend or reduce these predefined groups. We can add protection for web methods by appending +<name-of-webmethod> or remove protection by appending -<name-of-webmethod>.

To determine the protected web methods we can do the following:

1. Set the parameter to the desired value.

2. Restart the Instance Agent for the changes to take effect (!):
sapcontrol -nr <instance number> -function RestartService

5. Compare both lists.

For the settings 'NONE' and 'ALL' the compare will not work, because the output for protected methods will either be empty or the string 'ALL'.

With sapcontol 7.53 PL700 for example we get a list of 169 web methods in total, where the setting SDEFAULT will protect 154 and DEFAULT protect 87 of them.

Which is the default value for 'service/protectedwebmethods'?

The profile parameters Kernel default value is as of SAP Kernel 7.49 'SDEFAULT'.

Which web methods are not protected with setting SDEFAULT?

With SAPStartSrv 7.53 PL700 the following web methods are not protected with setting SDEFAULT:

AccessCheck	GetSecNetworkId	ListConfigurationFiles
AnalyseLogFile	GetSystemInstanceList	UpdateIntancePSE
GetInstanceProperties	J2EEGetTaskStack	WebDispGetUrlPrefixList
GetProcessList	J2EEGetVMCHistory
GetProcessList2	J2EEGetVMCHistory2

Please note: Some of them could be abused for information gathering.

Which authentication methods are supported in general by SAPStartSrv?

OS level authentication using Unix domain sockets or Windows named pipes,

Local Logon ticket (requested by web method RequestLogonFile),

Username and password for OS users (make sure to always use https to prevent sending credentials over an unencrypted channel),

Client certificate (X.509).

Which users are allowed to authenticate?

SAPStartSrv has no own user store. The authentication relies on users configured for access.

The user <sapsid>adm running the SAPStartSrv proccess is always allowed to authenticate.

Additional OS users may be defined by profile parameter 'service/admin_users'

OS user groups may also be defined by profile parameter 'service/admin_groups'.

Besides authentication with OS users it is also possible to allow additional users to authenticate with X.509 client certificates. Therefore, their certificates' DN has to be configured in profile parameter 'service/sso_admin_user_<xx>'.

Please note: This parameter also supports wildcards '?' or '*', which have to be used carefully.

What about authorizations?

This is a very valuable question. And this question may undermine the concept of protected web methods to a certain extend!

SAPStartSrv doesn't have an own user store, nor does it provide any concept to handle authorizations! We will see the possible impact in a few.

How to achieve protecting all web methods without any side-effects?

Simply by making sure every client is properly authenticated.

Why do we may not want to protect all web methods?

As stated before there is no authorization concept involved. Meaning an authenticated client will be able to call any available web method.

There may be scenarios involving clients for which we have to follow a risk based approach.

For example, there may be a 3rd party monitoring tool which should be allowed to check certain metrics only, e.g., using a web method providing read only access.
In this case we may not want this 3rd party application to be authenticated and implicit be able to also execute web methods allowing to exfiltrate (log) data, stop the system or to even make any changes.

For additional risk mitigation we should restrict access to the ports of the SAP Instance Agent to the necessary clients only.

Why should we still start with service/protectedwebmethods=ALL?

In any case we should start with setting parameter service/protectedwebmethods = ALL.

For standard scenarios there shouldn't exist any blocking points to do so.

For custom developed or 3rd party tools we should analyse if these clients should indeed have access to all web methods. If not, we have to make sure they communicate without prior authentication.

Then we have to exclude the relevant web method(s) from protection by appending them prefixed with "-" to the actual value, e.g., service/protectedwebmethods = ALL -GetAlerts.

Following this approach we have full transparency about web methods which are allowed to be called unauthenticated.

Please note: In such special scenarios we should strongly consider to implement the ACL(s) to protect access to SAPStartSrv in addition to proper network separation.

Authentication for common clients

sapcontrol

Sapcontrol by default will use OS level authentication.

If started on the same with user <sapsid>adm no additional credentials are needed.

sapcontrol may also be started with -user "<username>" "password" or -queryuser to authenticate with any user listed in profile parameter `service/admin_users` and present on OS level.

Please note: The OS user must have set a password. In SSH-key only environments this may not always be the case.

sapcontrol may be started with -prot NI_HTTPS to authenticate with an X.509 client certificate stored in the SAPSSLC.pse. This pse must be trusted by importing the trustchain (or self-signed certificate) into it.

Recent versions of SAPStartSrv will create a so called systemPKI (this is independent from the setting of profile parameter `system/secure_communiation`). SAPStartSrv allows to use this for X.509 based authentication. Even if SAPStartSrv uses by default an existing SAPSSLC.pse, the systemPKI (stored in the sap_system_pki_instance.pse) may be leveraged for authentication. To make sapcontrol use the systemPKI it has to be started as <sapsid>adm with -systempki <profile> (the correct pse will be addressed using SNI).
This can also be used to authenticate at a SAPStartSrv running on a remote host of the same SAP NW AS ABAP system (belonging to the same systemPKI).

sapcontrol on Windows

Sapcontrol on windows may also be started with -prot WINHTTPS. This would lead to sending a suitable client certificate - if present in the windows certificate store.

The certificates DN must be maintained in parameter service/sso_admin_user_<xx> to allow logon.

SAP MMC

SAP MMC supports X.509 authentication if we activate HTTPS in the "Security" properties of the MMC configuration.
It will send a suitable certificate (depending on the issuer) if present in the windows certificate store.
The certificates DN must be maintained in parameter service/sso_admin_user_<xx>.

SAP Host agent (SAPHostCtrl)

The SAP Host Agent may proxy requests to the SAPStartSrv.

To allow OS level authentication from SAPHostControl we can set the profile parameter service/admin_users = sapadm.

Simple Diagnostics Agent

The Simple Diagnostics Agent of FocusedRun uses the SAP Host Agent instance gateway to connect to the SAPStartSrv. SAP Host Agent will proxy requests from SDA to SAPStartSrv.

Software Update Manager (SUM)

SUM registers itself to the SAP Host Agent. SAP Host Agent will then be used to start SAPUp process. The SAPUp process will authenticate to SAPStartSrv on OS level by <sapsid>adm or by using the systemPKI.

Solution Manager Diagnostics (SMD Agent)

For SAP Diagnostics Agent to authenticate with user <smd>adm we can set the parameter for example to service/admin_users = daaadm. On Windows systems it would be service/admin_users = <domain>\SAPServiceDAA.

SAP Landscape Management 3.0

SAP LaMa 3.0 detects to-be-managed systems by collecting information about installed instances on a certain host by querying the corresponding SAP Host Agent. Afterwards it gathers relevant information about the instances directly from the SAPStartSrv by querying the web methods 'GestSystemInstanceList, ParameterValue, GetEnvironment, GetInstanceProperties'. Details about that process can be found in the attachment of SAP note 2039615.

While SAP LaMa is able to authenticate using username and password or a X.509 client certificate, all these credentials or client certificates of all connected systems would be stuffed on a single place, the SAP LaMa. So one may decide to unprotect the necessary web methods and perform the queries unauthenticated. This may also reduce the effort to configure the necessary prerequisits on each to-be-connected system.

Azure Monitor for SAP Solutions - SAP NetWeaver provider (AMS-NW)

The Azure Monitor for SAP Solutions (AMS) is a 3rd party monitoring tool from Microsoft which runs as a service in Microsoft Azure. As already described above it may be more secure to unprotect certain web methods for monitoring purposes instead of authenticate the monitoring tool and give full access to all web methods. According to the blogpost about technical details of AMS-NW the following web methods are used by AMS-NW v1:

For AS ABAP the following web methods are used:

GetSystemInstanceList, GetProcessList, GetQueueStatistic, ABAPGetWPTable, and EnqGetStatistic

For AS Java the following web methods are used:

GetSystemInstanceList, GetProcessList, GetQueueStatistic, and EnqGetStatistic

These web methods are queried from the Collector VM deployed in the Managed Ressource Group of AMS.

SUSE SAP Host Exporter

SUSE SAP Host Exporter is a 3rd party monitoring tool from SUSE which can be used to export metrics for usage in the Prometheus monitoring stack. At time of writing the following web methods are used:

GetProcessList, GetCurrentInstance, GetSystemInstanceList, GetQueueStatistic, EnqGetStatistic, GetInstanceProperties

nwa - System Overview (Flash component) in SAP NetWeaver AS Java

To display all information in the System Overview of nwa according to SAP note 2544271 both web methods 'PerfRead' and 'MtGetTidByName' and in addition to that according to SAP note 1883695 the web method 'J2EEGetProcessList' are necessary.

Since Flash has finally reached End-of-Life, a replacement for the System Overview in nwa was announced by SAP in SAP note 3013115 and published in SAP note 2981308. SAP confirmed that this replacement will still do unauthenticated requests on the three web methods PerfRead</code, MtGetTidByName and J2EEGetProcessList.

Log Viewer in nwa in SAP NetWeaver AS Java

The the Log Viewer (quick link /nwa/logs) uses the Java standalone web service proxy to communicate with SAPStartSrv.

Please note: For this communication my tests showed that the connection uses by default the IP address of $(SAPLOCALHOST). This means the communication will not work when service/http/hostname = 127.0.0.1 is set.

While SAP note 1631616 (dated to 2013(!)) and SAP note 2319551 (dated to 2016) indicate that authentication issues should be fixed in the Java standalone Web service proxy, other more recent SAP notes still addresses this issue:

According to SAP note 2577844 (dated to 2019) the web methods 'GetVersionInfo', 'ListLogFiles', 'ReadLogFile', 'ParameterValue', 'J2EEGetProcessList', 'PerfRead', 'MtGetTidByName', 'UtilSnglmsgReadRawdata', 'getTidsByName' should be necessarily unprotected to provide the functionality. SAP note 2506964 (dated to 2020) classifies this as a workaround. SAP note 2246609 (dated to 2016) remarks this as still necessary if logs from other instances should be displayed. This seems to be mixed up with requirements for Start&Stop (quick link /nwa/start-stop).

In my test environment (SAP NW AS Java 7.5, single instance) i could verify the Log Viewer needs in fact only the web methods GetVersionInfo, ListLogFiles, ReadLogFile and J2EEGetProcessList.

SSL Wizard in nwa in SAP NetWeaver AS Java

The SSL Wizard (quick link /nwa/ssl) uses also the Java standalone web service proxy to communicate with SAPStartSrv. It needs the web method GetAccessPointList (also described in SAP note 2510099) to display the correct status of SSL ports.

Go to blogpost Protecting web methods offered by SAP Host Agent.