Log file rotation and retention in ICM and WebDispatcher
With this I like to shine a light on the handling of log files of the ICM. While log file handling is a typical task of a SAP Basis Administrator, log files – especially ICM log files – are for sure involved when it comes to security analysis including forensics.
For example in the recent RECON vulnerability a security analyst could scan the ICM HTTP log files of a SAP NetWeaver AS Java for requests to the web service ‘/CTCWebService’ to identify exploitation. Therefore a good log file management is crucial.
Please note: SAP Web Dispatcher shares the same logging functionality. For simplicity I will talk in the blogpost about ICM while everything can for sure also be applied to SAP Web Dispatcher. The same should be true for the SAP HANA integrated WebDispatcher.
Disclaimer: I will not consider the log forwarding to a central log analysis or SIEM solution and its impact on local log file writing and retention of log files.
Log files based on connection types
The ICM supports logging for HTTP/HTTPS as well as for TCP traffic. The logging is controlled via the following vector parameters:
Please note: Vector parameter means there can be multiple parameters of the same area defined by increasing the number at the end of the parameter name. In logging related parameters this may be used to write pre-filtered requests into a separate log file, e.g., by specifying the relevant values in sub-parameter PREFIX or FILTER for analyzing special cases.
Defines the logging of incoming requests on ports of type TCP, indicated by a icm/server_port_<xx> with PROT=TCP.
Please note: At time of writing requests on ports of type TCPS are not logged!
Defines the logging of incoming requests on ports of type HTTP as well as HTTPS, indicated by a icm/server_port_<xx> with PROT=HTTP or PROT=HTTPS were the port is greater 0.
Defines the logging of outgoing requests, be it via HTTP or be it via HTTPS.
Please note: If there is no icm/server_port_<xx> with PROT=HTTP defined at all (meaning the default value for icm/server_port_1 was overwritten by setting the profile parameter without specifying a value) the ICM does not support outgoing HTTP, LDAP, LDAPS, or LDAP with STARTTLS traffic. In this case the ICM automatically spins up an implicit port 0 for protocol HTTPS. This would hinder from restricting outgoing HTTPS traffic.
At time of writing there is no logging for outgoing LDAP connections made by the ICM LDAP plug-in (which btw. is the successor of the legacy LDAP Connector). This did also not change with Kernel 7.81 since which ICM LDAP plug-in was decoupled from HTTP (see SAP note 2920871).
Defines the logging of security events and irregularities.
Log rotation and retention
Each of the mentioned parameters supports a similar set of sub-parameters which are documented on https://help.sap.com. For this blogpost I like to focus on the sub-parameters relevant for log rotation:
LOGFILE: defines the log file destination and name.
MAXFILES: defines the amount of log files to be kept after log rotation
MAXSIZEKB: defines the maximum size of a log file before a log rotation is performed
SWITCHTF: defines the period after which a log rotation is performed
In the following I recommend settings for each of the mentioned sub-parameters and give an explanation for the recommended value.
The value of this sub-parameter consists of a concatenation of the following parts:
$(DIR_LOGGING)$(DIR_SEP): This describes the log file destination. As of SAP Kernel 7.77 this can no longer be defined as an absolut path instead it must be defined using certain variables as specified in SAP note 2757613. Therefore I recommend to set the log file destination to the log directory of every instance using the variable $(DIR_LOGGING) followed by the OS dependent separator in form of the variable $(DIR_SEP) since when I rollout the parameter setting to many systems I do not want to deal with on which OS the instance is running on.
<string>: A string describing the content of the log file. I recommend to use
‘icm_tcp’ for icm/TCP/logging_<xx>,
‘icm_http’ for icm/HTTP/logging_<xx> and
‘icm_http_client’ for icm/HTTP/logging_client_<xx>.
I decided to have the keyword ‘icm’ in there, because there is also a HTTP log for the message server which may then use the string ‘ms_http’.
_$(SAPSYSTEMNAME)_$(SAPSYSTEM)_$(SAPLOCALHOST): The three variables will be substituted by the SID, the Instance Name and the hostname. This helps to make the log files uniquely identifiable, for example if you have to collect the raw files from all instances of a 3 system landscape and store them elsewhere. With this schema you do not have to take care about preventing files from being overwritten .
-%y-%m-%d: The placeholders are substituted by the date the log file was created on. This makes it obviously clear when the log file was created an in addition to that it makes it more easy to sort the files by date.
.log: Here I use the file extension associated with my preferred log analysis application.
%z: Finaly I add the placeholder %z which enables log files to be compressed after log rotation.
Please note: Starting with SAP Kernel 7.81 (as of SAP there will be no downport) the parameters icm/TCP/logging, icm/HTTP/logging, icm/HTTP/logging_client, icm/security_log support ‘%z’ as a placeholder in the LOGFILE sub-parameter. In earlier releases this placeholder will not work and leads to an error in the dev_icm. From my testing this can be seen as a warning and does not affect the creation of log files in general.
Please note: Starting with SAP Kernel 7.77 the parameters icm/TCP/logging, icm/HTTP/logging, icm/HTTP/logging_client, icm/security_log support ‘%o’ as a placeholder in the LOGFILE sub-parameter.
%o: When a new log file is opened due to log rotation, the previous file gets renamed by appending the suffix “.old” to it.
I recommend to switch the log file every day. This means a new log file will be created for the first request on a new day. This makes it clear when a log file was created and may help to keep the log files manageable in terms of their size.
Please note: Assuming a test system is not utilized on weekends at all, a new log file will be created for the first requests on Monday morning.
I set the number of log files to be kept to 90, since in case of an incident it would be good to have a history of log files which can be queried. The maximum retention period may be affected by local law, workers council or simply by the available disk space.
Please be aware of the explanation for the sub-parameter SWITCHTF. I’m are talking about 90 files not 90 days of log retention – even if I set SWITCHTF=day there may not be a file for every day.
Please note: the parameter icm/security_log also will support the sub-parameter MAXFILES starting with SAP Kernel 7.77
I do not define a maximum file size for a single log file to prevent from multiple files per day being created. This helps to prevent log files from being heavily rotated due to high load. For some high load scenarios this should be at the back of one’s mind, especially if there is no load distribution to several instances or on the WebDispatcher. Since the log file may become very large and query them could take some time.
Built in mechanism can be utilized to perform the log rotation and retention. A housekeeping via scripts is no longer necessary.
The new placeholder %z introduced with SAP Kernel 7.81 helps a lot to save space, since log files are very good compressible.