Log file rotation and retention in ICM and WebDispatcher

[JoeGoerlich]

In this blogpost I like to shine a light on the handling of log files of the ICM.

While log file handling is a typical task of a SAP Basis Administrator, log files - especially ICM log files - are for sure involved when it comes to security analysis including forensics. For example when the RECON vulnerability was disclosed a security analyst could have a look at the ICM HTTP log files of an SAP NetWeaver AS Java for requests to the web service '/CTCWebService' to identify exploitation. Therefore a good log file management is crucial.

Please note: SAP Web Dispatcher shares the same logging functionality. For simplicity I will talk in the blogpost about ICM while everything can for sure also be applied to SAP Web Dispatcher. The same should be true for the SAP HANA integrated WebDispatcher.

Disclaimer: I will not consider the log forwarding to a central log analysis or SIEM solution and its impact on local log file writing and retention of log files.

Updates:

2022-11-02: Added new parameter icm/loadinfo_logging.
2022-10-31: Added hint for new default values as of SAP Kernel 7.81 (ABAP Platform 2020). Added icm/security_log.
2021-12-16: Slightly rephrased and adopted to new stylesheet of blogs.sap.com

---

Access and request logs

The ICM supports logging for HTTP/HTTPS (incoming/outgoing) as well as for TCP traffic. The logging is controlled via the following vector parameters:

Please note: Vector parameter means there can be multiple parameters of the same area defined by increasing the number at the end of the parameter name. In logging related parameters this may be used to write pre-filtered requests into a separate log file, e.g., by specifying the relevant values in sub-parameter PREFIX or FILTER for analyzing special cases.

 

icm/HTTP/logging_<xx>

Defines the logging of incoming requests on ports of type HTTP as well as HTTPS, indicated by a icm/server_port_<xx> with PROT=HTTP or PROT=HTTPS were the port is greater 0.

Please note: As of SAP Kernel 7.81 (ABAP Platform 2020) the default value is PREFIX=/,LOGFILE=$(DIR_LOGGING)$(DIR_SEP)http-%y-%m-%d.log%z,MAXFILES=7,MAXSIZEKB=100000,SWITCHTF=day,LOGFORMAT=DEFAULT.

icm/HTTP/logging_client_<xx>

Defines the logging of outgoing requests, be it via HTTP or be it via HTTPS.

Please note: At time of writing there is no logging for outgoing LDAP connections from the ICM LDAP plug-in (which btw. is the successor of the legacy LDAP Connector). This did also not change with Kernel 7.81 in which ICM LDAP plug-in was decoupled from HTTP (see SAP note 2920871).

Please note 2#: As of SAP Kernel 7.81 (ABAP Platform 2020) the default value is PREFIX=/,LOGFILE=$(DIR_LOGGING)$(DIR_SEP)http-client-%y-%m-%d.log%z,MAXFILES=7,MAXSIZEKB=50000,SWITCHTF=day,LOGFORMAT=DEFAULT.

 

icm/TCP/logging_<xx>

Defines the logging of incoming requests on ports of type TCP, indicated by a icm/server_port_<xx> with PROT=TCP.

 

Other ICM logs

icm/security_log

Defines the logging of security relevant events of the ICM itself.

Please note: As of SAP Kernel 7.81 (ABAP Platform 2020) the default value is LOGFILE=$(DIR_LOGGING)$(DIR_SEP)dev_icm_sec-%y-%m-%d%z,LEVEL=3,MAXFILES=7,MAXSIZEKB=50000,SWITCHTF=day.

 

icm/loadinfo_logging

Defines the logging of load information.

Please note: As of SAP Kernel 7.85 (ABAP Platform 2021) the effective default value is LOGFILE=icm-load-statistic-%y-%m.log,MAXSIZEKB=10000,MAXFILES=2,FLUSH=ON,SWITCHTF=month,APPEND=TRUE,ACTIVE=TRUE.

Log rotation and retention

Each of the mentioned parameters supports a similar set of sub-parameters which are documented on https://help.sap.com.

For this blogpost I like to focus on the sub-parameters relevant for log rotation:

• LOGFILE: defines the log file destination and name.


• MAXFILES: defines the amount of log files to be kept after log rotation


• MAXSIZEKB: defines the maximum size of a log file before a log rotation is performed


• SWITCHTF: defines the period after which a log rotation is performed

In the following I recommend settings for each of the mentioned sub-parameters and give an explanation:

 

LOGFILE=$(DIR_LOGGING)$(DIR_SEP)<string>_$(SAPSYSTEMNAME)_$(SAPSYSTEM)_$(SAPLOCALHOST)-%y-%m-%d.log%z

The value of this sub-parameter consists of a concatenation of the following parts:

$(DIR_LOGGING)$(DIR_SEP)

This describes the log file destination. As of SAP Kernel 7.77  this can no longer be defined as an absolut path instead it must be defined using certain variables as specified in SAP note 2757613. Therefore I recommend to set the log file destination to the log directory of every instance using the variable $(DIR_LOGGING) followed by the OS dependent separator in form of the variable $(DIR_SEP) since when I rollout the parameter setting to many systems I do not want to deal with on which OS the instance is running on.

<string>

A string describing the content of the log file. I recommend to use
'icm_tcp' for icm/TCP/logging_<xx>,
'icm_http' for icm/HTTP/logging_<xx> and
'icm_http_client' for icm/HTTP/logging_client_<xx>.
I decided to have the keyword 'icm' in there, because there is also a HTTP log for the message server which may then use the string 'ms_http'.

_$(SAPSYSTEMNAME)_$(SAPSYSTEM)_$(SAPLOCALHOST)

The three variables will be substituted by the SID, the Instance Name and the hostname. This helps to make the log files uniquely identifiable, for example if you have to collect the raw files from all instances of a 3 system landscape and store them elsewhere. With this schema you do not have to take care about preventing files from being overwritten .

-%y-%m-%d

The placeholders are substituted by the date the log file was created on. This makes it obviously clear when the log file was created an in addition to that it makes it more easy to sort the files by date.

.log

Here I use the file extension associated with my preferred log analysis application.

%z

Finaly I add the placeholder %z which enables log files to be compressed after log rotation.

Please note: Starting with SAP Kernel 7.81 (as of SAP there will be no downport) the parameters icm/TCP/logging, icm/HTTP/logging,  icm/HTTP/logging_client, icm/security_log support '%z' as a placeholder in the LOGFILE sub-parameter. In earlier releases this placeholder will not work and leads to an error in the dev_icm. From my testing this can be seen as a warning and does not affect the creation of log files in general.

%o 

[optional] When a new log file is opened due to log rotation, the previous file gets renamed by appending the suffix “.old” to it.

Please note: '%o' is supported starting with SAP Kernel 7.53 in the parameters icm/TCP/logging, icm/HTTP/logging,  icm/HTTP/logging_client.
In the parameter icm/security_log support '%o'  is supported as of SAP Kernel 7.77.

 

SWITCHTF=day

It's recommend to switch the log file every day. This means a new log file will be created for the first request on a new day. This makes it clear when a log file was created and may help to keep the log files manageable in terms of their size.

Please note: Assuming a test system is not utilized on weekends at all, a new log file will be created for the first requests on Monday morning.

 

MAXFILES=90

Setting the number of log files to be kept to 90 in combination with SWITCHF=day and MAXSIZEKB=0, results in log retention of at least 90 days. Please remember the explanation for the sub-parameter SWITCHTF. These are 90 files and not exactly 90 calendar days of log retention - that're rather the requests of 90 days.

The maximum retention period may be affected by local law, workers council or simply by the available disk space.

Hint: The parameter icm/security_log also will support the sub-parameter MAXFILES starting with SAP Kernel 7.77. For all others it is supported as of SAP Kernel 7.45.

 

MAXSIZEKB=0

Not defining a maximum file size for a single log file, prevents from multiple files per day being created. This helps to prevent log files from being heavily rotated due to high load. For some high load scenarios this should be at the back of one's mind, especially if there is no load distribution to several instances or on the WebDispatcher. Since the log file may become very large and query them could take some time.

Please note: changing this value would interfere the log retention we achieve with the value in MAXFILES. If a different value than 0 is specified here, the MAXFILES=90 in combination with SWITCHF=day does no longer lead to a log retention of 90 days.

Conclusion

Built in mechanism can be utilized to perform the log rotation and retention. A housekeeping via scripts is no longer necessary.
The new placeholder %z introduced with SAP Kernel 7.81 helps a lot to save space, since log files are very good compressible.