Introduction
Many customers using SAP Enterprise Threat Detection (SAP ETD) have use cases where they would like to combine log information from their SAP systems with logs from other systems in their landscape, e.g. firewalls, virus scanners, proxies. SAP ETD provides different means to integrate non-SAP logs: The component Log Learning provides means to learn unstructured logs from sample data and allows mapping to the semantic data model of SAP ETD. For structured logs the recommended option is to build a custom parsing project in SAP HANA Smart Data Streaming.
Many devices and security products readily provide logs in the Common Event Format (CEF) defined and supported by the SIEM product HPE Security ArcSight. CEF provides a set of predefined attributes which are typical for security logs, for example, event name, source and destination IP addresses and ports, user names, URLs. In addition, CEF defines a set of attributes which can be used to represent custom information for a specific security product. The example below illustrates this. It uses some standard attributes and one custom attribute, namely cs1, here used to represent the name of a malware which was detected.
In order to make these logs available to SAP ETD we need to map the information in CEF format to the semantic events and attributes defined in SAP ETD (for more details cf. the product documentation at http://help.sap.com/sapetd). To some extent the mapping can be done in a canonical way. However, as most devices use custom attributes we need to adjust the canonical mapping for the specifics of a security product.
Below we explain an implementation such a mapping and how to adjust this to the CEF variant used by a specific security product. This has been built on the SAP HANA Smart Data Streaming (SAP SDS) component which is part of the product SAP ETD.
Overview of mapping CEF to SAP ETD
The mapping takes place in two steps:
Afterwards the resulting log events are handed over to a project which is part of SAP ETD and for further processing. The figure below shows how this is realized technically as a set of projects on SAP SDS.
Note: In this article we will not go into details how to import projects into SAP HANA Smart Data Streaming. You may want to consult chapter 2.4 of the SAP Enterprise Threat Detection Deployment and Configuration Guide available at http://help.sap.com/sapetd.
Step 1: Generic parsing of CEF data
The project cef_parse_data picks up data in CEF format either via file or by listening to a TCP port where CEF. Per default file directory, host name and port are configurable as shown in the figure below.
Should you need more flexibility, then you can adapt the entire project to your own needs.
The incoming log lines are parsed, taking into account escaping of characters as specified in chapter 1 of the CEF specification. We are consciously ignoring the syslog header fields dateTime and host as they are optional and do not provide any added value. Although we can parse any valid CEF attribute name, we restrict further processing to the attribute names defined defined in chapter 2 "ArcSight Extension Dictionary" of the CEF specification. A message will be output in the SDS console if we detect other attributes. The identified CEF attributes are then passed on to the mapping step. The parsed CEF data are provided in form of the stream CefStream. Subsequent projects can access these data by subscribing to the stream via an input binding.
Step 2: Product-specific mapping of CEF attributes to SAP ETD events and attributes
Projects named cef_<x>_2_etd are taking the parsed data and map them to the semantic events and attributes SAP Enterprise Threat Detection had defined. As mentioned above, many attributes can be mapped generically, however, frequently we still have to do a product-specific mapping. Therefore multiple mapping projects may exist and subscribe to the same stream.
The following figure shows an excerpt of the generic mapping of CEF attributes to the corresponding semantic attributes in SAP ETD. First comes the CEF attribute or combination thereof and then the SAP ETD attribute being mapped to. For example, line 15 shows that the CEF attribute externalId is mapped to the SAP ETD attribute CorrelationId,line 18 shows that CEF attributes deviceVendor and deviceProduct are concatenated with a "/" inbetween and mapped to SAP ETD attribute EventLogType.
Attached to this article you find a few sample mappings between CEF and SAP ETD: a generic one which can be used as template for own mappings, one for FireEye products and one for TrendMicro products. Each mapping is implemented as a specific SAP SDS project.
In order to distinguish for which CEF events a mapping shall take place we do a comparison with the CEF attributes for vendor, product and product version. The mapping in the figure below will only apply in case CEF attributes deviceVendor and deviceProduct both have the value "Generic".
In the mapping for FireEye products, for example, we only check for deviceVendor and use one mapping for all kinds of FireEye products.
If you create own mappings and use them actively, you should ensure that the checks for the specific products are mutually exclusive. Otherwise you may end up in SAP ETD with redundant log entries for the same original log.
As mentioned before the connection between different projects in SAP HANA Smart Data Streaming works via a publish/subscribe mechanism connecting output streams to input streams. All mapping projects subscribe to the CefStream and provide the output stream LogEventOut which needs to be connected via an output binding to the project transfer_log_event which writes log data into the SAP HANA database. This project is part of the SAP Enterprise Threat Detection product shipment.
The figure below shows the creation of an Input Binding in order to connect a mapping project to the generic parsing project. We assume that both parsing and mapping projects are running on the same SDS server.
In order to connect the output of a mapping project with the project transfer_log_event which takes care of the log transfer to teh SAP HANA database we need to add an Output Binding to our maping project.
Be aware that you will have to do the binding configurations for every single mapping project you want to use!
Having adapted the SDS projects to your needs and configured the relevant bindings you now only have to compile them and deploy to your SDS runtime. Then you can start providing CEF events from your product of choice to SAP Enterprise Threat Detection.
Now, have a look at the examples provided with this blog, try them out and adjust them to your own needs.
Many customers using SAP Enterprise Threat Detection (SAP ETD) have use cases where they would like to combine log information from their SAP systems with logs from other systems in their landscape, e.g. firewalls, virus scanners, proxies. SAP ETD provides different means to integrate non-SAP logs: The component Log Learning provides means to learn unstructured logs from sample data and allows mapping to the semantic data model of SAP ETD. For structured logs the recommended option is to build a custom parsing project in SAP HANA Smart Data Streaming.
Many devices and security products readily provide logs in the Common Event Format (CEF) defined and supported by the SIEM product HPE Security ArcSight. CEF provides a set of predefined attributes which are typical for security logs, for example, event name, source and destination IP addresses and ports, user names, URLs. In addition, CEF defines a set of attributes which can be used to represent custom information for a specific security product. The example below illustrates this. It uses some standard attributes and one custom attribute, namely cs1, here used to represent the name of a malware which was detected.
CEF:0|Security|threatmanager|1.0|100|worm successfully stopped|10|src=10.0.0.1 dst=2.1.2.2 spt=1232 cs1Label=Malware Name cs1=ILoveYou act=blockedIn order to make these logs available to SAP ETD we need to map the information in CEF format to the semantic events and attributes defined in SAP ETD (for more details cf. the product documentation at http://help.sap.com/sapetd). To some extent the mapping can be done in a canonical way. However, as most devices use custom attributes we need to adjust the canonical mapping for the specifics of a security product.
Below we explain an implementation such a mapping and how to adjust this to the CEF variant used by a specific security product. This has been built on the SAP HANA Smart Data Streaming (SAP SDS) component which is part of the product SAP ETD.
Overview of mapping CEF to SAP ETD
The mapping takes place in two steps:
- Generic parsing of CEF data
- Product-specific mapping of CEF attributes to SAP ETD events and attributes
Afterwards the resulting log events are handed over to a project which is part of SAP ETD and for further processing. The figure below shows how this is realized technically as a set of projects on SAP SDS.
Note: In this article we will not go into details how to import projects into SAP HANA Smart Data Streaming. You may want to consult chapter 2.4 of the SAP Enterprise Threat Detection Deployment and Configuration Guide available at http://help.sap.com/sapetd.
Step 1: Generic parsing of CEF data
The project cef_parse_data picks up data in CEF format either via file or by listening to a TCP port where CEF. Per default file directory, host name and port are configurable as shown in the figure below.
Should you need more flexibility, then you can adapt the entire project to your own needs.
The incoming log lines are parsed, taking into account escaping of characters as specified in chapter 1 of the CEF specification. We are consciously ignoring the syslog header fields dateTime and host as they are optional and do not provide any added value. Although we can parse any valid CEF attribute name, we restrict further processing to the attribute names defined defined in chapter 2 "ArcSight Extension Dictionary" of the CEF specification. A message will be output in the SDS console if we detect other attributes. The identified CEF attributes are then passed on to the mapping step. The parsed CEF data are provided in form of the stream CefStream. Subsequent projects can access these data by subscribing to the stream via an input binding.
Step 2: Product-specific mapping of CEF attributes to SAP ETD events and attributes
Projects named cef_<x>_2_etd are taking the parsed data and map them to the semantic events and attributes SAP Enterprise Threat Detection had defined. As mentioned above, many attributes can be mapped generically, however, frequently we still have to do a product-specific mapping. Therefore multiple mapping projects may exist and subscribe to the same stream.
The following figure shows an excerpt of the generic mapping of CEF attributes to the corresponding semantic attributes in SAP ETD. First comes the CEF attribute or combination thereof and then the SAP ETD attribute being mapped to. For example, line 15 shows that the CEF attribute externalId is mapped to the SAP ETD attribute CorrelationId,line 18 shows that CEF attributes deviceVendor and deviceProduct are concatenated with a "/" inbetween and mapped to SAP ETD attribute EventLogType.
Attached to this article you find a few sample mappings between CEF and SAP ETD: a generic one which can be used as template for own mappings, one for FireEye products and one for TrendMicro products. Each mapping is implemented as a specific SAP SDS project.
In order to distinguish for which CEF events a mapping shall take place we do a comparison with the CEF attributes for vendor, product and product version. The mapping in the figure below will only apply in case CEF attributes deviceVendor and deviceProduct both have the value "Generic".
In the mapping for FireEye products, for example, we only check for deviceVendor and use one mapping for all kinds of FireEye products.
If you create own mappings and use them actively, you should ensure that the checks for the specific products are mutually exclusive. Otherwise you may end up in SAP ETD with redundant log entries for the same original log.
As mentioned before the connection between different projects in SAP HANA Smart Data Streaming works via a publish/subscribe mechanism connecting output streams to input streams. All mapping projects subscribe to the CefStream and provide the output stream LogEventOut which needs to be connected via an output binding to the project transfer_log_event which writes log data into the SAP HANA database. This project is part of the SAP Enterprise Threat Detection product shipment.
The figure below shows the creation of an Input Binding in order to connect a mapping project to the generic parsing project. We assume that both parsing and mapping projects are running on the same SDS server.
In order to connect the output of a mapping project with the project transfer_log_event which takes care of the log transfer to teh SAP HANA database we need to add an Output Binding to our maping project.
Be aware that you will have to do the binding configurations for every single mapping project you want to use!
Having adapted the SDS projects to your needs and configured the relevant bindings you now only have to compile them and deploy to your SDS runtime. Then you can start providing CEF events from your product of choice to SAP Enterprise Threat Detection.
Now, have a look at the examples provided with this blog, try them out and adjust them to your own needs.
- SAP Managed Tags:
