This Article is referenced from the approach mentioned by Marcel Toerpe at SAP monitoring: A serverless approach using Amazon CloudWatch on SAP on AWS blogs. Check also his Github Repository for latest information on this solution
In this Article, we are going to discuss about an serverless approach of monitoring SAP based on Amazon CloudWatch and AWS Lambda. This approach describes us to effectively monitor SAP ABAP-based environments at a low cost without the need to deploy or manage any additional servers or agents using AWS Services. The solution can be deployed seamlessly with the AWS Serverless Application Repository or AWS CloudFormation. This allows us to easily publish our own metrics, such as SAP application-level performance data, and create thresholds and alarms in CloudWatch.
SAP has provided several notes which support SAP products on AWS:-
This Article will concentrate on the information more about AWS Services that can be used to monitor SAP, but relevant link has been provided for further information
Amazon VPC helps us to define a virtual network in our own way, in a logically isolated area within the AWS Cloud. We can launch your AWS resources, such as SAP instances on EC2, into this VPC. We can configure our VPC; we can select its IP address range, create subnets, and configure route tables, network gateways, and security settings.
AWS Lambda is a serverless compute service that lets us run code without provisioning or managing servers, creating workload-aware cluster scaling logic, maintaining event integrations, or managing runtimes. With Lambda, we can run code for virtually any type of application or backend service - all with zero administration.
Amazon CloudWatch is a monitoring and observability service built for developers, site reliability engineers (SREs), and IT managers. CloudWatch provides us the data and actionable insights to monitor SAP application. CloudWatch collects SAP monitoring and operational data in the form of logs, metrics, and events, providing us with a unified view of SAP system that run on AWS and on-premises servers. We can use CloudWatch to detect anomalous behavior in our SAP environments, set alarms, visualize logs and metrics side by side, take automated actions, troubleshoot issues, and discover insights to keep our SAP applications running smoothly.
The SAP Java Connector (SAP JCo) is a development library that helps a Java application to communicate with SAP systems via SAP's RFC protocol. It combines an easy-to-use API with unprecedented flexibility and performance. The SAP JCo supports both communication directions: inbound Remote Function Calls (Java calls ABAP) as well as outbound Remote Function Calls (ABAP calls Java). SAP provides below notes for further information:
AWS Secrets Manager helps us protecting secrets/credentials needed to access our SAP applications, services, and IT resources. The service enables us to easily rotate, manage, and retrieve database credentials, API keys, and other secrets throughout their lifecycle. Users and applications retrieve SAP secrets/credentials with a call to Secrets Manager APIs, eliminating the need to hardcode sensitive information in plain text.
AWS CloudFormation gives us an easy way to model a collection of related AWS and third-party resources, provision them quickly and consistently, and manage them throughout their lifecycles, by treating infrastructure as code. We can use a template to create, update, and delete an entire stack as a single unit, as often as we need to, instead of managing resources individually.
Using Amazon SNS topics, our publisher systems can fanout messages to many subscriber systems including Amazon SQS queues, AWS Lambda functions and HTTPS endpoints, for parallel processing, and Amazon Kinesis Data Firehose. The A2P functionality enables us to send messages to users at scale via SMS, mobile push, and email.
The following architecture diagram shows the monitoring setup and Flow that gets deployed with AWS CloudFormation
The generated Amazon CloudWatch rule triggers a Java-based Lambda function every minute and performs multiple Remote Function Calls (RFC) to the SAP system through the official SAP JCo Library, provided as Lambda layer. The connectivity is created by adding the Lambda to our existing Amazon Virtual Private Cloud (VPC) using also the latest improvements on networking for AWS Lambda.
The SAP RFC credentials, and connection information are stored securely inside AWS Secrets Manager and read on-demand to establish connectivity. The Lambda function extracts the SAP application level metrics, adds the respective custom namespace, and pushes it to CloudWatch. Users can then create and use their own custom dashboards inside CloudWatch.
CloudWatch also provide us feature to create Alarms based on Metrics which can be forwarded to Amazon Simple Notification Service (Amazon SNS) by which we can trigger email/SMS as per the settings.
For implementing the above architecture below are some of the pre-requisite that needs to be done
SAP Engine to which Lambda function is going to connect must have SAP NetWeaver ABAP 740 or higher version with components could be any of ECC, BW, S4, SRM, APO etc.
We can check SAP’s installation guides for implementing or upgrading the same version on the SAP landscape.
System’s Status can be checked for version of SAP that has been implemented on the system.
For the lambda function to fetch data from the ABAP instance using the SAP JCo is possible only if SAP Component ST-PI Release 740 SP 08 or higher is installed on the system. As this release contains some of the important Functional modules such as SWNC_GET_WORKLOAD_SNAPSHOT, which is called by the java code for collection of data which is delivered by SAP later to the mentioned ST-PI Release. SAP Notes 907035 can be checked for further information.
System’s Status can be checked for version of ST-PI that has been implemented on the system.
All configuration related to storing all the Statistical data of SAP system should be enabled. Data in T-code STAD and ST03 must be updating regularly with the latest data. SAP has provided guided procedure to keep Statistical data up to date. For more details below notes can be checked:-
We can check the Last Measurement Time to check if the data is getting updated in the system.
To allow AWS Lambda function to connect to SAP Instance with the required authorization, it is recommender to create a separate user for fetching such data from SAP. To do so, below steps needs to be done
Go to T-code PFCG and Upload the below Custom Role:-
Check the list of Roles that is inside the file and then proceed
Since the role is uploaded explicitly hence it is mandatory to generate the profiles for the same, to do so we need to go to Authorization Data
And then click Generate to populate the profiles for the role
After successful generation of profiles, we can see that same in the Role properties
Assign this role to the user which will be used by Lambda Function for fetching the Statistical data, we can either choose to create new user or existing user. It is recommended to use user as type System, as this User doesn’t need SAPGUI to fetch the data.
User Comparison needs to be done for completing the role assignment process
For connecting Lambda function to SAP Instance, we need to use Java code which will use SAP JCo Library
To create SAP JCo Library Lambda Layer, we first need to download Latest SAP JAVA Connector library from SAP Marketplace. More information can be checked on below notes:
This will download the file as zip to the local drive of the machine
Extract all the files which are available in the zip
Extraction has below files which are provided by SAP as part of Java Connector
Create a new folder java with sub-folder lib and add the below highlighted files
Then create a zip of java directory using standard OS tool
Cross check the files inside the java.zip using the OS standard tool
Go to AWS Lambda from AWS Console → click on Layer then click on Create Layer
Enter name of the Lambda Layer and upload the java.zip file. Also, specify the Compatible runtimes as JAVA 11 and 8. Click on Create to proceed with the creation of the lambda Layer
Below screen will appear after a successful creation of the Lambda Layer
This is end of Part – I, in next part Monitoring of ABAP System on AWS Using CloudWatch - Part II of this article you can find more information about Deployment, Testing, Post Configuration, Dashboard etc... of Monitoring ABAP Systems via AWS.
In this Article, we are going to discuss about an serverless approach of monitoring SAP based on Amazon CloudWatch and AWS Lambda. This approach describes us to effectively monitor SAP ABAP-based environments at a low cost without the need to deploy or manage any additional servers or agents using AWS Services. The solution can be deployed seamlessly with the AWS Serverless Application Repository or AWS CloudFormation. This allows us to easily publish our own metrics, such as SAP application-level performance data, and create thresholds and alarms in CloudWatch.
SAP has provided several notes which support SAP products on AWS:-
- 1656250 - SAP on AWS: Support prerequisites
- 1656099 - SAP Applications on AWS: Supported DB/OS and AWS EC2 products
This Article will concentrate on the information more about AWS Services that can be used to monitor SAP, but relevant link has been provided for further information
1. AWS SERVICES
1.1 Amazon VPC
Amazon VPC helps us to define a virtual network in our own way, in a logically isolated area within the AWS Cloud. We can launch your AWS resources, such as SAP instances on EC2, into this VPC. We can configure our VPC; we can select its IP address range, create subnets, and configure route tables, network gateways, and security settings.
1.2 AWS Lambda
AWS Lambda is a serverless compute service that lets us run code without provisioning or managing servers, creating workload-aware cluster scaling logic, maintaining event integrations, or managing runtimes. With Lambda, we can run code for virtually any type of application or backend service - all with zero administration.
1.3 Amazon CloudWatch
Amazon CloudWatch is a monitoring and observability service built for developers, site reliability engineers (SREs), and IT managers. CloudWatch provides us the data and actionable insights to monitor SAP application. CloudWatch collects SAP monitoring and operational data in the form of logs, metrics, and events, providing us with a unified view of SAP system that run on AWS and on-premises servers. We can use CloudWatch to detect anomalous behavior in our SAP environments, set alarms, visualize logs and metrics side by side, take automated actions, troubleshoot issues, and discover insights to keep our SAP applications running smoothly.
1.4 SAPJCO
The SAP Java Connector (SAP JCo) is a development library that helps a Java application to communicate with SAP systems via SAP's RFC protocol. It combines an easy-to-use API with unprecedented flexibility and performance. The SAP JCo supports both communication directions: inbound Remote Function Calls (Java calls ABAP) as well as outbound Remote Function Calls (ABAP calls Java). SAP provides below notes for further information:
1.5 AWS Secrets Manager
AWS Secrets Manager helps us protecting secrets/credentials needed to access our SAP applications, services, and IT resources. The service enables us to easily rotate, manage, and retrieve database credentials, API keys, and other secrets throughout their lifecycle. Users and applications retrieve SAP secrets/credentials with a call to Secrets Manager APIs, eliminating the need to hardcode sensitive information in plain text.
1.6 AWS CloudFormation
AWS CloudFormation gives us an easy way to model a collection of related AWS and third-party resources, provision them quickly and consistently, and manage them throughout their lifecycles, by treating infrastructure as code. We can use a template to create, update, and delete an entire stack as a single unit, as often as we need to, instead of managing resources individually.
1.7 Amazon SNS
Using Amazon SNS topics, our publisher systems can fanout messages to many subscriber systems including Amazon SQS queues, AWS Lambda functions and HTTPS endpoints, for parallel processing, and Amazon Kinesis Data Firehose. The A2P functionality enables us to send messages to users at scale via SMS, mobile push, and email.
2. ARCHITECTURE
The following architecture diagram shows the monitoring setup and Flow that gets deployed with AWS CloudFormation
The generated Amazon CloudWatch rule triggers a Java-based Lambda function every minute and performs multiple Remote Function Calls (RFC) to the SAP system through the official SAP JCo Library, provided as Lambda layer. The connectivity is created by adding the Lambda to our existing Amazon Virtual Private Cloud (VPC) using also the latest improvements on networking for AWS Lambda.
The SAP RFC credentials, and connection information are stored securely inside AWS Secrets Manager and read on-demand to establish connectivity. The Lambda function extracts the SAP application level metrics, adds the respective custom namespace, and pushes it to CloudWatch. Users can then create and use their own custom dashboards inside CloudWatch.
CloudWatch also provide us feature to create Alarms based on Metrics which can be forwarded to Amazon Simple Notification Service (Amazon SNS) by which we can trigger email/SMS as per the settings.
3. PREREQUISITES
For implementing the above architecture below are some of the pre-requisite that needs to be done
3.1 SAP NetWeaver ABAP
SAP Engine to which Lambda function is going to connect must have SAP NetWeaver ABAP 740 or higher version with components could be any of ECC, BW, S4, SRM, APO etc.
We can check SAP’s installation guides for implementing or upgrading the same version on the SAP landscape.
System’s Status can be checked for version of SAP that has been implemented on the system.
3.2 ST-PI
For the lambda function to fetch data from the ABAP instance using the SAP JCo is possible only if SAP Component ST-PI Release 740 SP 08 or higher is installed on the system. As this release contains some of the important Functional modules such as SWNC_GET_WORKLOAD_SNAPSHOT, which is called by the java code for collection of data which is delivered by SAP later to the mentioned ST-PI Release. SAP Notes 907035 can be checked for further information.
System’s Status can be checked for version of ST-PI that has been implemented on the system.
3.3 Statistical Records
All configuration related to storing all the Statistical data of SAP system should be enabled. Data in T-code STAD and ST03 must be updating regularly with the latest data. SAP has provided guided procedure to keep Statistical data up to date. For more details below notes can be checked:-
- 2369736 - Troubleshooting missing data in ST03N/ST03
- 552845 - FAQ: RFC Statistics in Transactions ST03/ST03N and STAD
We can check the Last Measurement Time to check if the data is getting updated in the system.
3.4 SAP RFC User
To allow AWS Lambda function to connect to SAP Instance with the required authorization, it is recommender to create a separate user for fetching such data from SAP. To do so, below steps needs to be done
Go to T-code PFCG and Upload the below Custom Role:-
DATE 20200423 153553
RELEASE 753
LOADED_AGRS ZSAPMONITOR
AGR_DEFINE 000ZSAPMONITOR DEVELOPER 20200408085327000000000000000DEVELOPER 20200423153520000000000000000
AGR_TCODES 000ZSAPMONITOR TR/SDF/SMON X 00000
AGR_TCODES 000ZSAPMONITOR TRSM50 X 00000
AGR_TCODES 000ZSAPMONITOR TRST03 X 00000
AGR_1250 000ZSAPMONITOR 000001S_ADMI_FCDT-HB53000100 M O000000
AGR_1250 000ZSAPMONITOR 000002S_APPL_LOGT-HB53000100 SX O000000
AGR_1250 000ZSAPMONITOR 000003S_BTCH_ADMT-HB53000100 U O000000
AGR_1250 000ZSAPMONITOR 000004S_DATASET T-HB53000100 SX O000000
AGR_1250 000ZSAPMONITOR 000005S_RFC T-HB53000100 U O000000
AGR_1250 000ZSAPMONITOR 000006S_TCODE T-HB53000100 S O000000
AGR_1250 000ZSAPMONITOR 000007S_TOOLS_EXT-HB53000100 G O000000
AGR_1250 000ZSAPMONITOR 000008S_USER_GRPT-HB53000100 U O000000
AGR_1251 000ZSAPMONITOR 000001S_ADMI_FCDT-HB53000100 S_ADMI_FCDPADM M O000000
AGR_1251 000ZSAPMONITOR 000002S_RFC T-HB53000100 RFC_TYPE FUNC U O000000
AGR_1251 000ZSAPMONITOR 000003S_RFC T-HB53000100 RFC_NAME SWNC_GET_WORKLOAD_SNAPSHOT U O000000
AGR_1251 000ZSAPMONITOR 000004S_RFC T-HB53000100 RFC_NAME RFC_GET_FUNCTION_INTERFACE U O000000
AGR_1251 000ZSAPMONITOR 000005S_RFC T-HB53000100 RFC_NAME RFCPING U O000000
AGR_1251 000ZSAPMONITOR 000006S_RFC T-HB53000100 RFC_NAME DDIF_FIELDINFO_GET U O000000
AGR_1251 000ZSAPMONITOR 000007S_RFC T-HB53000100 RFC_NAME /SDF/SMON_GET_SMON_RUNS U O000000
AGR_1251 000ZSAPMONITOR 000008S_RFC T-HB53000100 RFC_NAME /SDF/SMON_ANALYSIS_START U O000000
AGR_1251 000ZSAPMONITOR 000009S_RFC T-HB53000100 RFC_NAME /SDF/SMON_ANALYSIS_READ U O000000
AGR_1251 000ZSAPMONITOR 000010S_TCODE T-HB53000100 TCD /SDF/SMON S O000000
AGR_1251 000ZSAPMONITOR 000011S_TCODE T-HB53000100 TCD ST03 S O000000
AGR_1251 000ZSAPMONITOR 000012S_ADMI_FCDT-HB53000100 S_ADMI_FCDST0R M O000000
AGR_1251 000ZSAPMONITOR 000013S_TOOLS_EXT-HB53000100 AUTH S_TOOLS_EX_A G O000000
AGR_1251 000ZSAPMONITOR 000014S_RFC T-HB53000100 ACTVT 16 U O000000
AGR_1251 000ZSAPMONITOR 000015S_USER_GRPT-HB53000100 CLASS * U O000000
AGR_1251 000ZSAPMONITOR 000016S_ADMI_FCDT-HB53000100 S_ADMI_FCDST0M M O000000
AGR_1251 000ZSAPMONITOR 000017S_USER_GRPT-HB53000100 ACTVT 03 U O000000
AGR_1251 000ZSAPMONITOR 000018S_APPL_LOGT-HB53000100 ACTVT 03 SX O000000
AGR_1251 000ZSAPMONITOR 000019S_APPL_LOGT-HB53000100 ALG_OBJECTBCSGLWL SX O000000
AGR_1251 000ZSAPMONITOR 000020S_APPL_LOGT-HB53000100 ALG_OBJECTBCSWNC SX O000000
AGR_1251 000ZSAPMONITOR 000021S_APPL_LOGT-HB53000100 ALG_SUBOBJ SX O000000
AGR_1251 000ZSAPMONITOR 000022S_TCODE T-HB53000100 TCD SM50 S O000000
AGR_1251 000ZSAPMONITOR 000023S_BTCH_ADMT-HB53000100 BTCADMIN D U O000000
AGR_1251 000ZSAPMONITOR 000024S_DATASET T-HB53000100 PROGRAM SAPLSDEB SX O000000
AGR_1251 000ZSAPMONITOR 000025S_DATASET T-HB53000100 PROGRAM SAPLCRFC SX O000000
AGR_1251 000ZSAPMONITOR 000026S_DATASET T-HB53000100 FILENAME SX O000000
AGR_1251 000ZSAPMONITOR 000027S_DATASET T-HB53000100 ACTVT SX O000000
AGR_1251 000ZSAPMONITOR 000028S_BTCH_ADMT-HB53000100 BTCADMIN Y U O000000
AGR_1251 000ZSAPMONITOR 000029S_RFC T-HB53000100 RFC_NAME BDL_GET_CENTRAL_TIMESTAMP U O000000
AGR_1251 000ZSAPMONITOR 000030S_RFC T-HB53000100 RFC_NAME RFC_METADATA_GET U O000000
AGR_TEXTS 000ZSAPMONITOR E00000SAP Monitoring via Amazon CloudWatch
AGR_FLAGS 000ZSAPMONITOR COLL_AGR DEVELOPER 20200408085327DEVELOPER 20200408085327
AGR_FLAGS 000ZSAPMONITOR DEVCLASS DEVELOPER 20200408085327DEVELOPER 20200408085327
AGR_FLAGS 000ZSAPMONITOR MASTER_LANDEVELOPER 20200408085327DEVELOPER 20200408085327E
AGR_FLAGS 000ZSAPMONITOR RESP_USER DEVELOPER 20200408085327DEVELOPER 20200408085327
AGR_FLAGS 000ZSAPMONITOR FORCE_MIX DEVELOPER 20200408085718DEVELOPER 20200423153518
AGR_HIER 000ZSAPMONITOR 0000000200000001 00000010TR/SDF/SMON 01 X 0000000000
AGR_HIER 000ZSAPMONITOR 0000000300000001 00000020TRST03 01 X 0000000000
AGR_HIER 000ZSAPMONITOR 0000000400000001 00000030TRSM50 01 X 0000000000
AGR_HIERT 000ZSAPMONITOR D00000002Snapshot-Monitor
AGR_HIERT 000ZSAPMONITOR E00000002Snapshot Monitor
AGR_HIERT 000ZSAPMONITOR D00000003Systemlast u. Perform. Statistik
AGR_HIERT 000ZSAPMONITOR E00000003Workload and Performance Statistics
AGR_HIERT 000ZSAPMONITOR D00000004Workprozesse einer AS-Instanz
AGR_HIERT 000ZSAPMONITOR E00000004Work Processes of AS Instance
AGR_HIERT 000ZSAPMONITOR F00000004Synthèse des processus
AGR_HIERT 000ZSAPMONITOR I00000004Riepilogo processo di lavoro
AGR_HIERT 000ZSAPMONITOR N00000004Work-process-overzicht
AGR_TIME 000ZSAPMONITOR MENU DEVELOPER 20200408085435000000000000000DEVELOPER 20200408093202000000000000000
AGR_TIME 000ZSAPMONITOR PROFILE DEVELOPER 20200408085718000000000000000DEVELOPER 20200423153518000000000000000
AGR_LSD 000ZSAPMONITOR Check the list of Roles that is inside the file and then proceed
Since the role is uploaded explicitly hence it is mandatory to generate the profiles for the same, to do so we need to go to Authorization Data
And then click Generate to populate the profiles for the role
After successful generation of profiles, we can see that same in the Role properties
Assign this role to the user which will be used by Lambda Function for fetching the Statistical data, we can either choose to create new user or existing user. It is recommended to use user as type System, as this User doesn’t need SAPGUI to fetch the data.
User Comparison needs to be done for completing the role assignment process
3.5 SAP JCo Lambda Layer
For connecting Lambda function to SAP Instance, we need to use Java code which will use SAP JCo Library
3.5.1 Prepare SAP JCo Library
To create SAP JCo Library Lambda Layer, we first need to download Latest SAP JAVA Connector library from SAP Marketplace. More information can be checked on below notes:
This will download the file as zip to the local drive of the machine
Extract all the files which are available in the zip
Extraction has below files which are provided by SAP as part of Java Connector
Create a new folder java with sub-folder lib and add the below highlighted files
Then create a zip of java directory using standard OS tool
Cross check the files inside the java.zip using the OS standard tool
3.5.2 Create Lambda Layer
Go to AWS Lambda from AWS Console → click on Layer then click on Create Layer
Enter name of the Lambda Layer and upload the java.zip file. Also, specify the Compatible runtimes as JAVA 11 and 8. Click on Create to proceed with the creation of the lambda Layer
Below screen will appear after a successful creation of the Lambda Layer
This is end of Part – I, in next part Monitoring of ABAP System on AWS Using CloudWatch - Part II of this article you can find more information about Deployment, Testing, Post Configuration, Dashboard etc... of Monitoring ABAP Systems via AWS.
- SAP Managed Tags:
