How to configure tracing in SAP Mobile Platform 2.3.5 for Hazelcast
Purpose:
To implement tracing functions inside the Hazelcast classes.
Description :
Hazelcast is shipped with all SAP Mobile Platform 2.3.x versions.
What is Hazelcast ?
Hazelcast stores frequently accessed data in-memory and across an elastically
scalabel data grid. This enables any network of machines to dynamically cluster
and pool both memory and processors to accelerate application performance.
Nodes automatically discover and join the cluster which grows to meet growing
demands for low-latency performance, data volume and SLA's.
By default there is no entry point in to the Hazelcast debugging class from
within the SAP Control Center or the configuration of SAP SMP server.
Requirered Software:
- SAP SMP 2.3 or higher.
- A text editor.
- WinRAR ( Winzip is not recomended)
In addtition to the above software you need read/write file permission to the Windows file system and access
to the SMP supAdmin account.
Assumptions :
You have a working SAP Mobile Platform.
You are testing in a development or in a QA environment. It is not recommended to perform
these steps in a production environment unless you are directly working with SAP Product
Support or Engineering as this tracing should be done in a controlled environment.
Issue 1:
You are seeing Hazelcast errors in the SAP Mobile Server's error logs
and your users are unable to connect or login to the servers. In order to clear the condition
of no successful connections you are forced to restart the node or all the nodes in the cluster
to clear the issue.
Issue 2:
In reviewing the Hazelcast errors it is unclear why the errors are happening and there is no
clear root cause for the issue.
Example of a Hazelcast error message:
2014-11-18 14:03:31.513 ERROR MMS HttpChannel Thread-330 [com.sybase.sup.server.http.
DispatcherServlet] [] [] java.lang.IllegalStateException:
Hazelcast Instance is not active! java.lang.RuntimeException: java.lang.IllegalStateException:
Hazelcast Instance is not active! Line 48:
Caused by: java.lang.IllegalStateException: Hazelcast Instance is not active!
The other observation is that port 2480(Replication port) and 5001(Messaging port) are not available.
Even though we can see the Hazelcast error that it was not active the root cause of why it was not active is
never explained in the error message.
Solution:
Enable the debugging trace flag to display the full error messages from the Hazelcast libraries.
How to set the Hazelcast log level to debug:
1. Using the SCC GUI, set the "Other" log level to "DEBUG". Hazelcast debug logs will print in the server log file.
Note this log change also puts a lot of other 3rd party components to debug and will produce a large amount of unrelated
logging. Do not leave this option on for a long time as it will produce a large amount of log information.
2. We need do some SMP server changes and then restart every SMP node in the cluster. First stop the SMP server.
3. Backup \MobilePlatform\Servers\UnwiredServer\lib\ext\sup-server-rt.jar.
4. Open the sup-server-rt.jar with WinRAR. Do not use Winzip.
5. With the sup-server-rt.jar open you will need to extract out the file loggerBucketMappings.properties
Copy just that file to your desktop.
NOTE: Do not unzip the jar file. Doing so will change the manifest and the jar file will no longer be usable.
6. Edit the jar: \MobilePlatform\Servers\UnwiredServer\lib\ext\sup-server-rt.jar\com/
sybase/sup/server/logging/loggerBucketMappings.properties.
Figure 1.
7. Once you have the copy of the file loggerBucketMappings.properties move it to your desktop.
Use a standard text editor to edit the file.
8. Modify this line: Cluster=com.sybase.sup.server.cluster.*, sup.custom.cluster.* Adding ‘com.hazelcast.*
It needs to look like the line below: Cluster=com.sybase.sup.server.cluster.*, sup.custom.cluster.*, com.hazelcast.*
Figure 2.
9. Make note that we are adding all of the Hazelcast .* classes.
This will pull in all of the debug libraries that are needed to debug Hazelcast.
10. Once you have finished editing the file save it. Now we will need to add the file back in to the jar file.
11. Save the loggerBucketMappings.properties.
12. Then drag and drop the file back in to WinRar.
Figure 3.
You will get a popup screen. Click OK and the modified copy will be added and replace the original loggerBucketMappings.properties.
If you haven’t done so already back up the original copy of the file found here:
\MobilePlatform\Servers\ UnwiredServer\lib\ext\sup-server-rt.jar.
Then copy in the modified sup-server-rt.jar One you have copied the file in place you can start Unwired Server.
13. Each of the above steps must be done for every SMP server node in the cluster.
If all the nodes are the same version and patch levels then you can copy the sup-server-rt.jar to all other nodes.
You should still back up the original files that are already in place.
Not doing so could cause an issue when rolling back.
You want the system to be exactly the way it was found and have the original file back in place when done.
14. In SAP Control Center, Set "Cluster" log bucket to "DEBUG" level.
15. Now that we have the Hazelcast log level set to debug we can now proceed to testing.
16. Run the servers and perform any testing with the application and so on.
When the issue happens you need to capture the logs from the logs directory
C:\SAP\MobilePlatform\Servers\UnwiredServer\logs. Look for the servername-server.log file.
Debug trace output:
Figure 4.
Analysis
Finally we have useful output that we can understand.
Looking at the error it is clear that Hazelcast is unable to communicate to the other nodes
and considers the node off line and no longer available as we can see in the error message.
Summary
Even though the SAP unwired servers are up and running Hazelcast is letting us know there is a communication issue going on between the nodes in the cluster. It is recommended to use this SAP open source utility:
https://github.com/SAP/sap_mobile_platform_server_utilities
With the combination of the trace flag for Hazelcast and the Network check utility you can quickly narrow down the cause of Hazelcast errors. There are a number of reasons for Hazelcast errors. With additional debugging turned on it is a much faster way of getting to the root cause.
- SAP Managed Tags:
