Demystifying Event Stream Processor Server Errors
Sybase Event Stream Processor error messages generally follow a standard format. However, ESP uses a number of open source components. The format of the error message depends on which component throws the error. This blog post attempts to shed a little light on some of the various ESP server error messages. A future blog will touch on errors thrown by other components.
The errors most relevant to the project’s runtime state will show up in ESP’s “esp_server.log” file. Finding this log file can be tricky but the documentation offers some good guidance (Project Logging). The second most important log file related to the project is the “stdstreams.log” file which contains version and licensing information.
Breaking down the format:
YYYY-MM-DD HH:MM:SS.SSS | ZZZ | container | [SP-XX-MMMNNN] (seconds) sp(pid) ModuleName(Window_or_Stream) error message text
YYYY-MM-DD HH:MM:SS.SSS – The UTC timestamp for the message.
ZZZ – The ID of the lwp/thread that printed the message (this would be helpful if we were examining a core file and needed to match the message up with a thread).
|LOG_EMERG||0||system is unusable|
|LOG_ALERT||1||action must be taken immediately|
|LOG_NOTICE||5||normal but significant condition|
Detailed analysis of an error message:
Now, let’s go back and look at the example I provided:
2013-04-05 16:43:59.725 | 110 | container | [SP-3-148001] (23.418) sp(24460) ConnectionReader(StockQuoteInStream) connection reset error: 2013-04-05 16:43:59 reset(): could not read file ‘/software/ESP51/ESP-5_1/sampledata/STOCK_QUOTE.csv’
We know the date and time the error was logged (2013-04-05 16:43:59.725) and the thread ID (110). Since it was logged at level 3, it was an error so we should pay attention to it. I normally run my projects with the logging level set to 4 and I pay close attention to messages logged at level 3 or lower.
- A connection failed to reset. Probably some parameters are specified wrong. Check and correct the model.
Not exactly the most helpful suggestion. What line of code should I look at in my project? What parameters need to be corrected? If we look at the rest of the message, it makes more sense.
The “ConnectionReader” module that raised the error might be a good place for the technical support engineer to start looking if they needed to review the code to gain a better understanding of the error. But for everyone else, this bit might not be that meaningful.
The name of the stream or window “StockQuoteInStream” is a good clue. We can search through the project’s CCL code to see where this stream is referenced.
The last bit of the error is the most telling:
could not read file ‘/software/ESP51/ESP-5_1/sampledata/STOCK_QUOTE.csv’
This tells me that the stream “StockQuoteInStream” likely had a CSV Input Adapter (because it was trying to read the file) attached to it and that it was unable to open the file. I could now check to see if the file exists, if I have permission to read it or see if there was perhaps a typo in my project’s CCL.
As I mentioned, the second most important log related to the project’s runtime state is the “stdstreams.log”. This log file contains the product’s actual version string which is always important to know when contacting technical support. For example:
Sybase Event Stream Processor Engine 5.1.0/20130417.1/SP02 PL00/linux/x86_64/64-bit/OPT/Wed Apr 17 22:53:14 PDT 2013
It also contains messages related to licensing issues. The “esp_server.log” always has a message about a license check:
2013-08-14 19:46:33.974 | 18122 | container | [SP-3-100005] (0.000) sp(18122) —— License check started ——
2013-08-14 19:46:34.004 | 18122 | container | [SP-3-100005] (0.030) sp(18122) —— License check succeeded ——
This message is a bit misleading. It might give you the impression that all is well with your license but all it really tells us is that it ran a license check.
If you look at the “stdstreams.log”, all might not be so well:
2013/08/14 19:46:34 Warning (131254): Checked out graced license for 1 ESP_CORE (2013.0417) will expire Fri 23 Aug 2013 12:00:00 AM UTC.
2013/08/14 19:46:34 Error (131302): Failed to obtain license(s) for ESP_CORE feature from license file(s) or server(s).
2013/08/14 19:46:34 Error (131073): Cannot find license file.
2013/08/14 19:46:34 Error (0): The license files (or license server system network addresses) attempted are listed below.
2013/08/14 19:46:34 Error (0): License feature name: ESP_CORE
2013/08/14 19:46:34 Error (0): License filename: /work/nstack/ESP51_SP2/SYSAM-2_0/licenses
2013/08/14 19:46:34 Error (0): License search path: /work/nstack/ESP51_SP2/SYSAM-2_0/licenses
2013/08/14 19:46:34 Error (0): FLEXnet Licensing error:-1,359. System Error: 2 “No such file or directory”
2013/08/14 19:46:34 Error (0): For further information, refer to the Sybase Software Asset Management website at http://www.sybase.com/sysam
Immediately I can spot a few problems:
- It checked out a “graced license” which means it is a temporary license (normally good for 30 days).
- It expires in a few days. I need to get a new license or fix the problem with the license I have or else I won’t be able to run any projects.
- Error 131073 “Cannot find license file” is the underlying reason for it being unable to checkout a license.
- It tells me exactly where it was looking for the licenses.
So if your project ever fails to start, be sure to look at both the “esp_server.log” and the “stdstreams.log”.
Hopefully this blog has helped take out some of the mysteries of ESP server error logs and given you a start on where to look when you want to troubleshoot an ESP problem.