In a the series titled "A Deeper Look At Sybase", this installment is about Sybase CEP. This product is also known as 'Sybase ESP', or 'Sybase Aleri Streaming Platform', for reasons I'll explain below.
First, CEP is short for Complex Event Processing. This describes an event-driven application model whereby an incoming event must be responded to within a very short time, typically milliseconds. The incoming events are sent by some other system and arrive at the application at a high rate. To process an event, input from other data sources, typically located in databases, also need to be taken into account.
If that all sounds a bit vague, think of automated stock trading systems as operated by big banks: such systems make split-second decisions about buying or selling securities based on real-time events, such as change patterns in the prices of certain stock. Many aspects need to be taken into account before deciding on the next buy-or-sell action, not in the last place the impact on the bank's risk position (as the various types of risk involved have become decidedly more important ever since the credit crisis broke out in 2008).
Sybase CEP -you guessed it- is a platform for implementing such Complex Event Processing systems, and is considered the leading CEP platform on the market today.
Functionally speaking, there is no reason why you couldn't implement such systems in a classic SQL relational database. However, the density of the incoming event stream is typically much too high, and the allowed latency much too low, to first store the data in an RDBMS like ASE or Sybase IQ and then kick off some SQL code that reads a database table to figure out what events happened recently. Such a polling-based approach is simply too slow to be able to respond within milliseconds.
For this reason, a CEP engine -also known as a 'stream query engine'- operates on a stream of incoming data rather than on a data already stored in a database.
Despite the difference with classic SQL-based database processing, there are some similarities too. Sybase CEP provides CCL (Continuous Computation Language) as its API. CCL has many similarities with SQL but operates in a stream environment. For example, the following piece of CCL code captures the highest price of SAP stock over every period of 15 minutes:
INSERT INTO MaxPrice
SELECT Trades.Symbol, MAX (Trades.Price)
FROM Trades KEEP 15 MINUTES
WHERE Trades.Symbol = 'SAP'
In this CCL code, 'Trades' corresponds to a stream of incoming events, presumably a stock market ticker feed. The SELECT query is automatically evaluated for every incoming SAP stock trade to record the maximum stock price over the past 15 minutes (which is then written into MaxPrice, another stream, which could in turn be written into a database table). In CEP lingo, this is dubbed 'continuous querying'.
As you can see, this CCL query has a strong SQL look-and-feel. The main conceptual difference is that in classic SQL database environment, the data being queried is rather constant, but the precise queries against that data vary strongly. In a CEP environment, the data being queried changes constantly and at very high speeds, but the queries against the data remain mostly constant.
The CCL API makes it easier to develop the application logic for a CEP system than with the original Aleri scripting API. This matters, because the competitive advantage that these financial customers gain from using CEP technology fully lies in the way they program the CEP engine. In other words: the guy with the smartest CEP algorithms will make more profits on the stock exchange.
This brings me to perhaps the most attractive aspect of Sybase CEP: the flexibility of developing application logic and quickly deploying this to a production system. Before CEP engines came along, this type of system had to be written from the ground up in C/C++, essentially hard-coding the business logic, which makes app development much slower and less flexible. Sybase CEP includes an Eclipse-based studio for efficient application development and debugging.
In this context, CCL's conceptual similarities with SQL are attractive since they make Sybase CEP easier to use for developers with SQL skills, thus reducing the learning curve.
Given Sybase's focus on the financial market segment, it should not come as a surprise that financial customers are the primary users of Sybase CEP, for purposes such as algorithmic trading and risk management. They are not the only ones though: when you think about high-density streams of events or data that need to be analysed in real-time, things like telco networks spring to mind. Indeed, there is increasing interest from telco operators to use CEP for things like network analysis.
I guess the possible applications do not need to stop here either: anything featuring high-density data/event streams and the need to analyse/act/alert in real-time, should consider using CEP (just thinking out loud here: what about using CEP for fraud detection, RFID, security/surveillance). In fact, it was Gartner who pointed out that, rather than looking at CEP only as a really fast stream query processor, it should really be seen as a real-time analytics tool.
A CEP-based system tends to require two main resources: CPU and memory. This makes sense since the CEP processing is all done in-memory in order to meet the low-latency requirement. Since CEP will need to pull data out of other data sources, such as reference data from an RDBMS, Sybase CEP comes equipped with adapters and interfaces for most things software that are worth talking to (including message buses, databases, third-party applications, various types of files, and more).
Recently a major new version of Sybase CEP was released. This product is also referred to as 'Sybase ESP' (Event Stream Processing) as well as 'Sybase Aleri Streaming Platform'. Despite the different names, these all refer to the same thing (CEP is more of a technologists' name, whereas the other two sound a little more marketing-inspired). In addition, you may also encounter the name 'Corel8' when reading about Sybase CEP.
To get some clarity in this terminology soup, here is some history. Back in 2008, Sybase partnered with Coral8, at the time one of various CEP vendors, to use its CEP engine in the Sybase RAP product (Risk Analytics Platform -- as you probably expected, a product used by financial customers doing securities trading). Later, Corel8 was acquired by Aleri, another CEP vendor, and started integrating the best parts of Corel8 into Aleri. In 2009, Sybase acquired Aleri, thus inheriting the integration project of those CEP engines. In mid-2011, this integration was completed and Sybase released its new CEP product, named 'Sybase Aleri Streaming Platform'.
What matters about all of this is that the new Sybase CEP release has the best of both worlds: Aleri had a superior CEP engine, but Corel8 had the better API with CCL (Aleri's API was based on a proprietary scripting language that was powerful, but not as easy to use). The new Sybase CEP release now combines the strong points of both.
For more information, visit http://www.sybase.com/cep.
Coming up next: rob.verschoor/blog/2011/12/22/a-deeper-look-at-sybase-column-encryption-in-sybase-ase.
1 Comment
