Some Thoughts on Big Data

At the forefront of Big Data technology is the financial services industry, especially firms engaged in electronic trading, according to Sybase’s Neil McGovern in High Frequency Traders. As other verticals develop Big Data tech, capital markets firms must work to stay in the lead.

Big Data is technology’s hype phrase du jour, bandied incessantly about by analysts and the media. Every vendor seems to have a Big Data product, position or solution.

But what does it mean in capital markets?

It might mean we get to be a bit smug. We have been the leaders in Big Data for several years, and everybody else is simply playing catch up.

“Alright, Neil,” I’m sure you’re thinking, “you’ve put us on a pedestal, now defend it.”

First, let’s look at the patterns in the Big Data talk from analysts and vendors. A large picture emerges that might not be clear from just looking at one analyst or vendor because each has its own area of focus.

But after talking to participants in the financial service industry and reading an array of marketing content from many sources, I have concluded the following:

1.    Big Data is about lots of data. Buckets of it. Zillabytes, Yadabytes, even Kermitbytes! (sorry – my kids watched The Muppet Movie last night).
2.    Big Data is about unstructured data. All the data that you can’t be bothered putting into structured format – this is its new home!
3.    Big Data is about finding value in that sea of unstructured data. That means performing big-time analysis on lots and lots of data to learn precious lessons for your business.
4.    Big Data is real time. Once you have found something interesting, you can leverage it right away, making your business more agile and ready to pounce on opportunity.

This all sounds epic and intimidating, but the financial services industry has an impressive track record of depending on Big Data and using it with aplomb. What does this sound like?

1.    Market data volumes are really big. Capital markets firms have used advanced technology for years to store decades of market data for their quants. More than one third of respondents in a Sybase survey indicated that they required more than a decade of market data for analysis and back-testing.
2.    Finding value in the data is what quants are all about. Devising trading strategies from market analysis has developed from simple strategies, such as paired models in the 1980s, to the intricate gaming strategies of today.
3.    Trading strategies that include unstructured data (a fun example is Twitter-based trading strategies) are increasingly important. In 2012 we expect to see strong growth in the use of unstructured data in alpha discovery and trading strategies.
4.    Real-time execution is taken to the physical extremes by collocated high-frequency trading. You don’t get much more real-time than that.

So as an industry, we’ve earned some time in smug mode, watching all of the other verticals catch up with us. But those other verticals will not have to develop this technology as we did because it already exists. They won’t have to reinvent the wheel; they can just learn by watching us.

Big Data’s rising eminence might mean that we can kick back and be smug while they catch up. Or it could mean that we can keep our noses to the grindstone and remain this technology’s vanguard.

The choice is ours.