Skip to Content
Author's profile photo Jeff Wootton
Jeff Wootton
January 16, 2014 Less than a 1 minute read

How to simulate event streams

I had a great question today about whether there’s an easy way to simulate event streams for doing a proof of concept or for testing.  Yes, in fact, there are several ways to do this:

  1. Use the ESP random tuples generator
  2. Build a datafile in either csv or xml format supported by either the file input adapter for playback tool and then it will read the data from the file and stream it into ESP.  The advantage of using the playback tool is that it can play it at a controlled rate, whereas the file input adapter will just read it in as fast as possible.  The playback tool supports csv, so it’s easy to use Excel to create a sample data file (and save it as .csv) or you could write a script to generate sample data  (be sure to understand the data format supported by the playback tool before you start building your data set)
  3. The playback tool can also playback data in a database.  So you could use a data generation tool (like this one) to build some test data in a DB and then use the ESP playback tool to play it as a simulated stream

Assigned Tags

      1 Comment
      You must be Logged on to comment or reply to a post.
      Author's profile photo Jeff Wootton
      Jeff Wootton
      Blog Post Author

      I realize now that in this post I forgot to mention another useful option that we occasionally use:  build a streaming project to use as a data source.

      Here's what I mean:

      Build a project in CCL.  You will need an input stream, even if nothing feeds it. But then add a Flex Operator with a timer. The timer can trigger the output - set it to whatever frequency you want. And then, within the flex stream, you can construct the events. It can be as simple or sophisticated as you want.  You can use the built-in random() function to add random "jitter".  You can use variables to cycle through values.

      And then if you want to get more sophisticated, you can use that input stream to accept input events from an application or even just the manual input tool in Studio, to let the user change certain conditions that control the values of events being produced.

      Then run this project and it sits there generating events.

      Now create your "real" event processing project and just use an input binding to connect to the project that is the event source