Monday, November 26, 2007

Pigeons, Complex Event Processing and how to make millions with JBoss Drools

I'm still waiting for the JBoss bouncers to hand me my coat and ask me to leave this blog. Mark gets to talk about Unifying Rules and Processes. Fernando and Michael are very proud of the 2nd version of the Business Rules WebApp . And I get to talk about pigeons. Yep, Pigeons; birds that fly, sometimes useful for carrying messages and have one hidden talent.

During the cold war, the Soviets (allegedly) trained pigeons to inspect ball-bearings on the production line . The Pigeons would sit in comfortable little boxes while the shiny silver ball bearings steamed past on a conveyor belt. If the pigeon spotted any that were defective, they would peck a button and the broken bearing was gone. Since the fall of the Berlin wall, all the pigeons have been gainfully re-employed over at Google.

Thankfully the pigeons didn't go to work at a Bank in the City (have you ever seen anything with feathers drive a Ferrari?) . While the pigeons would be very good at responding to simple market events events (Market up , sell; Market Down , Buy). more complex analysis escapes them; For example ; if the market is down for the 30 mins, and Shares in Acme corp are down more than 10% than the average ,and I have seen 3 buy orders for that share in the last 60 seconds = I think the market is about to turn = buy shares in Acme corp.

Never mind pigeons; most humans would find that difficult - think about trying to read the stock ticker prices (the ones you see rolling across the screen at MSNBC) for all stocks, while trying to hold the buy and sell information for the last 30 minutes in your head. And do that not only for one , but for the couple of hundred different types of shares in the market. And while keeping an eye on your own trading position so that you're not exposed to one sector of the market (e.g. keeping enough cash , not too many property or technology shares. No wonder most traders make their millions and burn out before they're 30 - that sort of Complex Event Processing (CEP) will wear you out.

Most IT applications are like pigeons; they can only handle simple events. Press Button. Do something.

The way to make millions is to design applications that can handle these complex events, and apply sophisticated business rules to the (evolving) situation. And do it quickly enough (milliseconds) to seize the opportunity before somebody else does. A keep on doing it as long as the market is open.

Funnily enough, Complex Event Processing is part of the vision for Drools. With enough support, I'm sure we could convince Mark to stand up at JavaPolis and use a set of Pigeons on his slides. I suppose it's better than using pictures of lego people to explain how to do projects using Agile.


  1. How will Drools compare to BEA's WebLogic Event Server? What do you think will be the differentiators when you do deliver this capability?

    Kind regards,

    William Louth
    JXInsight Product Architect

  2. BEA's WebLogic Event Server is an OEM'd version of Esper Which uses a streaming sql type language.

    Our's will be an orthogonal extension of our existing rule language aimed at supporting temporal logic constructs, although we'll provide a streaming sql parser too (as it's a subset).

  3. I spent a little time studying the query compilation in Esper and it looks rather simple. If I understand the code correctly, it doesn't generate efficient query plans. the other limitation I see with esper is that performance will drop significantly if the rule count increases.

    The temporal extensions I've blogged about can handle a broader range of temporal logic. I've already implemented a simple version of temporal logic in jamocha. The code is up on sourceforge. Hopefully edson can use that a point of reference and implement the same thing for drools. I also described an aggregate pattern extension for RETE, which is much more powerful than streamSql. It's similar to the concept of cubes in MOLAP.

  4. There are a number of limitations with Esper and other low end systems, the most obvious of which is the limitation on the clock representation. Clock representations are to do with representing the time an event happened, was it when it entered the network, is there a threshold, is it progammatic. We are designing our system to support a number of clock representations. We are also looking into interval based event representations; which are bit like composite events that are made up of a number of smaller interval events (i.e. start, stop). Over all we are aiming at doing more complex temporal reasoning, so our scope is much bigger than that of streaming sql.

    We'll be unveiling our vision on this with a usable prototype in January. We won't have everything done by then, but we'll have an initial base implementation and a roadmap.

  5. PigeonRank is .html, not .htm -- broken link.

  6. Bitcoin HYIP - Invested $300 and Paid $900 in one hour
    Giant-investment is a private investment group
    Test Plan * 300% after one hour
    Standard Plan * 500% after one hour
    Premium Plan * 800% after one hour
    Invest Now
    Principal Guaranteed