Friday, December 21, 2007

Drools Extensible Process Definition Language (ePDL) and the Semantic Module Framework (SMF)

Old school Drools 2.x users might remember the Semantic Module Framework and how it would allow for domain specific XML representations, such as the House Example DSL.

One of the issues with the SMF in Drools 2.0 was that rules are too hard to build programmatically, so the complexity was great. In contrast process nodes are trivial to build and wire together, so I have decided to resurrect the SMF for our process language; actually I have done this for rules too so they both using the same framwork, but I'm not expecting users to extend the rules side. I have forward ported the XML framework from Drools 2.0 and simplified it.

The parser creates DOM like fragments which are processed by builder handlers. The parser also maintains a parent hierarchy back to the root node. The SemanticModule is a configuration of handlers for elements in a given namespace (uri). We keep a registry called SemanticModules, with the uri as a key for each SematnicModule. The code to retrieve a SemanticModule is:
public SemanticModule getSemanticModule(String uri) {
return this.modules.get( uri );
}
The SemanticModules can be built programmaticaly and/or via discovered configuration files. The default process model handlers are already in the SemanticModules registry, via concrete implementations RulesSemanticModule and ProcessSemanticModule, discovered modules are wired together using the DefaultSemanticModule class.

For each xml element the parsers reads in it looks up the Handler, by first getting the SemanticModule for the namespace and then retrieving the handler from the SemanticModule with the getHandler(...) method:
public Handler getHandler(String name) {
return this.handlers.get( name );
}
So how does a Handler work? It has two methods, start(...) and end(...), called when the parsers enters and exits the element; inbetween which a DOM like fragment is built. The start method has access to the element attributes and the end method has access to the DOM like fragment that was built.

The start method initialises the DOM fragment build process using the startConfiguration() method:
reader.startConfiguration( localName, attrs );
The end method finalises the returns the DOM fragment using the endConfiguration() method:
Configuration config = reader.endConfiguration();
Lets look at a complete start method now for an ActionNode:
public Object start(String uri, String localName,
Attributes attrs, ExtensibleXmlParser reader) throws SAXException {
xmlPackageReader.startConfiguration( localName, attrs );
RuleFlowProcessImpl process = ( RuleFlowProcessImpl) reader.getParent();
ActionNodeImpl actionNode = new ActionNodeImpl();
actionNode.setName( attrs.getValue( "name" ) );
process.addNode( actionNode );
((ProcessBuildData)reader.getData()).addNode( actionNode );
return actionNode;
}
Above you can see we start the configuration for the DOM fragment and create an ActionNode setting its properties; note we don't yet set the content body, as we don't have that information yet, this is created end the end method:
public Object end(String uri, String localName,
ExtensibleXmlParser reader) throws SAXException {
Configuration config = reader.endConfiguration();
RuleFlowProcessImpl process = ( RuleFlowProcessImpl ) reader.getParent();
ActionNodeImpl actionNode = ( ActionNodeImpl ) xmlPackageReader.getCurrent();
actionNode.setAction( "logger.Warn(" + config.getText() + ")") );
return actionNode;
}
Both properties files and programmatic apis can be used to register handlers for a SemanticModule. For instance if we wanted to programmatically register a logger handler we would do it as follows:
SemanticModule module = new DefaultSemanticModule( "http://domain/org/mydsl" );
module.addHandler( "logger", new LoggerHandler() );
Or we can do it via properties file:
uri=http://domain/org/mydsl
store=org.drools.xml.StoreHandler
Note to use a properties file the PackageBuilderConfiguration needs to be told about these configurations:
semanticModules = mydsl.conf
So the above LoggerHandler and its registration allows for domain specific XML like below:
<process  name="process name" id="process name" package-name="org.domain"
xmlns="http://drools.org/drools-4.0/process"
xmlns:mydsl="http://domain/org/mydsl"
xmlns:xs="http://www.w3.org/2001/XMLSchema-instance"
xs:schemaLocation="http://drools.org/drools-4.0/process drools-processes-4.0.xsd" >

<nodes>
<start name="start node" />

<action name="action node" dialect="java">
list.add( "action node was here" );
</action>

<mydsl:logger name="test logger" type="warn">
This is my message
<mydsl:logger>

<end name="end node" />
</nodes>

<connections>
<connection from="start node" to="action node" />
<connection from="action node" to="test logger" />
<connection from="test logger" to="end node" />
</connections>

</process>

Thursday, December 20, 2007

When to use Rules and Processes and their overlap

Paul Haley, founder and chairman of haley.com, has just written a very in depth blog covering rules and processes and when to use which and the various ways they overlap. It's pretty much a "must read" for people trying to understand this space; heh, and he mentions our work in this efforts at the end :)
Business Rules Process Management

Modify Block for Java Dialect

In one more post in the series: "What are you guys doing right now?", :) we just added support to modify block in Java dialect consequences.

This is a feature that was already supported by MVEL and we decided to port it to Java so that it would benefit everyone, independently of the chosen dialect.

Basically, a modify block is a block of code that allow attributes of a given fact to be changed in the consequence of your rules, even with shadow facts turned off. In other words, if you don't want to pay for the shadow fact performance and memory cost, you can disabled them as long as you only change fact attributes inside modify blocks.

Example:

rule "Eat Cheese"
when
$p: Person( status == "hungry" )
$c: Cheese( )
then
retract( $c );
modify( $p ) {
setStatus( "full" ),
setAge( $p.getAge() + 1 )
}
end

So, inside of the modify block, you may write a comma separated list of Java expressions (set attribute values, call methods, etc) that are applied to the object scoped to that modify call. All of them are executed in sequence and inside a single fact update scope.

The grammar pseudo BNF looks like this:

modify ::= 'modify' '(' <expression> ')' '{' [ <expression> [ ',' <expression> ]* ] '}'

This feature was added both to trunk and 4.0.x branches and will be released in Drools 4.0.4 and Drools 4.1.0.

Happy drooling,
Edson

Tuesday, December 18, 2007

Drools and JBoss World

JBoss World is on for February the 13th 2008 at Orlando. I'll be doing a Drools talk there and I'll also be on Bob McWhirter's OSS round table, where we will talk about successful strategies for OSS communities.

Monday, December 17, 2007

Drools Javapolis BOF on our Business Logic Platform

I did a Drools BOF at Javapolis, where I presented what I believe is the future :) our vision on a unified and integrated business logic platform. You won't go to separate proprietary vendors for rules, processes and cep - each with their own runtime, api and server side management/deployment; your business logic is too important to be tied into closed source proprietary frameworks. We will provide the end users with a single platform with fully integrated rules, processes and cep with a single runtime, api and server side management/deployment. I have made available the slides from the presentation here, which are at a technical level, and you'll see this delivered in phases of 2008.

Friday, December 14, 2007

Stream support in Drools - a glimpse

As some of you, our users, already know, we are working on a series of features for CEP and ESP support for the next major release of Drools. We weren't talking much about it (busy researching and writing code), but I think it is time to share with you a little bit of what we developed on the Stream support side.

The first thing to understand is the concept of an "Entry Point".

When we work with streams, where the volume of data is usually huge, it is mandatory that we develop a way of scoping where that data will evaluated. Another mandatory requirement is the ability to effectively work with multiple streams in parallel, and for that, we need to effectively parallelize parts of the network, so that we don't create bottlenecks among different streams.

So an "Entry Point" is a channel through which you can assert data into the engine. All facts asserted through one entry point are only visible by patterns associated with that entry point and all entry points are independent of each other, thread safe and can run in parallel.

So, to illustrate the idea, nothing better than a simple rule:


import event org.drools.StockTradeEvent;

rule "Correlate Trade Order"
when
Customer( $id : id )
StockTradeEvent( customer == $id, $asset : asset ) from entry-point HomeBrokerStream
StockTradeEvent( asset == $asset, status == Trade.CONFIRMED ) from entry-point StockExchangeStream
then
// correlate events
end


So, in the above example, you have one single rule correlating events from the working memory (Customer) and from 2 different streams (HomeBrokerStream and StockExchangeStream) at the same time it keeps each of the streams processing in parallel.

The good thing about the implementation we did is that we leverage the characteristics of RETE to support the streams, like node sharing, indexing etc. So, if you have multiple rules with patterns that are associated with the same streams, the patterns will share nodes, index facts, etc transparently.

Another interesting characteristics is that we handle heterogeneous streams. I.e., each stream is not limited to a single type of event/fact and this is all transparent to the user. The engine select the events/facts from the stream based on the patterns the rules are looking for.

From a code perspective, all the user needs to do is get the entry point interface from the session for each of his streams and it to insert the events/facts:


StatefulSession session = ruleBase.newStatefulSession();

// one thread may do this:
EntryPointInterface broker = session.getEntryPoint( "HomeBrokerStream" );
// insert each event as usual
broker.insert( event );

// another thread may do this:
EntryPointInterface stock = session.getEntryPoint( "StockExchangeStream" );
// insert each event as usual
stock.insert( event );


There is still a lot to do, but we are getting there.

Happy drooling,
Edson

Wednesday, December 12, 2007

Drools solver Javapolis 2007 slides

The BOF was a success and as promised, I 've posted the slides online:
People asked a bunch of interesting questions afterwards.
One question I felt I left a bit unanswered: how do you weigh your constraints in DRL? So, here's a code example that should answer that question too:

rule "twoExamsInARow"
when
// ...
then
insertLogical(new IntConstraintOccurrence(
"twoExamsInARow",
10, // weight
$leftExam, $rightExam));
end

rule "twoExamsInADay"
when
// ...
then
insertLogical(new IntConstraintOccurrence(
"
twoExamsInADay",
1, // weight
$leftExam, $rightExam));
end

rule "softConstraintsBroken"
when
$total : Double() from accumulate(
IntConstraintOccurrence($weight : weight),
sum($
weight)
);
then
scoreCalculator.setSoftConstraintsBroken($total.intValue());
end

So the 2 in a row constraint is 10 times heavier than the 2 in a day constraint.

Friday, December 07, 2007

Drools November Download Stats

For 3 consecutive months in a row we've had approximately 20K download attempts for -bin.zip, what is interesting though is that the BRMS has a lot more download attempts this month, likewise Eclipse, especially if you combine the eclipse3.3 and eclipse3.2 download attempts; that's almost 22K eclipse download attempts. so over all I'm really pleased :)

20230 drools-4.0.3-bin.zip
14822 drools-4.0.3-brms.zip
14297 drools-4.0.3-eclipse3.3.zip
13477 drools-4.0.3-src.zip
8313 drools-4.0.3-examples.zip
7501 drools-4.0.3-eclipse3.2.zip


Let's hope that the features we have planned in 4.1.0 in Q1 will help us break that 20K barrier for the -bins :)

Thursday, December 06, 2007

Life and Death, Rules and RuleFlow

Yesterday I told a friend what I do for a living, and she's still in shock. I don't think the words 'grabbing people's brains and shoving them into a PC' was the most tactful explanation. I wouldn't recommend explaining Rules and RuleFlow to your boss in those terms either. Unless they already think you're some sort of Frankenstein and you have best friend called Igor.

A better way might be to use the workflow example from the recent IJTC conference. It explains the importance of Rules and Ruleflow in a way that even your boss can understand. Strangely enough for a Drools blogpost, you don't even need a computer to understand it;

The Health services in Bangladesh (like many elsewhere) can't get enough doctors. Training more is not a solution ; those that do qualify tend to leave for higher rates of pay elsewhere. So, given the desparate need for trained medical staff in rural areas(to curb child mortality rates), what are health workers to do?

The solution that the Health workers came up with was IMCI - or Integrated Management of Childhood Illness. It takes what the Knowledge in Doctor's head's and writes it down as a simple guide that health workers can follow. When a sick child is brought into the remote clinic the health worker is able to follow the simple step-by-step instructions to make quite a sophisticated diagnosis.

I've no medical training beyond simple CPR (and if you're relying on that then you're in real trouble) but even I can understand it.

Look at the pale blue box. It's a set of medical rules: Are there any danger signs? What are the main symptoms? What combination of these symptoms are there? What is the age of the child? How long have they been ill? Depending on the outcome of the rules, go to the next set (the pink / yellow /green) boxes and apply the rules that you find there.

That's Rules and RuleFlow.

  • Rules are 'when something is present , then do this'. And not just single rules, but many of them. Together, loads of simple rules allow you to come up with quite a sophisticated diagnosis.

  • Ruleflow allows you to group your rules. If you're a health worker with a sick child you want to do the most important checks first. Depending on the outcome, you then apply the next set of medical rules: Pink if they need urgent referral to the hospital, yellow if the clinic can cope , or green if the child can be looked after at home.
As gory as it sounds, everybody, including the doctors, are happy that their 'brains have been put into a PC' (or in this case , a set of paper cards). The Doctors are happy because they can (guilt free) move to better paying jobs. The medical workers using the system are happy because they can help the sick children that they see every day. And the children gain because the better availability of medical knowledge (via Rules and RuleFlow) is literally the difference between life and death.

Paul Browne also writes on the People and Technology blog.

Testing rules - introduction

Quite some time ago, I created the fit-for-rules project, which was built using the FIT framework (Framework for Integrated Tests). I was quite a fan of fit - however for reasons unknown it has not yet taken off in the mainstream (perhaps its just too strange, I think its brilliant, but then I like egg nogg lattes at Christmas - no idea why, its not something I would normally like at all).

Recently we have been working on an integrated testing tool to encourage people to write tests for rules the TDD people do for code. This tool, still doesn't have a name, so feel free to suggest some names. In terms of the tool itself, I spent some time looking into BDD (behavioral driven development - not the disease), and tried to adopt the BDD nomclamenture.

So, the basic test item is a Scenario. A scenario is essentially a sequence of data followed by expectations (ie in a given scenario, expect things). Expectations can be data, or rules firing etc.

Its not that different to a rule:

Given some data -> Then fire the rules -> Expect some results -> Modify some data -> Fire the rules -> Expect some results.

You may only want one given/expect cycle, but as rule session are (can be) stateful, this will allow it. Well a picture is worth a thousand words:

You can see it looks not unlike the guided editor. You set up facts, and then simulate rules firing (actually they do fire, unless you tell it to not fire certain rules). You can then check the data and rules after the fact (rinse, repeat). These tests are just as important as the rules, and will be stored as such.



Simulating a date is optional, but handy as often rules are dependent on time.

Suggestions for a name for this?

Wednesday, December 05, 2007

Rule Constraints and Code Constraints now works for Splits

Recently I got the Drools dialect system working for processes, as discussed in this previous blog article. The dialect system is a pluggable framework to allow code to be written with different languages. Previously Split constraints would only work with rules, so the next stage was naturally to allow for constraints to work with code, like traditional process engines - ofcourse the code constraint would also work with the dialect system. Below is a screenshot of a Split node with both MVEL and Java code constraints and MVEL and Java rule constraints.

Tuesday, December 04, 2007

Showcase: Chooz-it (choosing the right technology)

Paul Smith, a Drools user and long time mailing list lurker, has launched a new service that he has been cooking up for some time : www.chooz-it.com.au.

Choose it is a web site/service that connects suppliers with consumers (particular for service industries). It allows suppliers to provide their availability, and also does bookings/fulfillment. What this means is if you offer say a dog washing service, you can provide this service through chooze-it, and people can book your services (according to your availability) and you won't be double booked, payment dealt with etc. Obviously there is much more to it then this, so see below for a more detailed explanation of Chooze-it in Paul's own words:

Choozing the right technology


A little under two years ago I set out to build a system that would allow for users to search for service suppliers and allow those suppliers to expose their services, availabilities and take bookings for those availabilities. The system had to be generic enough to encompass as many businesses as possible and needed to be built on very solid foundations as scalability was going to be one of the top requirements. I also wanted to showcase some of the latest ‘bells and whistles’ that the Java/JEE world had to offer. Early in the piece I had been tackling the issue of how to separate the business logic so that it became a nice separate pluggable module leaving the rest of my persistence/service code relatively unpolluted. Drools was then in its early days being just before version 3 was released. I played with it and straight away realised that this had some seriously powerful implications. The use of a logic or rules engine separated logic nicely and the inclusion of domain specific languages made the rules very easy to read and understand. Back then the Eclipse plugin wasn’t yet available so debugging was a pain but not insurmountable. All of the logic became easily unit testable because I separated it totally from the database. This left the cactus tests free to just test that things actually get put in the database. Another relatively new player was EJB3 and JPA. Having gone through the pain of EJB2 but not yet confident enough to use Spring, I opted to give the EJB3 path a go as there was good support already in JBoss AS 4. It had a few initial teething troubles (mainly down to the spec not being finalised) but on the whole it was very stable and easy to use. No config files (hallelujah), just annotate the files and let JBoss do its thing at deploy time. The session facades worked well too. Once I’d ironed out the security domain issues it worked a treat. This gave me the framework to build a high performance service layer. It’s broadly speaking based on SOA because I wanted to allow for the possibility of integration with PC based booking systems. This is for later development but I’d like to allow for the transfer of booking and availability information between the online system and the customers booking system. So a service architecture was a definite must have.


Notifications came into the picture sometime later in development because I needed to have guaranteed email and sms delivery. This came right at the time when JBoss MQ was on the way out and JBoss Messaging was coming in, so naturally I opted to go with the latest and greatest despite the pain that generally comes along with this. The results were not favourable initially as for some reason, the queue JNDI lookups kept failing. Everything looked to be configured correctly as per the doco but it just wouldn’t work. I raised a Jira issue but didn’t get a resolution. It took til version 4.21GA of JBoss AS before I got it working. Somebody fixed something in that version of the code and it all started working like a charm. When the messaging descriptor is configured correctly JBossAS it just picks up on the create SQL and generates the tables if they aren’t already there. I’m confident it will work well because when I first started building the server, the SMTP config was wrong and nothing was happening but as soon I fixed the config all the emails etc just got picked up from the queue and sent out when the app server came back up.


The front end isn’t really that flash. It’s struts tiles utilising a smattering of Ajax. Nothing really new there but we do have a little bit of stuff to interact with the Google Calendar API. I used the sslext tags to handle the changes from HTTP to HTTPS and back again. It generally seems to work well but has got a couple of quirks.


Functionality wise, the system offers suppliers the ability to signup and configure their business details, services and service availabilities. Users can come in and browse for suppliers, look at their service offerings and check for availabilities. Once logged in they can either book a specific availability or make a booking enquiry for a service. Depending on the supplier configuration for the service they can also pay the deposit for the service. The booking can then be managed from both the customer and supplier perspective eg) the status can be confirmed or the customer can request cancellation. Manual bookings taken by phone can also be entered. There is also the facility for employees to be configured and added to teams. Each service can then have a team of service providers or just one provider. Automatic invoicing was built in to trigger on a monthly basis so the supplier can look at their invoice as a .pdf and then pay it online. I used itext to generate the .pdf dynamically when they click on the link. It’s pretty quick but the API is a bit clunky. I also implemented a generic payment gateway interface over the gateway suppliers API that allows me to easily change payment gateway providers if I need to. I could have just used PayPal or something similar but I didn’t like the idea of having the users go outside of the site.


It’s easy to make technology decisions based on how new something is or how great it might sound to say that it’s being used. The proof of how good the technology is really doesn’t happen until it makes it to production, and doesn’t fall over, but up until now I’ve been very happy with my technology choices.


So that, in a nutshell, is www.chooz-it.com.au and the technologies behind it.

Saturday, December 01, 2007

The Missing BRMS tutorial

James Childers (of the macjavadev blog) has helpfully come up with a (slightly humorous) Real Quickstart guide (I like to call it the "missing manual"). Its good advice.

We do have plans to have more 5, 10 minute tutorials. Of course to save some work, we are changing some aspects of the user interface to try and get rid of the annoying confusing bits (especially around package management, and navigation). So with the BRMSv2, expect to see more of these "ease into it" style tutorials (I was quite fond of the olde codehaus 1, 2 and 5 minute tutorials, it was what got me into drools in the first place).

In the mean time, sit back, relax, or something, and take a look at the following.

Quickstart: Drools 4.0.3 Web Admin (BRMS)

The documentation on Drools is a little bit light on the "where the hell do I start" part, so here's a quickstart guide. I'll walk you through getting the equivalent of a "Hello, World" app up in Drools, i.e. the simplest setup possible. This will use the Drools web management interface, and will not require any actual code whatsoever. This assumes that you, much like myself, in fact know next to nothing about Drools, and just want to play with it before getting into Rete algorithms and truth maintenance and all that crap.

Isn't that nice.

There are only three requirements.

  1. You have your own servlet container: Tomcat, Glassfish, whatever.

  2. You have in your possession a JavaBean in some .jar flie somewhere. Any bean will do.

  3. You don't require screenshots. Sorry, I'm lazy. Maybe in the future.

That's it.

  1. Download the Drools BRMS from JBoss.

  2. Deploy it to your app server. I like Glassfish.
    [0932][jchilders@Ivan:~/webapps]$ unzip drools-4.0.3-brms.zip
    [0932][jchilders@Ivan:~/webapps]$ sudo asadmin
    Use "exit" to exit and "help" for online help.
    asadmin> start-domain domain1
    asadmin> deploydir ./drools-brms/

    Log in to it. (Mine deployed to http://localhost:8080/drools-jbrms/). User ID is "", password is "", without the quotes.

  3. You should see five options on the left: Info, Rules, Packages, Deployment, and Admin. Click on Admin.

  4. Click on "Create New Category".

  5. Name your category something clever. We'll call ours "BannerAds". Save it.

  6. Click on "Packages". There is a button on this page that will allow you to create a new package. It is small and has no label, but does have a tooltip. Click on it.

  7. Give your package a name and optional description. We'll call ours "MyPackage". Save it.

  8. There is another tiny, hard-to-find button on this page that allows you to create a new model. Click on it.

  9. This model will reference the jar file that has your bean it in I mentioned earlier. Name it something witty like "MyBean" and save it.

  10. You should be at a screen that will let you upload your jar. Do it.

  11. Click on the "Save Changes" button.

  12. Holy crap we're almost there! Now click on the "MyPackage" package in the tree, then "Edit Package Configuration" in the main window.

  13. In the "Header" section in the main window you need to actually tell Drools about your bean. To do so:
    import com.mycompany.MyBean

    No semicolon at the end. If you want to get crazy and are lucky enough to own more than one JavaBean, do more than one import.

  14. Wow we should actually be able to make a rule now! AWESOME!

  15. Click on the massive 16px button that will let you "Create New Rule"

  16. Give it a name and category and type it as "Business rule". Save.

  17. Ok, now comes the real test. Click on the big green plus sign to the right of "WHEN". You should see a window that says "Add a condition to the rule..." If everything was set up correctly -- and that's a big if -- then you should see two selects: "Fact", and "Condition Type". If all it says is "Add a condition to the rule..." and nothing else, then you screwed up. Also these instructions may be wrong. It is much more likely that you screwed up because I have a blog. If it doesn't work please go to step RTFM, else continue.

  18. Fact. Click on it. Select your bean.

  19. Figure out the rest on your own. You're now to the point where you can create rules based upon your bean. Doing so using the visual editor is pretty self-explanatory.