Monday, December 11, 2006

Free all day JBoss Rules BOF in London

I'm looking to do a free all day BOF in London, potentially Thursday 11th of January, 9.30 to 4.30 - with after session drinks, for those that want to chat some more over a beer and maybe some food? This will be an informal day for those with existing understandings of jboss rules, (we won't be explaining what a working memory is) and want to know more and also find out about whats coming up in 3.2 and we can all share our experiences and of course you can pump the core developers with hard Qs. We would love for volunteers to take 20-30 minutes to casually present their own projects and problems they have solved with jbrules, let me know if you are interested. Ideally I'd like to get between 10 and 20 people, please email me if you are interested. Micheal is leaving to go back to Australia mid january, so ideally it will be before then.

Mark
JBoss Rules Project Lead
Share/Bookmark

Sunday, December 10, 2006

Dynamically Generated Classes as JBoss Rules Facts (Edson Tirelli)

As most users know, JBoss Rules is designed to use standard POJO JavaBeans as facts. This design directive intends, among other things, to:

  • make integration faster: as there is no need to replicate your Business Model in an engine proprietary business model
  • improve performance: as the engine is design to work as optimized as possible when running over the POJO beans. Also, no need to copy attribute values between mapped structures.

In real world, though, supporting POJO beans may not be enough to achieve above goals.

As we all know, Java, as a platform, provides several ways of developing applications that put dynamic resources at good use. And this is using both platform standard features and a whole lot of tools out there in the internet, both open and proprietary.

For instance, we often are asked about "How to use dynamically generated class' beans as facts?". This usually happens in companies that create applications that allow users to define part or the whole of the business model without touching java code. These applications usually have an embedded rules engine, and you may want the engine to reason over these dynamically generated business models.

And the good news is: you can do that with JBoss Rules, and 3.0.5 made it even easier!

The only issue is to make sure you are using a tool to generate your beans that will truly generate standard JavaBeans. I mean, a tool that:

  1. allows you to state specifically what is the package and class name for the generated class: this is mandatory in order to write efficient rules, as the first constraint the engine will apply over your facts are the class type of the facts your rules will reason over
  2. generates a no-argument default constructor (as per the JavaBeans spec)
  3. correctly generates the getXXX() methods for your properties (as per the JavaBeans spec)
  4. ideally will allow you to define what are your bean's key properties and will automatically generate equals()/hashCode() methods for it using these properties (allowing for consistent reasoning based on equality instead of only identity)
  5. ideally will generate fast accessors for your properties, allowing your beans to be high performing

Most bean generation tools will allow that. Just to name a few open source tools (it is not intended to be a full list):

  • ASM: this is by far my preferred framework. I bit lower level than other tools, but allows you to do anything you want (you are writing bytecode after all), and it is as fast as one can be (from my experience).
  • BCEL: another popular framework from Apache.
  • CGLIB: a higher level framework that is used in a lot of projects.

Above frameworks allows you to create a dynamic POJO JavaBean Business Model and use it as your rules business model. Above is just a small list and I'm sure there are a lot of frameworks out there that allow you to do it.

Unfortunately, one also popular framework that fails to work well with JBoss Rules is Apache DynaBeans.
I'm not saying that DynaBeans are bad to all, as there are some use cases for it described in its documentation. What I'm saying is simply that DynaBeans are bad for integration with JBoss Rules. You can read more about it in Mark's post.

So, you have your Business Model (that happens to be dynamically generated), that complies at least with requirements 1-3 listed above, and you want to use it when writing business rules. What you need to do?

You simply must make your business model available for the engine. JBoss Rules will need your classes in 2 distinct situations, that may occur in sequence or not:

  1. Rules compilation
  2. Rules execution

Let’s talk about how to use them in each of the above steps.

It is important to note that all this is closely related to how Class Loaders work in the Java platform, but since you are using dynamically generated classes, I assume you have knowledge of java Class Loader architecture.

1. Rules Compilation

When you write a rules file, whatever the format you use (DRL, DSL, Decision Tables), JBoss Rules needs to compile it before using. When compiling a rule base, the classes you use as facts must be "available".

For instance, if you write a rule base like this:

package org.drools.sample;

import org.drools.sample.facts.Person;
import org.drools.sample.facts.Cheese;

rule "Likes Cheese"
when
Person( $likes : likes )
Cheese( type == $likes )
then
// do some stuff
end


You must have classes Person and Cheese "available" for compilation. The concept of "available" varies though according to the compiler you are using. Also, Person or Cheese or even both classes may be dynamically generated classes. The engine does not care about it, but the compiler certainly does.

JBoss Rules uses JCI (Java Compiler Interface) as an abstraction layer for the compiler layer. JBoss Rules is integrated and tested with two underling compilers: JDT (default) and JANINO.

They will generate the same results, but from a compilation requirements perspective, they are a bit different, so let’s talk about how to make compilation works with each of them.

1.1. Janino

Janino
is not the default compiler for JBoss Rules but you can activate it either by using a PackageBuilderConfiguration.setCompiler() method or by setting the "drools.compiler=JANINO" system property.

For JCI+Janino to compile your rule base, it is enough to have dynamically generated classes available into your context class loader. So, for instance, if your dynamically generated classes were loaded into the same ClassLoader that loaded your Main application class, only thing you need to do is to call, before creating your PackageBuilder:

Thread.currentThread().setContextClassLoader( Main.class.getClassLoader() );


This will ensure that your PackageBuilder will use the provided class loader to find the classes for compilation. That will obviously succeed as the given class loader is the one that loaded your dynamic classes.

1.2. JDT

JDT is the default compiler for JBoss Rules. If you don't set the "drools.compiler" property, nor change it using the PackageBuilderConfiguration.setCompiler() method, it will be used to compile your rules.

Although, JCI+JDT have an additional requirement: to compile your rule base, the context class loader must be able to provide the actual classes’ bytecode, not only the loaded classes. In other words, you must provide a custom class loader that can provide an input stream for the byte code of each class used in your rule base.

In the above rule example, your custom class loader must return the input stream to the byte code of Person and Cheese classes, does not matter if the classes were dynamically generated or not.

For dynamically generated classes, it is enough for the custom class loader to implement the method:

public InputStream getResourceAsStream(final String name);


I have written a simple example that has a ClassLoader that uses a simple HashMap to store dynamically generated classes bytecodes and return them when the appropriate getResourceAsStream method is called.

So, what you need to do is similar to what you do in Janino:

Thread.currentThread().setContextClassLoader( myCustomClassLoader );


But, your custom class loader must comply with the above requirement.

2. Rules Execution

Rules execution may happen immediately after rules compilation or not. For example, you may compile a rule base and serialize it at build time. At runtime you simply deserialize the rule base and execute it.

The only thing you need to do at execution time is to make sure the same ClassLoader hierarchy used to load your rule base is used to load your fact classes. Again, this is not related to dynamically generated classes, but the problem of multiple ClassLoader shows up more frequently when using dynamically generated classes, because this one of the situations when people usually use multiple ClassLoaders inside the same application.

Just to understand the problem, remember that in java, a class C loaded by loader L1 is different from the same class C loaded by loader L2. So, if you load above rule base with class loader L1 and assert into working memory a Cheese class instance loaded by class loader L2, your rule will never match and obviously never fire.

So make sure your rule base does not load your classes in a different class loader than your application is using to load it.

Example

I created a very simple example of JBoss Rules using dynamically generated classes as facts and committed it here. It is not intended to be a general purpose solution, but rather a didactic example of one possible solution.

I am using ASM as the bytecode generation framework and kept the API functional but simple in order to not overload the example.

The example uses a Customer bean that is a regular POJO class and two dynamically generated classes: Order and LineItem.

Feel free to drop questions you may have to the JBoss Rules users list.

[]s
Edson
Share/Bookmark

Friday, December 08, 2006

Rush Hour and Content Based Routing (Michael Neale)

Recently I (and Mark) attended JBoss World in Berlin. We ran a few sessions on rules, and had 2 excellent end user presentations, all was well received. Its good to see the cool stuff people are doing with rules, so I thought I would share a summary.

Content Based Routing - Michael Frandsen (Syngenio)
Content Based Routing (CBR) is used in messaging systems when you need to, well, route messages, based on, er, content ! Pretty obvious really ! Normally this is part (in some form) of messaging middleware or things like ESBs (enterprise service bus). In fact, its part of JBoss ESB (something I confess I am not overly familiar with, what with working on our own product, we don't get to look around at our stable mates that often, but I do know they use rules).

CBR is also known as "smart routing" and can be used for clever things like routing messages based on transaction value (ie higher value purchases get special treatment when load balancing) or based on things like who the customer is, the possibilities are endless.

Michael Frandsen (JSR-94 lead/contributor) has been working on CBR as part of his day job. He used rules (starting with Drools 2.5, moving up to 3.0.4) and XPath to provide declarative routing rules based on message content (which can be arbitrarily complex as needed, and perhaps involve large numbers of rules). Generally XPath or similar is used, as it provides a declarative way to make assertions over the *contents* of a message (normally the actions in routing is very simple, just telling it where to send the message !).

Michael's architecture on how the rule engine can reason over message contents for CBR.


The issue with XPath is that it is not known for its raw performance, but for IO and stateless things like message routing, you would think this is not really an issue. Couple this with the fact that a lot of the times the routing rules are minimal, I would have though XPath would shine compared to a rule engine for message routing.

Oh how wrong I was. Michael showed that even for the simplest cases, the smallest number of rules, that JBoss Rules blew away XPath implementations - this shocked me as normally rules shine with stateful scenarios (but all his message routing rules are stateless), but would not have as much advantage for stateless.

Moreover, when the complexity and number of rules scaled up (Michael pushed it to 1000 rules, which would be a LOT of rules for routing, but in some enterprises, it is possible) XPath was much slower then the rule based approach.

From this table you can see that JBoss Rules is at least 2 (sometimes 3) orders of magnitude faster then XPath. The nice scaling of throughput as the rule count increases also shows RETE really paying off. Note that this is for STATELESS CBR, and in no case is the rules approach slower, always faster. You can also see the jump from Drools 2.5 to Drools 3 (JBoss Rules) which re-inforces the power of full-RETE.


Using DRL+DSL for message routing also made for very attractive routing rules (none of this is using XPath):

You can see Michael's presentation here.

Traffic control (Leo van den Berg)

UPDATE: The presentation for this in PDF format can be downloaded from here.
Leo van den Berg, working with the University of Valencia and the Dutch ministry of transportation, has been working on a rule based traffic management system. We all know and love gridlock in our major cities, and what they are doing is building open source tools and applications for "traffic engineers" to use.

Traffic control systems do things like display messages on electronic boards, close and open lanes, traffic lights, accident reports, emergency response etc.

The architecture used uses JMS heavily to collect "event" data as asynchronous messages, and feed it (after some filtering) to the rule engine. The rule engine collects/reasons over this data, and they makes conclusions (included in Leo's demo was triggering an SMS warning to tell and engineer to open an optional lane).

Leo even had a cool live demo - using their thick client front end - which had awesome stuff like live video feeds from the relevant traffic cameras. Seeing a use case like this was great to see. I imagined it like a Giant Mind watching over us (well, those in Amsterdam etc at least). But you can relax, at the end of the day, its the traffic engineers (humans) who actually do the real button pushing/decisions based on the informations and suggestions given to them. So its not SkyNet just yet ;)

Hopefully I can show you some more of this application in a future post. The really nice thing is that what they are working on will be open source as well.


Ich bin ein Berliner !

Share/Bookmark

Friday, December 01, 2006

New visual rule builder (Michael Neale)

I have been beaving away on a new UI/rule modeller specifically for the web (well, at least the web initially, hopefully we will also do it stand alone in the plug in soon).

This came about for 2 reasons:
1) DSLs have proven to be quite popular, and we want to make them available on the web (essentually a very handy templating tool for capturing rule data), and
2) I thought it would be nice to have a helpful and user friendly UI to "coach" people in authoring optimal rules (developers can also look at the drl output, see what makes it tick !).
(and 3: I had some spare brain cycles during JBoss world to work on it! and I thought the above 2 aims could be combined).

Since a picture is worth a thousand words, here is a screenshot:



This should be available in the up coming rule repository user interface. Visually its still quite rough, but its getting there, and the model behind it is solid. I would appreciate any feedback on how to make this a more "intuitive" interface to use (and don't confuse it with decision tables or scorecards, thats another beastie, this is just for a single rule at a time management).

I am integrating this with the (previously seen) dsl-aware business rule editor, as it is a nice fit (some people will use DSLs exclusively to hide details, others may use the modeller exclusively, perhaps a mix - yet more may prever vanilla drl, or a mixture in a package).

One of the nice things about it is that it requires little/no configuration (whereas there is some skill to configuring a dsl - this editor is driven off the model: facts/fields define what operators are relevant etc) - of course a dsl based rule reads naturally, but it is also easy to understand what the rule is doing visually if the model matches the domain. (A future fun little task will be to generate read only statements of the rules - a la the rule builder for email clients for filtering out that pesky spam).

So hows it work: A thats a trade secret ! Oh wait, no its not, its already in the SVN repository probably being crawled all over by google as we speak ! (oh my embarrasing code !). It works as it knows about the facts (model) available to the rules, what operators are applicable, bound variables, and has a limited set of actions it can take (mostly asserting fact, retracting, modifying fields, globals etc - I may add more as needed, but DSLs can template more things as needed).

I am planning to make the operator names/conditional element names configurable (they will have prettier defaults then this, I just ran out of time to do that, the idea is that anyone understanding basic logic should be able to use this, and if they know first order logic, all the better !).

It also has some basic validation (for instance you can't remove a fact that is needed in an action) - lots more can be done with validation to ensure correct rules at the time of entry (rule analysis module is another, seperate topic and a hobby of mine !)

Enjoy !
Share/Bookmark

Wednesday, November 29, 2006

JBoss Rules 3.0.5 is released

Version 3.0.5 is now available, from downloads and the plug-in update site. This is the latest release of our current "production" branch of JBoss Rules.

Watch this space for news of a milestone release very soon of the next major version.

Thanks to all out there who submitted bugs/fixes/patches and enhancements.

See below for Release Notes - JBoss Rules - Version 3.0.5 (note you can always get these from our JIRA issue tracking system when you want):

Bug



  • [JBRULES-410] - More than one Query definition causes an incorrect Rete network to be built.

  • [JBRULES-440] - Unable remove a rule package when it was added after it has been added and removed

  • [JBRULES-455] - java.lang.NullPointerException at org.drools.util.LinkedList.remove(Unknown Source) in 3.0.4 while not in 3.0.3

  • [JBRULES-459] - Literal constraint using 'null' triggers a NPE.

  • [JBRULES-475] - Error in documentation concerning the joining of constarints

  • [JBRULES-481] - LeftInputAdater.updateNewNode does incorrect propagation

  • [JBRULES-499] - Rule not rendering activation group correctly

  • [JBRULES-505] - Using a GLOBAL value in predicates in more than one rule - rule fails to evaluate correctly.

  • [JBRULES-506] - Exception thrown when retract, reassert and retract again

  • [JBRULES-523] - JoinNode and NotNode with empty binder throw exception with getConstraints()

  • [JBRULES-532] - Query feature doesnt seem to work if there is more than one query.

  • [JBRULES-539] - Combining self-exists() with calling QueryResults twice returns to much

  • [JBRULES-542] - Drools core 3.0.4 - Drools does not remove itself as a listener when Facts are Retracted

  • [JBRULES-548] - NPT when binding a variable and comparing that variable to another property of the same object the variable was bound from

  • [JBRULES-550] - NPE while removing and adding same package

  • [JBRULES-557] - Dynamically generated classes not accessed by drools classloader

  • [JBRULES-562] - Security Permission problem in Websphere 6.1

  • [JBRULES-566] - Not-Constraint is not working properly



Feature Request



  • [JBRULES-442] - Building from source failed: Serialisation / externalisation error

  • [JBRULES-444] - patch to support Notes attribute on decision table Package

  • [JBRULES-483] - Open Decision Table API to allow custom RuleSheetListeners

  • [JBRULES-544] - getQueryResults behavior: return null


Share/Bookmark

Sunday, November 26, 2006

Business Rule System: To use or not to use? (Edson Tirelli)

One of the most frequently asked questions around is "Why would I want to use a Business Rules Engine?". Usually this question is immediately followed and is related to: "When should I use a Business Rules Engine and when should I not?".

Googling out there you will find several posts, papers and books about it. Examples:

From JBoss Rules Documentation
From Jess Documentation

Those questions are usually answered in terms of the benefits of using the core rules engine component (algorithm), and are all true. We can even state that they justify the need for themselves.

Although, one frequently under looked aspect is the benefit of rules governance you achieve when using a BRMS.

Business Rules require lifecycle management in the same way most other current Enterprise Technologies do.

If one looks at classical application development in an enterprise environment, no one can even imagine it happening nowadays without a control process, involving orchestration of activities, with authorization control for each of the steps, does not matter what is the actual used methodology. The whole process is usually supported by specialized tools, varying from Development Tools, Code Versioning Systems, Automated Integration Tests, and specially, Configuration Management tools.

In the same way, once Rules Engines allow us to separate Business Decision Logic from the application core code, it also allows us to implement a governance infrastructure for Business Rules. It means:


  • Allows specialized teams to work on each of the roles in a Business Rules lifecycle. For instance, business rules analysts are responsible for maintaining the business rules, while configuration management teams are the ones responsible for the deployment and application core developers worry only about using (interpreting) rules results.


  • Allows for fine grained control on the access and maintenance of the rules that control the Enterprise Business.


  • Allows for comprehensive auditing of implemented rules and all changes happening in the corporate environment.


  • Allows for rule sharing between different applications and corporate business processes. This as we know, allows for consistency between corporate processes and decision making, as well as reduced maintenance costs: a change in the single rule instance will reflect throughout the whole organization.



  • These advantages are in sync with SOA architecture goals, as well as governance requirements.

    So, when analyzing the advantages of using a BRMS, also think about Business Rules Governance and the benefits it brings to the organization.

    For more details on the requirements and features of such platform, you can refer to Mark's post about the JBoss Rules server.

    [EDIT: Adding related links posted by fellow readers in the comments, as here it is more visible to other readers than staying in the comments.]

    Thank you James for posting:

    "There are a number of reasons why business rules can be better than traditional code and a lot of benefits to be gained.
    There are also many excuses for not taking on business rules as an approach. I wrote more on these topics in my FAQ."

    Also, thank you Rajgo:

    "Especially, considering that rules are intrinsically more structured than code, it makes all the more sense to use a business rules Management System.
    In addition to governance & management, and decision automation, BRMS can provide businesss rules analysis that one could use to refine their policies to meet business goals"
    Share/Bookmark

    Saturday, November 18, 2006

    Drools 3.1 milestone 1 is comming

    As some of you already know, Drools 3.1m1 will be out of the door any minute now. The Drools name is used for project development releases, which are also unsupported - eventually the code is hardened and released to production as JBoss Rules 3.2. This new release is a a big improvement over the 3.0.x production branch in many ways and it will be intersting to see where we will go from there.

    For those curious about it, I will briefly talk about the major changes one expect to see when moving to it, from a core/compiler perspective.

    Performance

    After about 5-6 weeks of struggling led by Mark himself, most of the anchors slowing down JBRules 3.0 were simply left behind. By this I mean, no more internal structures for remembering matches, no more reference linking hell, no more overuse of array copies. Core was basically streamlined with a huge reduction on memory consumption and a significant performance boost. Benchmarks like manners, that stress the engine pattern matching algorithm are seeing improvements in the order of 300%, while memory comsuption is down to 20% of the original.
    Does that means that 3.0 performance was bad? I would say not at all! Performance was actually good on 3.0, specially when compared with previous drools versions and other open source engines. But no big improvement was possible in the 3.1 version without breaking some paradigms we had before. And Mark did an amazing job pushing every little aspect to the limit to find exactly what we needed to change.
    Does that mean real world applications will see improvements like that? Probably not in that order of magnitude, as benchmarks are designed to stress engines to its maximum, consequently generating extreme cases. Real world applications will certanly see great improvements, specially in the memory consumption side, but not in that order of magnitude. Actually I'm quite curious to test it in real use cases and see what will be the improvements.

    Mark has provided a quick summary of the main enhancements:
    • Unrolled loops for constraint evaluation
    • High performance custom collections implementation
    • Tuple is now the entry, avoid Entry object creation.
    • Tuple, a wrapper for FactHandle[] during propagations, now uses a linked list as per Jess from "Implementing A High-Performance Symbolic Reasoning Engine In Java"
    • Composite key indexing for left and right memory
    • Left and right value caching for join nodes, to avoid repeated value retrieval
    • Improved alpha node hashing
    • Extensive use of Single/Composite design to avoid setting up a for loop when there is only 1 child
    • Primitive support, avoids continuous primitive wrapping

    Primitive type support

    As users know, JBRules 3.0 automatically wraps any primitive types he needs to work with in its respective Object Wrappers. That was a limitation that was causing 2 undesired side effects:
    1. Performance really suffers when one needs to wrap/unwrap primitives.
    2. Code using primitive wrappers is much more verbose and cluttered.
    Example:
    rule "return value"
    when
    $person1 : Person( $age1 : age )
    person2 : Person( age == ( new Integer( $age1.intValue() + 2 ) ) )
    then
    ...
    end

    As you can see in the above example, the rule's writer needs to be aware and keep wrapping/unwrapping primitive values to work with the engine.

    JBRules 3.1m1 has now full support to primitive types, not only helping it to perform better and consume less memory but also making rules much more clear. Look at the same rule in 3.1m1:
    rule "return value"
    when
    $person1 : Person( $age1 : age )
    person2 : Person( age == ( $age1 + 2 ) )
    then
    ...
    end
    It is now clear and straight forward to understand what the rule writer wanted his rule to do.

    Shadow Facts support

    As it is well known, JBRules uses POJO as its first and preferred fact representation structure. We design the engine targeting it to perform most efficiently on beans, as we understand that this is the most simple and productive way of integrating a rules engine in real world applications: work with user's own business objects model. Although, this kind of integration demands a carefull planing on the engine side to avoid collateral effects, as the engine do not own the beans. The application or the user can change any bean property on the fly and not notify the engine.
    So it is mandatory that the engine implements some kind of control over the values being used for reasoning, keeping the whole engine state consistent. This is implemented by Shadow Facts, that are shallow copies (cache) of the bean values while reasoning over it, allowing updates only on safe points.
    Shadow Facts are implemented in JBRules 3.1m1 as dynamic proxies and besides being mandatory to have from an engine perspective, they are almost invisible for the engine users. I say almost, because being dynamic proxies, there is only one requirement users need to meet: a nonarg default constructor for the classes asserted into the engine. Besides that, users don't have to worry about it. Only to know, their facts are being taken care appropriatelly. The proxy is not exposed to users, not even in consequence block.

    New nodes

    Evolving the engine also requires support to some new conditional expression elements. Three new CEs are included in 3.1m1, being them:

    FROM: allows reasoning over facts not previously asserted into the engine. Specially useful for retrieval of facts on the fly from database tables or any other external source. Example:
    rule "From"
    when
    $cheese : Cheese(type == "stilton" ) from cheesery.getCheeses()
    then
    // do stuff
    end
    COLLECT: allows reasoning over collections of objects from working memory. Example:
    rule "Collect"
    when
    $cheeseList : ArrayList(size > 2) from collect( Cheese( price < 30 ) ) ;
    then
    // do stuff
    end

    ACCUMULATE: a more flexible and customizable version of Collect, allows to iterate and calculate values from a set of facts in the working memory. Example:
    rule "Accumulate"
    when
    $person : Person( $likes : likes )
    $cheesery : Cheesery( totalAmount > 100 )
    from accumulate( $cheese : Cheese( type == $likes ),
    init( Cheesery cheesery = new Cheesery(); ),
    action( cheesery.addCheese( $cheese ); ),
    result( cheesery ) );
    then
    // do stuff
    end
    Conclusion

    These are the main features you will se in this milestone from a core/compiler perspective. New milestones will bring even more news, specially related to the Rule Server under development as described in a previous post.

    So, stay tunned, enjoy, and please provide us your feedback.

    Edson
    Share/Bookmark

    Friday, November 17, 2006

    Declarative Behavioural Modelling (DBM) and jBPM/JBoss Rules integration

    Production Rule systems, such as JBoss Rules are just one form of declarative programming; I believe that to meet the challenges of tomorrows technology we need to be looking at a unified tooling and platform approach to declarative programming. Workflow/BPM has been the predominant declarative approach for a while and Production Rules are one of the first technologies from the AI family to go mainstream. One of the nice things about Production Rule systems is they can be expanded to handle a number of domains declaratively and a variety declarative tooling can be built on top - as shown in the diagram below. I hope to use JBoss Rules as a vehicle to bring more of these technologies into the mainstream - Event Stream/Management Processing, Backward chaining and Constraint Programming are all essential items to a modern enterprise and each allows you to model the different behaviours of your system - I see these as shorter term goals. Genetic Algorithms, GIS, Neural Networks and the Semantic Web are all interesting long term goals.



    For now we need to focus on the two mainstream technologies rules and workflow and figure out how provide a much needed unified modelling environment.

    Deployment
    Declarative programming often involves the generation of content which can expose a whole heap of problems. The jBPM way is to own the control of this using their deployment tool to couple the process definition and the java classes. You write your process definition xml file and select its dependant class files which it then deploys as a PAR; which is a JAR containing the xml file and the compiled dependant classes. This gives you control over the java classes, making sure that a particular process definition always works with the correct version of the java classes. We can extend this to allow the process definition xml to reference one ore more drl files which it then makes available as a rulebase using variable reference in the jBPM context. There are two ways we can approach this. Either jBPM compiles the rulebase to a binary and serialises that to the PAR and reducing the runtime dependencies or it can add the drl to the PAR needing a runtime compilation of the drls into Rule Bases. Users can then easily look-up and reference the working memory from their own concrete handler implementations.

    While this is an effective and simple way to solve the deployment problem it has a number of downsides. The PAR has ownership of those java classes and their versions, this may clash with existing deployed classes for other applications - so care needs to be taken here. While processes rarely change and many consider them to be code and thus more suited to the traditional development and deployment, rules on the other hand are both code and data and can have continual updates - this can make traditional deployment systems quite cumbersome. Further to this large organisations can have thousands of processes and tens of thousands of rules - suddenly maintaining this in a standard source control management system doesn't sound so appetising.

    The next release of JBoss Rules will include a BRMS which will allow a more enterprise approach to management of rules, please see my previous blog "JBoss Rules Server" for more details on this. The basic idea is that all rules are stored in the BRMS where they are versioned with full meta data and categorised, I can configure specific versions of those rules into a RuleBase that I can then make available - the rule base itself is also versioned. With this approach the jBPM process definition can specify a URI to the BRMS for the rule base it needs, which is lazy retrieved on first use. While this does provide better management for a large number of rules the key benefit this provides is that the application and the BRMS are now aware of each other and the BRMS can feed the application regular updates to the rule bases its using; allowing rules to live up to their role of being data.

    If this approach proves successful we can look at managing other knowledge assets within the BRMS, such as the classes and the process definitions themselves. We can build versioned bundles which depend upon other versioned bundles which means that during deployment we can have a more fine grained way to solve the initial problem of the PAR owning the classes - with this approach it would be up to the server to manage versions and bundle dependencies. OSGi tries to solve some of these issues, and its something I want to explore as a long term solution to this.

    Tooling
    The previous section covered deployment; which is actually very boring and not of interest to most users. What they are really interested in is the tooling. Currently jBPM nodes are handler based; you create a concrete class that implements an interface and retrieves the variables it needs from the jBPM context, the jBPM node then specifies that class as its handler. While simple this concept is very powerful in its flexibility and is one of jBPM's strong points and why it is favourite with BPM developers. However there are times when you want something a bit more integrated and declarative, especially when working hand in hand with business users. We can't integrate DRLs any more than the previous deployment example provides; however as the above diagram shows Production Rule systems facilitate a number of very good ways to declaratively author rules, the most common of which are decision trees, decision tables, score cards, structured natural language, gui driven rule authoring and rule flow. What users really want is an ability to specify one of these authoring tools and apply them to a jBPM node so that they can declaratively control the decision or task node - this offers real value. Further more the "glue" to hold these together, i.e. the passing of the relevant variables from the jBPM context and the calling and execution of the rules must be handled seamlessly in the background. I expect that the actually author page should be embedded in one of the existing jBPM process tabs in eclipse.

    Stateful
    The above sections cover the stateless use of JBoss Rules with jBPM, this means that the life time of a working memory is for the execution of the current node, there are no long lived Working Memories. The reason for this is because jBPM supports the persistence of long lived processes, it is not generally expected that a Working Memory will be suspended and persisted else where. For this reason we currently consider it an exercise for the end user to manage, although we will try and provide a number of useful enhancements to enable JBoss Rules to play well in this environment.

    DBM
    So far I have only covered the unification of a Production Rule system and Workflow/BPM system; however this still only allows you to model a specific and limited set of problems. For a bit of background many of you may have heard of the AI Winter where AI was hyped and failed to deliver and has been relegated to mostly academic areas. Production Rules is one of the AI technologies now going mainstream, even though it’s been around for 25 years - what’s enabling this? I think it's mostly due to good tooling and better documentation, mere mortals are simply not able to work in shell environments and be productive, let alone decipher some cryptic paper on how to use and implement a specific technology. Many of the other technologies have worked in silos again with limited or no tooling. Both probability/uncertainty and genetic algorithms have had a number of papers showing how they can work well with a Production Rule system, yet this hasn't had any real uptake. It is my hope that the introduction of those systems into JBoss Rules with good tooling and documentation can help take these technologies such as these mainstream and improve the way software is written. With the way that hardware is scaling with the move to multi core processors and grid computing existing procedural ways of programming will not be enough and we will need paradigms such as these and distributed agent computing to really make the most of these environments.

    Post Comment
    Share/Bookmark

    Wednesday, November 15, 2006

    Washington Rules !

    -- Michael Neale
    Both Mark and myself recently returned from the highly informative Business Rules Forum (in Washington DC).

    The business rules industry seems to be in quite good shape, and there was a strong showing from all the vendors. I thought I would start with some summaries in general of what we saw, and what I thought.

    The engine performance panel
    James Owen - an industry commentator and rule practitioner and evangelist of many years experience, kindly hosted a panel on performance challenges for rule engines (for the geeks). It was a real whos-who of rule engine core tech.

    Take a look at the panel of braniacs below (James Owen is missing, he was talking).

    From left to right (Dr. Ernest Friedman (Jess), Dr. Charles Forgey (Fair Isaac, inventor of RETE), Rev. Daniel Selman (ILog JRules product manager), Rabbi Mark Proctor (some bloke, JBoss Rules), Pedram Abrari (CTO Corticon)



    Innovations I found interesting

    There were strong offerings from all the vendors on rule management/editing/higher level tools, and an understanding that a lot of the core tech is being commoditised. Nevertheless, there were many different approaches. Here is an outline of the products/companies that tickled my fancy:
    • Rule Burst
      • I love these guys as they are Australian ;) (same as me for those who don't know). Their approach is rules capture and management inside Word documents, with a strong natural language engine. They used to be called "Softlaw" and specialise in turning complex legistlation into executable rules by marking up the actual legal documentation !
      • I also like them as they are solving hard problems for important customers which really shows the power of inference engines and rule management systems. They have been around for quite some time (> 15 years) and got their start with Prolog ! They tooling is also impressive, and they have a fairly unique way of allowing "backwards" chaining rules to ask questions and automatically generate questionnaires based on the rules.
    • Corticon
      • These guys are really approaching the problem from a different angle. They call it "rule modelling" and their Workbench/IDE/Environment is very impressive, and allows people to quickly model facts (vocabulary I think they call it), rules, and test/verify/analyse/find logic faults and gaps all in in the one desk top tool, and deploy it (for example) as a web service. Their CEO is also a MD (Medical Doctor) and he got his start as he saw the potential of rule based systems in medical diagnosis.
    • Pega Systems
      • Pega weren't so much focusing on rules but also on process management, but even more so as whole new way of building apps "in place" - as in you take a web applications user interface, and switch in to design mode and you can modify the process/rules/fields that are "behind" the screen. Impressive, they have been really going hard with R&D, and it shows. Rules are just one part of their vision.
    • ILog
      • They didn't participate in the "product derby" (kind of an Uber demo that gets shown while we are eating lunch) this year, but I did attend a session or two with Daniel Selman, the JRules Product Manager. I was particularly impressed with his presentation on "Mission Critical applications" and how it applies to rule engines. ILog JRules is clearly used everywhere in really sensitive applications, and Daniel did a great job outlining the real world challenges of maintaining complex applications and evolving them AND THE OBJECT MODEL over time (the JRules "Business Object Model" approach really shines for this, and now supports re factoring etc). ILog have impressive credentials in this space.
    • Haley
      • One of the benefits of this conference is that we get to speak with real world rule practitioners, and I heard great feedback about Haley's implementation of NLP and NLU (Natural Language Processing and Natural Language Understanding) - which is an area that greatly interest me. On the one hand I view NLP as a little like voice recognition (annoying) or handwriting recognition (we all remember *that* version of the Apple newton) - ie hard to get right (and near enough is NOT good enough). But Haley seemed to have made this work for rules - so well done !
    So it was an impressive showing from the Vendors. I am sure I missed some out, but this is all I had time to write notes on.

    SOA and rules
    The "rule service" approach (stateless rule services exposed nominally via WSDL) seems to have become popular, and certainly seems applicable for a large class of rule problems (business decisioning, single point of truth etc.).

    Future challenges
    No one seemed to agree on a single way forward for modelling facts, some were talking MDA (MDA was a strong theme), some were purely WSDL/Web Service, and utilising OWL was even mentioned a few times (personally over the long term, I think OWL ontology's may be The Way, but I am only new to them so I shouldn't show my ignorance just yet). Modelling facts is a real challenge, and very few people were also thinking about how to cope with model evolution over time (as no one will get it right first go).

    Rule standards are another major challenge, I am not expecting anything concrete for some months, perhaps a year. It was great to meet Said Tabet (of RuleML fame) and hear his opinions, and its also great to see vendors like ILog take RIF from W3C very seriously.

    Truth, Justice and the American way
    I found the US to be surprisingly familiar in every way, more so then Britain. The "tipping" culture took a while to get used to, I felt embarrassed as I had to ask people when to tip or not tip, how much etc. A feature of this seams to be to deliberately cause conflict. On the flip side, the service for everything was excellent. It was a hoot hanging around with James Owen, the ultimate Texan ! From my sampling of several Texan acquaintances over the years, I can say that I like Texans !

    Pretty Washington
    The hotel was fantastic, and I was impressed by the array of Autumn colours (Autumn seems to be taking its time in the UK).

    Next
    Next week we will be in Berlin for JBoss World (I am arriving a little early to do the tourist thing, as a native to the Land Down Under, there is plenty over this side of the globe I have not yet seen - its all good !). If you are there, make sure you find Mark Proctor or myself, we have t shirts to give away !

    Post Comment
    Share/Bookmark

    Thursday, November 09, 2006

    JFDI a new Business Action Scripting Language

    We are working on a new non-imperative conseqeuence language for JBoss Ruless, called JFDI which is a sort of business action scripting language. We have placed this project over at Codehaus as we are hoping that it will get taken up by other declarative systems, whether they are rule or process engines.

    When we first thought about doing this we got a deluge of emails like "have you looked at groovy, rhino, judoscript etc" - the answer was obviously "yes". The last thing wanted to do was yet another language and we looked hard into using those languages, but they all fell short. Not short in features, but short becuase they did too much!!! Here was the main feature set:
    1) A rule consequence should be "when this then that", not "when this maybe that", its quite simply a list of actions to be performed when this situation occurs. Therefor it should not contain any imperative code, no "if", "switch" statements, its purely focused on expressions, method/function calls and we will allow a foreach loop. If someone needs to do some imperative code it should be encapsulated within a method or a function which is then called from the consequence.

    2) The language should not be ambiguous and thus should be typed, although we may eventually allow type inference. Support for refactoring is definitely a long term goal.

    3) A rule base may have thousands of rules, currently with JBoss Rules you end up with a minimum of two classes per rule and then a further class for each function, predicate, return value or 'eval' used. This large number of classes can cause a number of problems from perm gen issues to slowing down the system's classloader. Therefore it must be able to run interpreted, no bytecode generation. A consequence can be serialised to disk and lazy loaded on first use. While this is interpreted, as much as possible is done at "compile" time with no runtime introspection, so it's a tree of cached java.lang.reflect.Method calls. Although eventually we would like optional bytecode generation optimisations, but generally it is not the execution of the consequence that is the bottleneck.

    4) A simple groovy/ruby like expression syntax. We need non-verbose ways to reference sub fields and to declare and reference maps and arrays, with support for inline anonymous maps and arrays. We would like further support for variable interpolation and <<EOF type stream writing.

    5) The traditional way to "inject" variables into a scripting language is to put each key/value pair into a dictionary or context. We use to do this before drools-2.0-beta-14, it is not performant, instead we need to allow the application to determine at compile time how to resolve those external variables.

    6) We need to be able to decorate a variable, at compile time, with additional methods - this way in a consequence a user can do "myVar.getFactHandle()" even if the variable itself does not support that method.

    7) Native support for more complex types like BigInteger, BigDecimal and currency types is needed - so we need more control over that.

    8) In-built support for FactTemplates (a better DynaBean) so they can be used as if they were real classes.

    9) A less verbose way to set setters "instance(field1=z, field2=42)". Ability to call a constructor and set fields in a single statement "new BarBaz(field1 = "val", field2 = "x")".

    10) The dependancy must be small, a few hundred kb.

    Bob McWhirter has been busy beavering away on this and the project is getting to an almost useable state; although there is no content on the website you can start to look over the unit tests to get an idea of how the language is turning out.
    http://svn.codehaus.org/jfdi/trunk/test/org/codehaus/jfdi/interpreter/
    http://svn.codehaus.org/jfdi/trunk/test/org/codehaus/jfdi/

    A quick look at the language itself:

    //fields
    instance.field = value;
    instance(field1=z, field2=42)
    instance.map["key"] = value;
    instance.array[0] = value;

    // method call with an inline map and array
    instance.method( [1, 2,"z", var], {"a" => 2, "b" <= c} );

    // standard constructor
    bar = new BarBaz("x", 42)

    // calls default constructor, THEN setters
    bar = new BarBaz(field1 = "val", field2 = "x")


    We are still trying to decide on the for each loop syntax. Bob wants to offer both the crappy java5 syntax (otherwise people will complain) and something more readable:

    foreach item in collection {
    func(item)
    func2(index) # index available automatically as a counter?
    }

    For ( item : collection ) {
    func(item)
    func2(index) # index available automatically as a counter?
    }


    We don't just plan to use this language for consequences, it will also be used from predicates, return values and 'eval's as well as the upcoming 'accumulate' and 'from' conditional elements.

    $cheese : Cheese( /* additional field constraints go here */ )
    from session.getNamedQuery("thename").setProperties( {"key1" => "value1", "key2" => "value2" }).list()


    So if you like what you see why not roll up your sleaves and get involved. A free T-Shirt to the first person that hassles bob to apply some patches :)

    Post Comment
    Share/Bookmark

    Just say no to DynaBeans

    The subject of DynaBeans just came up on the mailing list, and it is one I've been asked about before - so I thought I would do a blog my answer.

    Dynabeans was written as a solution for Struts back in 2000, they are not JavaBean compliant – thus they have no value outside of that provided with commons BeanUtils, thus coupling your enterprise system to BeanUtils forever – no other apps are aware of them, or able to script them like they can JavaBeans. They cannot be used with any database schema mapping tools, so you cannot easily leverage db schema generation tools. JBRules 3.0 has no support for mapping utilities thus you’ll only be able to script those from within evals. If you only use evals you will have crippled the engine to work like a simple chain of command scripting engine – under no circumstances do I recommend this. Evals are a worst use case for conditions that cannot be expressed in field constraints, as field constraints and conditional elements become more powerful the dependency of evals is reduced. Further more DynaBeans are backed by a map, each read or write results in a hashmap read or write – this is not a scalable solution.

    A much better solution that works now for JBRules 3.0 is to have runtime class generation. See this blog entry here: http://sixlegs.com/blog/java/death-to-dynabeans.html

    Tools like cglib make this trivial now:

    BeanGenerator bg = new BeanGenerator();
    bg.addProperty("foo", Double.TYPE);
    bg.addProperty("bar", String.class);
    Object bean = bg.create();

    That produced bean can have hibernate mappings generated at runtime, so you’ll get runtime hibernate support. JBRules can compile drls at runtime using the classloader those beans where generated with thus providing full field constraint use of them and pojo like access in the consequence - much nicer :)

    For JBRules 3.2 we will provide a feature called FactTemplates – this is akin to jess/clips DefTemplates. These allow people to define Facts without the need for a compiled class, which is preferable to some who want complete encapsulation of their rules and business objects within the drl. FactTemplates are backed by an array and thus read/writes are much faster. FactTemplates support both named and int values to specify the field for the value; named set/get result in a HashMap lookup so have similar performance to DynaBean. However JBRules provides compile time optimisation of those named fields and swaps them to int lookups – we will have support for this in the JFDI language too, which we also hope will be support in jBPM. FactTemplates will also provide mapping facilities which means they can be mapped to any underlying structure – be it JavaBean, a Hashmap or a DynaBean.

    Post Comment
    Share/Bookmark

    Sunday, November 05, 2006

    Rete with Lazy Joins

    I've just spent the last four weeks stuck in front jProfiler taking JBoss Rules performance to the next level and the results have been great and well beyond what I hoped. To achieve this I wrote custom collections, unrolled loops and cached variables that are used repeatedly for join attempts. I really feel I've taken traditional Rete implementations with the following well known performance enhancements node sharing, alpha node hashing and beta node hashing to the limit. Yet I'm still way off OPSJ levels of performance for manners128 and waltz50.

    My initial thought was that OPSJ must be doing some kinda of compile time/static agenda, like JRules attempts. This idea was slated for investigation when I next tackle performance again. The idea with compile time/static agenda's is that you arrange the nodes and their memories so that propagations and joins occur in such a way that they mimick the results of a simple lifo style agenda and thus they can fire as soon as they hit the terminal node. You gain speed as you are no longer determining all cross products and conflict sets, this is kind of a cross between Rete and Leaps.

    I have recently had the pleasure of exchanging a few emails with Charles Forgy, Earnest Friedman-Hill and a few others where ofcourse I just had to take the opportunity to quiz Charles Forgy on this. Here was his reply "OPSJ does not do compile-time static analysis to avoid computing conflict sets. It computes complete conflict sets, and it applies full MEA analysis to the conflict set (not some less-expensive composite pseudo-MEA)". I was gob smacked and straight away went to off to verify this:

    Waltz on OPSJ (results given to me):
    Added To Agenda: 29,910
    Fired : 14,067

    Waltz on JBoss Rules:
    Added To Agenda: 31,841
    Fired : 14064

    The differences are most likely due to my "less-expensive composite pseudo-MEA" conflict resolution strategy. These results proved Charles' statement true and after having spent four weeks taking true Rete to what I thought was the limit it left me feeling like the twelve year old kid who thought he could play football with the adults.

    Anyway not to be detered I went back to racking my brains again. One of the great things about systems like OPSJ and Jess is they give you something to aim for, without those markers I would probably have given up on trying to improve Rete long ago and opted for a compile time/static agenda like JRules has done. So here goes my next attempt at double guessing OPSJ :)

    In my latest drools code my hashing system does two things, firstly it allows for indexing of composite fields and secondly it returns a bucket where it guarantees that all iterated values are already true for the indexed fields, you do not need to test them again - only further non "==" fields for that join node. Inside a join node I return this bucket and join and propagate for each iterated value. However imagine if instead of iterating over that bucket you simple give a reference from the incoming token to that bucket and propagate forward – so by the time it reaches the Agenda there have been no joins but it has references to all the buckets that contain potential successful joins – I use the term "potential" as the bucket may contain un-indexed fields due to field constraints with non "==" operators. The joining process is then delayed right until the activation is finally fired. Obviously there are still a huge number of details to work out and I’m still not totally sure if this would work, especially if a join node refers to variable constraints that are in a buckets that contains non "==" operators, as you cannot be sure that you should be testing and joining those.

    For Manners i'm pretty sure this works as the "find_seating" rule as two ‘not’ nodes with only "==" constraints. For waltz it’s more difficult as all joins also have "!=" in them. Still I have enough of an idea now that I can start messing with code to try and do a proof of concept.

    Post Comment
    Share/Bookmark

    Sunday, October 22, 2006

    Beyond ORM

    A rule has many similarities to a query. It contains one or more propositional and first order logic statements organised into a network to filter data additions and changes - we call this a "discrimitation network" as it discrimates against data that does not match its statements. Any data that successfully matches all statements for a rule arrives at a Terminal Node; at this point we can either execute a series of actions (ignoring the role of the Agenda, for the moment) on that "row" of data or return it as part of a query request.

    Like a database we index our data to provide high performance. There are two types of indexing - Alpha Node hashing and Beta Node indexing. Alpha Node hashing is used to index literal constriants, so that we only propagate the data to the rules that match that literal value. Beta Node indexing is used to index data used in joins - we have two facts, person and cheese, we record who owns which cheese and there is a rule that says "when the person's cheeses are out of date then email that person" we have a join between the person and the cheeses, so we index each cheese against its owner. We also attempt to share indexes, so if rules use the same literals or joins where possible we try and share those indexes - this reduces memory consumption and also avoids duplicate constraint evaluations.

    However a Production Rule system provides many features beyond a traditional database. Many of the features, marked with a * are planed for JBoss Rules releases next year.

  • Efficient First Order Logic with 'exists', 'not' and 'forall'* quantifiers as well as cardinality qualifiers with 'accumulate'* and 'collect'*.

  • Abilitity to mix reasoning of data inside and both outside of the Working Memory using 'from'*.

  • Object Validation* so only objects that are valid can exist. "name length must be less than 30".

  • Backward Chaining* for complex inferencing.

  • Ontology* support for rich Object Models, probably via some Semantic Web OWL support.

  • Efficient Truth Maintenance. Truth relationships can be set in place to ensure the Working Memory never breaks this statement.A "Red Alert" object can only exist while there are 3 or more emergencies.

  • Event Stream Processing(ESP)* can analyse sets of data over time windows. "Determine the average stock ticker price in the last 30 seconds".

  • Event Correlation Processing(CEP)* can analyse data sets with temporal comparisons between object.


  • Firstly, before ORM advocates try and organise a public lynching, let me just state that "Beyond ORM" does not mean to replace ORM, there are clearly many applications where ORM is preferable - particularly when dealing with truly massive datasets or where you want to be able to represent relational data in differing form. However there are situations which can benefit from a richer and better integrated solution with features as described above. There is still a lot of work to achieve all of the above and we then need to consider how to cluster working memories to provide fault tolerance and some way to also make working memories transactional. If we can solve those problems we can look to integrate JBoss Cache, JBoss Rules and JBoss ESB for a next generation Data Centre. This is is a long term R&D proposal, but I thought I would sketch down the basic ideas in the diagram below.



    Post Comment
    Share/Bookmark

    Wednesday, October 11, 2006

    JBoss Rules Server

    Many people are asking what our plans are for a Rules Server - so I thought I would write them up in visio and blog them, for bonus measure I have added some screenshots of the GWT gui that Michael is working on.













    Post Comment
    Share/Bookmark

    Monday, September 04, 2006

    New header icon


    Share/Bookmark

    Monday, July 10, 2006

    Why Java code is bad for rules and a declarative alternative

    One of the selling features of Drools, and one of the reasons we are often chosen over competitors, has always been the ability to allow the use of Java code in specific parts of rules; expressions and consequences. This makes for a lower learning curve as Java developers can start writing consequences without additional training; whether its updating a value, sending messages or retrieving information from a database. This alignment with Java, or "Java like" languages, for Rule Engines is often touted by marketing as a reason to choose their systems over others. On the face of it this looks great and its something that management can relate to - less training, leveraging existing skill sets, where's the down side?

    The use of a Java language detracts from the real value of a Production Rule System like Drools. The moment you allow the use of Java code in a rule you encourage the use of imperative programming. Rule Engines offer a Turing complete system with a declarative language, if you read the Drools manual we spend some time explaining propositional and first order logic and how they can be used to describe any scenario in a declarative manner - this is an incredibly power approach to software development. In quick summary Java is imperative because you have describe how to do things, this can often take up many lines of code and means you have to read all those lines of code to understand what it is doing. As the code gets more complex the more lines of code you have to read, eventually to the point that the complexity becomes large enough to make the code difficult to understand and maintain; thus creating a dependency on the original author who becomes the only person able to efficiently maintain the code. Declarative programming allows you to express complex rules using keywords; each rule identifies a specific scenario and details the corresponding actions. Those actions should also be specified in a declarative manner, otherwise we reduce the value of investment that we made in defining and authoring the conditions of the rule. This is very much in spirit with the Business Rules Approach methodology. For those more interested in this methodology I recommend the following three books:
    Business Rules and Information Systems: Aligning IT with Business Goals
    Tony Morgan
    ISBN: 0201743914

    Principles of the Business Rule Approach
    Ronald G. Ross
    ISBN: 0201788934

    Business Rules Applied
    Von Halle
    ISBN: 0471412937

    So the moment we start putting in complex nested if structures and loops into our consequence we increase the complexity of maintenance. Instead the consequence should focus on the calling of functions or object methods. Each function or method has a clear and documented role; it specifies the parameters it takes, the operations it makes on those parameters and what it returns - they also align themselves better to be represented with a simple domain specific language.

    While Drools has no plans to drop its Java support and we soon hope to be adding back in Groovy and later Javascript - we also want to introduce a language that will support the declarative approach, rather than detract from it. This new language will then become the "default" standard that we push for rule authoring.

    So what should such a language include?

    • full expression support

    • Auto-box and auto-unbox

    • Object method calls

    • Object field assignments

    • Object creation

    • simple declerative data structures

    • basic if/switch/loop support (to be used sparingly)


    I'm still trying to decide if we need support for anonymous classes - there may be times when we need to specify callbacks or action listeners on objects. However this obviously adds a level of complexity that I wish to avoid inside of consequences and may well be something that should be farmed out to function or method. Control structures like if/switch/loop might also be made configurably "optional" and discouraged. The language should be able to work in both interpreted and compiled mode. Compiled mode allows for maximum speed execution and is ideal for small to medium systems - large systems in the thousands of rules will suffer permgen issues and it may be best to use interpreted mode.

    I have done some searching but most languages are "complete" in that they are full blown languages for application development - we need something much smaller and simpler. This is more inline with what templating languages, like FreeMarker and Velocity, already have - but is not available as a stand alone language. I have recently discovered "Simple Declarative Language" which seems to go in the right direction; but has no support for function or method calls or expression evaluation - that I can see - however with the absence of anything else on the market it may be a good place to start in building our own solution.

    Post Comment
    Share/Bookmark

    Tuesday, June 20, 2006

    Real World Rule Engines

    Here is an excellent article, introduction reproduced below, from our very own mailing list mentor Geoffrey Wiseman:
    http://www.infoq.com/articles/Rule-Engines


    For many developers, rule engines are buzzwords, or black boxes on an architectural diagram: something to be feared or admired from afar, but not understood. Coming to terms with this, is one of the catch-22s of technology:

    • It's difficult to know when to use a technology or how to apply it well until you've had some first-hand, real-world experience.

    • The most common way to gain that experience is to use an unknown technology in a real project.

    • Getting first-hand experience using a new technology in a production environment is an invaluable experience for future work but can be a major risk for the work at hand.



    Over the course of this article, I'll be sharing my practical experience with rule engines and with Drools in particular to support in-market solutions for financial services, in order to help you understand where rule engines are useful and how to apply them best to the problems you face.

    Why Should I Care?


    Some of you will have already considered using a rule engine and will be looking for practical advice on how to use it well: patterns and anti-patterns, best practices and rat-holes.

    Others haven't considered using a rule engine, and aren't sure how this is applicable to the work you're doing, or have considered rule engines and discarded the idea. Rule engines can be a powerful way to externalize business logic, empower business users, and solve complicated problems wherein large numbers of fine-grained business rules and facts interact.

    If you've ever taken a series of conditional statements, tried to evaluate the combinations, and found yourself writing deep nested logic to solve a problem, these are just the sorts of entanglements that a rule engine can help you unravel.

    Some of our more complicated financial services work, when rephrased in a rule approach, began to look markedly more comprehensible. Each step in converting procedural conditional logic to Drools business rules seemed to expose both more simplicity and more power at once.

    Finally, if you're not convinced by the above, consider this: rule engines are a tool, another way to approach software development. Tools have their strengths and weaknesses, and even if you aren't making immediate use of this one, it's helpful to understand the tradeoffs so that you can assess and communicate applicability in the future.

    Post Comment
    Share/Bookmark

    Friday, June 02, 2006

    Rule Execution Flow with a Production Rule System

    Some times workflow is nothing but a decision tree, a series of questions with yes/no answers to determine a final answer. This can be modelled far better with a Production Rule System, and is already on the Drools road map.

    For the other situations we can use a specialised implementation of Agenda Groups to model "stages" in rule engine execution. Agenda Groups are currently stacked, like Jess and Clips modules. But imagine instead if you could model linear Agenda Group execution – this is something I have been thinking about for a while to allow powerful and flexible modelling of processes in a Production Rule System. A successful implementation has clear advantages over two separate engines – as there is an impedance mismatch between the two. While there is little issue using a rule engine with workflow, using workflow to control linear execution of a rule engine will very suboptimal – this means we must seek a single optimal solution for performance sensitive applications.

    Let’s start by calling these special Agenda Groups "nodes", to indicate they are part of a linear graph execution process.

    Start rules don't need to be in a node and resulting target nodes will detach and evaluate once this rule has finished:

    rule "start rule"
    target-node "<transition>" "<name>"
    when
    eval(true)
    then
    // assert some data
    end


    The start rule and the nodes can specify multiple target nodes and additional constraints for those target nodes; which is explained later. The start rule can fire on initialisation, using eval(true), or it could have some other constraints that fire the start rule at any time during the working memory life time. A Rule Base can have any number of start rules, allowing multiple workflows to be defined and executed.

    The start rule dictates the next valid target-nodes - only activated rules in these nodes can fire as a result of the current assertions. While the activated rules in other nodes will not be able to fire, standard rules and Agenda Groups will react, activate and fire as normal to changes in data.

    A node rule looks like a normal rule, except it declares the node it’s in. As mentioned previously a node can contain multiple rules; but only the rules with full matches to the LHS will be legible for firing:

    rule "rule name"
    node "<name>"
    when
    <LHS>
    then
    // assert some data
    end


    There is an additional node structure, which the rules are associated with, and specifies the resulting targets:

    node "node name"
    target-node "<transition>" "<name>"
    end


    Target nodes are only allowed to evaluate their activated rules once the previous start rule has finished or the previous node is empty because it has fired all its rules. Once a node is ready to be evaluated, we "detach" it and then spin it off into its own thread for rule firing, all resulting working memory actions will be "queued" and assert at safe points, so Rete is still a single process. Once a node is detached the contained rules can no longer be cancelled, they must all fire – further to this no further rules can be added. All our data structures are serialisable so suspension/persistence is simply a matter of calling a command to persist the detached node off to somewhere.

    As well as a rule specifying the LHS constraints for it to activate, the previous node can specify additional constraints. A rule can be in multiple nodes, so if two incoming nodes specify additional constraints they are exclusive to each other - in that the additional constraints of the non current incoming node will have no effect:

    node "node name"
    target-node "<transition>" "<name>" when
    <additional constraints>
    end
    end


    Further to this a node can specify multiple targets each with its own optinonal additional constraints. Sample formats are showing below:

    node "node name"
    target-node "<transition>" "<name>"

    target-node "<transition>" "<name>" when
    end

    target-nodes "<transition>" "<name>"
    "<transition>" "<name>"
    "<transition>" "<name>"

    target-nodes "<transition>" "<name>"
    "<transition>" "<name>"
    "<transition>" "<name>" when
    end
    end


    Further to this we need additional controls to implement "join nodes" and to also allow reasoning to work with both the transition name as well as the node name.

    This highlights the basics for linearly controlled execution of rules within a Production Rule system. It also means we can model any BPM process, as it’s now a simplified subset, but allow it to be done in a highly scalable way that integrates into very demanding tasks. Further to this we can still have standard agenda groups and rules that fire as a result of data changes. This provides for a very powerful solution that is far more powerful than the simple subset that most workflow solutions provide.

    Post Comment
    Share/Bookmark

    Thursday, June 01, 2006

    What is a Rule Engine

    Drools is a Rule Engine but it is more correctly classified as a Production Rule System. The term "Production Rule" originates from formal grammer - where it is described as "an abstract structure that describes a formal language precisely, i.e., a set of rules that mathematically delineates a (usually infinite) set of finite-length strings over a (usually finite) alphabet". Production Rules is a Rule Based approach to implementing an Expert System and is considered "applied artificial intilligence".

    The term Rule Engine is quite ambiguous in that it can be any system that uses rules, in any form, that can be applied to data to produce outcomes; which includes simple systems like form validation and dynamic expression engines: "How to Build a Business Rules Engine (2004)" by Malcolm Chisholm exemplifies this ambiguity. The book is actually about how to build and alter a database schema to hold validation rules which it then shows how to generate VB code from those validation rules to validate data entry - while a very valid and useful topic for some, it caused quite a suprise to this author, unaware at the time in the subtleties of Rules Engines differences, who was hoping to find some hidden secrets to help improve the Drools engine. jBPM uses expressions and delegates in its Decision nodes; which controls the transitions in a Workflow. At each node it evaluates a rule that dicates the transition to undertake - this is also a Rule Engine. While a Production Rule System is a kind of Rule Engine and also Expert System, the validation and expression evaluation Rule Engines mention previously are not Expert Systems.

    A Production Rule System is turing complete with a focus on knowledge representation to expression propositional and first order logic in a concise, non ambigious and declarative manner. The brain of a Production Rules System is an Inference Engine that is able to scale to a large number of rules and facts; the engine is able to schedule many rules that are elegible for execution at the same time through the use of a "conflict resolution" strategy. There are two methods of execution for Rule-Based Systems - Forward Chaining and Backward Chaining; systems that implement both are called Hybrid Production Rule Systems. Understanding these two modes of operation are key to understanding why a Production Rule System is different.

    Forward Chaining is 'data-driven' and thus reactionary - facts are asserted into the working memory which results in rules firing - we start with a fact, it propagates and we end with multiple elegible Rules which are scheduled for execution. Drools is a forward chaining engine. Backward Chaining is 'goal-driven', we start with a conclusion which the engine tries to satisfy. If it can't it searches for conclusions, 'sub goals', that help satisfy an unknown part fo the current goal - it continues this process until either the initial conclusion is proven or there are no more sub goals. Prolog is an example of a Backward Chaining engine; Drools will adding support for Backward Chaining in its next major release.

    The Rete algorithm by Charles Forgy is a popular approach to Forward Chaining, Leaps is another approach. Drools has implementations for both Rete and Leaps. The Drools Rete implementation is called ReteOO signifying that Drools has an enhanced and optimised implementation of the Rete algorithm for Object Oriented systems. Other Rete based engines also have marketing terms for their proprietary enhancements to Rete, like RetePlus and Rete III. It is important to understand that names like Rete III are purely marketing where, unlike the original published Rete Algorithm, no details of implementation are published; thus asking a question like "Does Drools implement Rete III?" is nonsensical. The most common enhancements are covered in "Production Matching for Large Learning Systems (Rete/UL)" (1995) by Robert B. Doorenbos

    Business Rule Management Systems build value on top of an Rule Engine providing systems for rule management, deployment, collaboration, analysis and end user tools for business users. Further to this the Business Rules Approach is a fast evolving and popular methodology helping to formalise the role of Rule Engines in the enterprise.

    For more information read the following two chapters from the manual:
    Introduction and Background
    Knowledge Representation

    Post Comment
    Share/Bookmark