Monday, March 30, 2009

Drools Quick Start - Stateful Knowledge Session

Here is an extract from the quick start guide I just writing on using stateful sessions:

Stateful sessions are longer lived and allow iterative changes over time. Some common use cases for stateful sessions are, but not limited to:
  • Monitoring
    • Stock market monitorig and analysis for semi-automatic buying
  • Diagnostics
    • fault finding, medical diagnostics
  • Logistics
    • parcel tracking and delivery provisioning
  • Compliance
    • Validation of legality for market trades.
Unlike a stateless session the dispose() method must be called afterwards to ensure there are no memory leaks, as the KnowledgeBase containes references to StatefulKnowledgeSessions when they are created. StatefulKnowledgeSession also supports the BatchExecutor interface like StatelessKnowledgeSession, the only difference is that when used with stateful the FireAllRules command is not automatically called at the end.

We can use a fire alarm example to explore the monitoring use case. The simple example has just 4 classes. We are monitoring the rooms in a house, each room has one sprinkler. If a fire starts in a room, we represent that with a single Fire instance.
public class Room {
private String name;
// getter and setter methods here
}

public classs Sprinkler {
private Room room;
private boolean on;
// getter and setter methods here
}

public class Fire {
private Room room;
// getter and setter methods here
}

public class Alarm {
}
In the previous section on stateless sessions the concepts of inserting and matching against data was introduced. That example assumed only a single instance of each object type was ever inserted and thus only used literal constraints. However a house has many rooms, so rules have the need to express joins that constraint to the desired objects, this can be done using a binding as a variable constraint in a pattern. This join process results in what is called cross products, which are covered in the next section.

When a fire occurs an instance of the Fire class is created, for that room, and insert it. The rule uses a binding on the room field of the Fire to constrain to the Sprinkler for that room, which is currently off. When this rule fires and the consequence is executed the sprinkler is turned on
rule "When there is a fire turn on the sprinkler"
when
Fire($room : room)
$sprinkler : Sprinkler( room == $room, on == false )
then
modify( $sprinkler ) { setOn( true ) };
System.out.println( "Turn on the sprinkler for room " + $room.getName() );
end
Where as the stateless session used standard java syntax to modify a field, in the above rule we use the modify keyword, which acts as a sort of with statement, that contains a series of comma separated java expressions. Stateless sessions typically do not use inference, which can be explicitly turned off by using the "sequential mode", so the engine does not need to be aware of changes to data, however a stateful session does. The modify keyword allows the setters to modify the data, while make the engine aware of those changes so it can reason over them, this process is called inference.

So far we have rules that tell us when matching data exists, but what about when it doesn't exist? What about when there stops being a Fire? Previously the constraints have been propositional logic where the engine is constraining against individual intances, Drools also has support for first order logic that allows you to look at sets of data. The 'not' keyword matches when something does not exist. So for a Room with a Sprinkler that is on when the Fire for that room stops existing we can turn off the sprinkler.
rule "When the fire is gone turn off the sprinkler"
when
$room : Room( )
$sprinkler : Sprinkler( room == $room, on == true )
not Fire( room == $room )
then
modify( $sprinkler ) { setOn( false ) };
System.out.println( "Turn off the sprinkler for room " + $room.getName() );
end
While there is a Sprinkler per room, there is just a single Alarm for the building. An Alarm is created when a Fire is occurs, but only one Alarm is needed for the entire building, no matter how many Fires occur. Previously 'not' was introduced, the compliment to ths is 'exists' which matches for one or more of something.
rule "Raise the alarm when we have one or more fires"
when
exists Fire()
then
insert( new Alarm() );
System.out.println( "Raise the alarm" );
end
Likewise when there are no Fires we want to remove the alarm, so the 'not' keyword can be used again.
rule "Lower the alarm when all the fires have gone"
when
not Fire()
$alarm : Alarm()
then
retract( $alarm );
System.out.println( "Lower the alarm" );
end
Finally there is a general health status message, that is printed when the application first starts and after the Alarm is removed and all Sprinklers have been turned off.
rule "Status output when things are ok"
when
not Alarm()
not Sprinkler( on === true )
then
System.out.println( "Everything is ok" );
end
The above rules should be placed in a single drl file and saved to the classpath using the file name "fireAlarm.drl", as per the stateless session example. We can then build a KnowledgeBase as before, just using the new name "fireAlarm.drl". The difference is this time we create a stateful session from the kbase, where as before we created a stateless session.
KnowledgeBuilder kbuilder = KnowledgeBuilderFactory.newKnowledgeBuilder();
kbuilder.add( ResourceFactory.newClasspathResource( "fireAlarm.drl", getClass() ),
ResourceType.DRL );
if ( kbuilder.hasErrors() ) {
System.err.println( builder.getErrors().toString() );
}
StatefulKnowledgeSession ksession = kbase.newStatefulKnowledgeSession();
With the session created it is now possible to iteratvely work with it over time. Four Rooms are created and inserted, a Sprinkler for each room is also inserted. At this point the engine has done all it's matching, but no rules have fired. calling "fireAllRules" on the ksession allows the matched rules to fire, currently that is just the health message.
Room kitchen = new Room( "kitchen" );
Room bedroom = new Room( "bedroom" );
Room office = new Room( "office" );
Room livingRoom = new Room( "livingroom" );

ksession.insert( kitchen );
ksession.insert( bedroom );
ksession.insert( office );
ksession.insert( livingRoom );

Sprinkler kitchenSprinkler = new Sprinkler( kitchen );
Sprinkler bedroomSprinkler = new Sprinkler( bedroom );
Sprinkler officeSprinkler = new Sprinkler( office );
Sprinkler livingRoomSprinkler = new Sprinkler( livingRoom );

ksession.insert( kitchenSprinkler );
ksession.insert( bedroomSprinkler );
ksession.insert( officeSprinkler );
ksession.insert( livingRoomSprinkler );

ksession.fireAllRules()

> Everything is ok
We now create two fires and insert them, this time a referenced is kept for the returned FactHandle. The FactHandle is an internal engine reference to the inserted instance and allows that instance to be retracted or modified at a later point in time. With the Fires now in the engine, once "fireAllRules" is called, the Alarm is raised and the respectively Sprinklers are turned on.
Fire kitchenFire = new Fire( kitchen );
Fire officeFire = new Fire( office );

FactHandle kitchenFireHandle = ksession.insert( kitchenFire );
FactHandle officeFireHandle = ksession.insert( officeFire );

ksession.fireAllRules();

> Raise the alarm
> Turn on the sprinkler for room kitchen
> Turn on the sprinkler for room office


After a while the fires will be put out and the Fire intances are retracted. This results in the Sprinklers being turned off, the Alarm being lowered and eventually the health message is printed again.
ksession.retract( kitchenFireHandle );
ksession.retract( officeFireHandle );

ksession.fireAllRules();

> Turn on the sprinkler for room office
> Turn on the sprinkler for room kitchen
> Lower the alarm
> Everything is ok
Every one still with me? That wasn't so hard and already I'm hoping you can start to see the value and power of a declarative rule system.

Sunday, March 29, 2009

Drools Boot Camp : San Francisco 1-7 June 2009 Now On

The Drools Boot Camp has the go ahead, a big thank you to Third Pillar for sponsoring the meeting room.

The Drools team meeting forms the basis of the week long event, but we make it open to the public, free of charge, to participate as an informal boot camp. You may attend as little or as much of the event as you wish, there is no commitment, just please give us notification of when you wish to attend as there are limited spaces ( mproctor at redhat d0t com ).

We will provide some dedicated seminars, including a Drools 5.0 primer on the first day for those that want to know what's new in Drools 5.0 which is followed by community based lighting talks. If you wish to give a lightning talk then please contact me asap.

The Drools Boot Camp registration page is now open. If you are hoping to attend, please put your details there and send me an email, so that we can estimate and manage attendance. We only have spaces for about 20 or so people, so please do hurry. Registration wiki page can be found here, you will need to login to edit the page.

Location
The location is San Francisco, during the JavaOne period, at the Hilton Hotel (750 Kearny Street). Rooms are available from $103 per night, at the booking time for my team and I.


Dates and Times
Start : Monday 1st of June 09.30
End : Saturday 7th of June 17.00

Time Tabled Sessions
We plan to do a few time tabled sessions. The current proposals are subject to change, and suggestions are welcome - we don't plan to do a "getting started" or "best practice" session. Other than the primer the sessions will be in BOF casual discussion style seminar with an emphasis on audience particitation for Q&A and suggestions, not dedicated slides with pre-made material, beyond useful illustrations, so expect lots of walking through code and white board use :) It can also be a great time to give feedback and suggestions for those topics.

Drools 5.0 Primer : Monday 1st of June 10.00 to 13.00
  • The new features in Drools 5.0 will be covered at an end user level. This includes all the drools projects, Guvnor, Flow, Fusion and Expert. This is suitable for anyone with minimal previous Drools experience who just want to know "what's new".
Community Lightning Talks : Monday 1st of June 13.30 to 14.30
  • 20 minute community lightning talks about their experiences with drools.
    • Tomasz Krywicki, from Sun, will talk about Drools in some of their software.
    • Adam Mollenkopf, from Fedex Custom Critical, will present "Using Drools for Tracking and Situational Awareness".
    • Ales Justin, JBoss Microcontainer Lead, will discuss MC integration with Drools.
Drools 5.0 Guvnor Internals : Tuesday 10.00 to 12.00
  • The internals of Guvnor will be discussed and how all the parts come together and current issues and directions we wish to take the project. This is suitable for anyone with minimal previous Drools experience, but ideally you should be aware of GWT, JCR and Guvnor in general. This is targetted at people who wish to extend Guvnor with new features or re-use/embed existing aspects of Guvnor.
Drools 5.0 Domain Specific Workflow : Wednesday 10.00 to 12.00
  • We will show how Drools Flow can be extended for domain specific workflow, with a focus on the pluggable Work Items and Eclipse. We will also cover the pluggable workflow framework for customised execution behaviour, such as is used to provide OSWorkflow compatability. This is suitable for anyone with a java and eclipse background, ideally you should have some understanding of what Drools and especially Drools Flow is.
Drools 5.0 Advanced Engine : Thursday 10.00 to 13.00
  • This will be a deep dive for the technical on Rete enhancements. We will cover the asymmetrical Rete implementation, as compared to the traditional symmetrical approach, and how we enhanced it for event processing and workflow. This is targetted at experienced people, who already have an understanding of Rete and wish to understand the latest research and innovations - not for the feint hearted.
If you are using Drools in a novel or interesting way and wish to do a short presentation, that is very much welcome, just please email me for confirmation so we can make sure we are prepared and even promote that. If a few people wish to do this, we can make a dedicated end-user speaking session.

Agenda and what goes on
While we have the time tabled sessions, there is no formal agenda beyond that. The Drools team will be there planning their next road map, hacking on new features and fixing some bugs - this will be a continuous process throughout the week. We encourage you to participate and ideally contribute and we will provide lots of hand holding and mentoring. Outside of the time tabled sessions you'll probably feel out of place if you aren't coding away on a laptop.

Ideally, but not essential, you will already have ideas on areas you wish to improve and we'll assist. Last year we got great contributions to Guvnor from Joe at Recondo Technologies and Michael at Franklin American, neither had any prior experience on hacking Drools, although both turned up knowing what it was they needed added to Drools for their companies.

Limited time will be given to those just wishing to discuss how to use Drools in their applications and best practices, but not looking to do any coding. The basis of this time will be on how interesting it is to the Drools team, and whether their is a knowledge exchange that is useful to us - i.e. interesting and uncommon use cases. This event really is summarised as "Don't ask what it is Drools can do for me, but what I can do for Drools" :)

Here are two pictures from last year, where we fitted everyone out with Drools T-Shirts :)

Saturday, March 28, 2009

The first Drools book is on the way from Packt Publishing

The first Drools book is on the way from Packt Publishing:
JBoss Drools Business Rules
http://www.packtpub.com/jboss-drools-business-rules/book

The author of the book is Paul Brown who many of you will know from the mailing list and his O'Reilly articles:
Using Drools in Your Enterprise Java Application
Give Your Business Logic a Framework with Drools

Here is copy/paste from the website for the chapter outlines:

What you will learn from this book
  • Understand the basics of business rules and JBoss rules with minimal effort
  • Install the required software easily and learn to use the Guvnor, which is a user-friendly web editor that's also powerful enough to test our rules as we write them
  • Learn to write sophisticated rules and import the fact model into the Guvnor and then build a guided rule around it, which makes your web pages a lot clearer
  • Gain complete knowledge of what we can do with the Guvnor rule editor, and then use the JBoss IDE as an even more powerful way of writing rules, and automate processes for discounts, orders, sales, and more
  • Know the structure of the rule file through the example of a shipping schedule, which will help you with your own shipping schedule
  • Test your rules not only in the Guvnor, but also using FIT for rule testing against requirements documents; run unit tests using JUnit for error-free rules and interruption-free services
  • Specifically, non-developers can work with Excel spreadsheets as a fact model to develop business processes without learning any other new technology
  • Work with DSLs (Domain-Specific Languages) and rule flow to make writing rules easy; which makes staff training quicker and your working life easier
  • Deploy your business rules to the real world, which completes your project successfully, and combine this into a web project using the framework of your choice to provide better services
  • Benefit from concepts such as truth maintenance, conflict resolution, pattern matching rules agenda, and the Rete algorithm to provide advanced and faster business systems so that staff efficiency is maximized

Chapter 1, Drooling over JBoss Rules. This chapter gives you a good platform to understand business rules and JBoss rules. We look at the problems that you might have (and why you're probably reading this book). We look at what business rule engines are, and how they evaluate business rules that appear very simple and how they become powerful when multiple rules are combined.

Chapter 2, Getting the software, deals with installation. This chapter explains setting up Java, setting up Business Rule Management System (BRMS)/Guvnor running on the JBoss App Server, setting up Eclipse, and installing the Drools Plug-in. It also details the installation of the Drools examples for this book and the Maven to build them.

Chapter 3, Meet the Guvnor, deals with writing our rules using the 'Guvnor'. Guvnor is the user-friendly web editor that's also powerful enough to test our rules as we write them. We take up an example to make things easier. Then we look at the various Guvnor screens, and see that it can not only write rules (using both guided and advanced editors), but that it can also organize rules and other assets in packages, and also allow us to test and deploy those packages. Finally, we write our very first business rule—the traditional 'Hello World' message announcing to everyone that we are now business rule authors.

Chapter 4, Guided Rules with the Guvnor. This chapter shows how to use the Guvnor rule editor to write some more sophisticated rules. It also shows how to get information in and out of our rules, and demonstrates how to create the fact model needed to do this. We import our new fact model into the Guvnor and then build a guided rule around it. Finally we test our rule as a way of making sure that it runs correctly.

Chapter 5, From Guvnor to JBoss IDE. This chapter pushes the boundries of what we can do with the Guvnor rule editor, and then brings in the JBoss IDE as an even more powerful way of writing rules. We start by using variables in our rules example. Then we discuss rule attributes (such as salience) to stop our rules from making changes that cause them to fi re again and again. After testing this successfully, we look at text-based rules, in both the Guvnor and the JBoss IDE, for running 'Hello World' in the new environment.

Chapter 6, More Rules in the jBoss IDE. This chapter looks again at the structure of a rule fi le. At the end of this chapter, we look at some more advanced rules that we can write and run in the IDE.

Chapter 7, Testing your Rules. This chapter explains how testing is not a standalone activity, but part of an ongoing cycle. In this chapter we see how to test our rules, not only in the Guvnor, but also using FIT for rule testing against requirements documents. This chapter also explains Unit Testing using JUnit.

Chapter 8, Data in Excel, Rules in Excel. This chapter explains how to use Excel Spreadsheets (cells and ranges) as our fact model to hold information, instead of the write-your-own-JavaBean approach we took earlier. Then we use Excel spreadsheets to hold Decision tables, to make repetitive rules easier to write.

Chapter 9, Domain-Specific Languages [DSL] and rule flow. This chapter aims to make our rules both easier to use, and more powerful. We start with DSLs—Domain-Specifi c Languages. This chapter follows on from the 'easy to write rules' theme from the previous chapter and also discusses both ruleflow and workflow. It would be great to draw a workfl ow diagram to see/control what (groups of) rules should fi re and when. Rule flow gives us this sort of control.

Chapter 10, Deploying rules in real life. This chapter shows you how to deploy your business rules into the real world. We look at the pieces that make up an entire web application, and where rules fit into it. We see the various options to deploy rules as part of our application, and the team involved in doing so. Once they are deployed, we look at the code that would load and run the rules—both home-grown and using the standard RuleAgent. Finally we see how to combine this into a web project using the framework of your choice.

Chapter 11, Peeking under the covers. This chapter looks at what happens under the cover by opening up the internals of the Drools rule engine to understand concepts such as truth maintenance, confl ict resolution, pattern matching, and the rules agenda. In this chapter, we explore the Rete algorithm and discuss why it makes rules run faster than most comparable business logic. Finally we see the working memory audit log and the rules debug capabilities of the Drools IDE.

Chapter 12, Other Drools features. This chapter deals with the other advanced Drools features that have not yet been covered. This includes Smooks to bulk load data, Complex Event Processing, and Drools solver to provide solutions where traditional techniques would take too long.
Approach
This book takes a practical approach, with step-by-step instructions. It doesn't hesitate to talk about the technologies, but takes time to explain them (to an Excel power-user level). There is a good use of graphics and code where necessary.
Who this book is written for
If you are a business analyst – somebody involved with enterprise IT but at a high level, understanding problems and planning solutions, rather than coding in-depth implementations – then this book is for you.

If you are a business user who needs to write rules, or a technical person who needs to support rules, this book is for you.

If you are looking for an introduction to rule engine technology, this book will satisfy your needs.

If you are a business user and want to write rules using Guvnor/JBoss IDE, this book will be suitable for you.

This book will also suit your need if you are a business user and want to understand what Drools can do and how it works, but would rather leave the implementation to a developer.

Thursday, March 26, 2009

Who do you know and how do you know them?

No, this isn't a police interrogation. This article is about facilitating the management of complex object model relationships within the context of a rules engine. Rule engines are fantastic at solving a vast array of business problems with the greatest of ease, however, with a caveat being that any object you want to reason over must be in working memory.

One of the problems I encountered when working with Drools was that the developer was faced with the task of asserting objects into working memory. Coming from an ArtEnterprise background this was a bit odd to me. ArtEnterprise is a forward chaining inference engine derived from ART(Automated Reasoning Tool)/CLIPS(C Language Integrated Production System). It is a powerful and feature rich expert system development language with a Common Lisp syntax.

In ArtEnterprise, there is no distinction between working memory and non-working memory; If you want to write rules to reason over some set of objects, you simply do it. In other words, you do not have to worry about an object being in memory or not. After some research, I soon learned that many Java based rules engines such as JESS and iLog also require you to actively assert the objects your rules will reason over. This is quite an onus to place on the developer, especially in the case of complex object models.

To say that the object model I was dealing with was complex, along with the business rules that were needed, is an understatement. The client, a large availability services company, was looking for an agile approach to dealing with ever changing business rules specific to individual clients. Without a solution to facilitate the assertion of objects, I would not be able to sell a rules based approach.

Enter the Java Persistence API (JPA). JPA is a persistence framework that allows developers to map Java domain objects to tables in a relational database. This object/relational mapping is expressed either directly in the domain object via Java Annotations, or in a separate XML file bundled with the application.

Fortunately for me, as it turned out, our project utilized the annotation based mappings. For example, our object model contains a Server object to represent, not surprisingly, a physical server . The following is a snippet of the class declaration
@Entity
public class Server extends {
@ManyToOne
private Architecture architecture;

@OneToMany
private Set<OperatingSystem> operatingSystems;

@OneToOne
private Keyboard keyboard;
}
The @Entity annotation on the Server class declaration specifies that this class should be persisted. The @ManyToOne annotation on the architecture field means that many servers can have the same architecture. While the @OneToMany annotation on the operatingSystems field means that a server can have multiple operating systems. And finally, the @OneToOne annotation on the keyboard field means that a server can only have a single keyboard attached.

By inspecting all of the classes and fields from the object model via reflection, I was able to determine which were entities. The following is a code snippet that is used to determine if a Class is persistable via JPA:
protected boolean isPersistentEntity(Class clazz) {
return clazz.isAnnotationPresent(Entity.class) ||
clazz.isAnnotationPresent(MappedSuperclass.class) ||
clazz.isAnnotationPresent(Embeddable.class);
}
Associated collections and maps were handled similarly except there was an additional level of indirection to check the generic type the collection held.
protected void handleCollectionField(Class clazz, Field field) {
ParameterizedType parameterizedType = (ParameterizedType) field.getGenericType();
Type types[] = parameterizedType.getActualTypeArguments();

for (Type parameterType : types) {
Class parameterClass = (Class) parameterType;

if (isPersistenantEntity(parameterClass)) {
...
}
}
}
The final piece of the puzzle was to use this information along with Velocity to generate traversal rules. Velocity is a templating engine that can be used to generate code based on data and a template defined in Velocity Template Language (VTL). The engine takes a template and merges it with the data to produce some type of output. The data, in my case, are the class and field information. The template is of the rule that will be generated. The following is the template I defined for handling one-to-one and many-to-one associations:
#set( $ruleName = $className +  $fieldName + "FieldWalker")

rule '$ruleName'
agenda-group "$constants.TRAVERSAL_AGENDA_GROUP"
when
$className : $className ($fieldName: $fieldName != null)
then
insert($$fieldName);
end
Rules were generated for each association an object has. Please note that this generation is happening at build time, not runtime. For example, given the previous Server class declaration, the following rules would be generated:
rule 'ServerarchitectureFieldWalker'
agenda-group "Traversal"
when
$Server : Server($architecture : architecture != null)
then
insert($architecture);
end

rule 'ServerkeyboardFieldWalker'
agenda-group "Traversal"
when
$Server : Server($keyboard : keyboard != null)
then
insert($keyboard);
end
So what this means is that anytime a Server object is asserted into memory, the Architecture, Keyboard, and any other associations the server has will automatically be asserted also. Since Architecture and Keyboard are also persistable, any associations they have will also be asserted, and so on.

I also introduced the capability to mark either a Class or association as "non traversable". For example, say that your Server is within a Rack, and the rack contains multiple servers. If we would not like the server's rack to be asserted into memory we can markup the rack field on the server with a @DoNotTraverse annotation to to prevent this from happening.
@Entity
public class Server extends {
@ManyToOne
private Architecture architecture;

...

@DoNotTraverse
private Rack rack;
}
With these rules now in place, the developer is now freed from having to worry about whether they have all objects asserted into working memory or not. This allows them to focus on solving the "real" business problems.



David Sinclair is an independent software designer specializing in rules based programming. David frequently works with Chariot Solutions, one of the top Java and open source consultanting firms in the Mid-Atlantic region.

Edson off on holidays

Edson has left on holidays, so I thought I'd share this photo of him leaving on the train.

Tuesday, March 24, 2009

More translations

There are some more translations in the works, thanks to some community members:
  • Japanese by Shigeaki Wakizaka
  • Simplified Chinese by Richard Qin
I have committed these into trunk, but they are not yet available as part of the build while they are worked on/reviewed etc. But you can find them in the messages package, and its possible for people to enable them. Soon we will bake it into the build so they are available the usual way (via locale settings).

(below is a screenshot of the Japanese version running in Google Chrome)The main downside is the extra build times for GWT (and slight increase of war size - as GWT essentially builds different front ends (so the end user doesn't pay a price for all the versions available). But no one else should notice any downside to this.

If you are interested in helping out, take a look at them (any changes should be against the trunk version if possible).

Friday, March 20, 2009

XMLUnit for xml assertions

Thought i'd blog this little snippet. Was struggling with xml comparisons for unit testing, I initially tried XMLUnit but found that it was too sensitive. After a bit of digging in the docs, I found how to configure it to be much more forgiving on element ordering, whitespace etc. Here is what I did, hopefully others will find this useful:

protected void setUp() throws Exception {
XMLUnit.setIgnoreComments( true );
XMLUnit.setIgnoreWhitespace( true );
XMLUnit.setIgnoreAttributeOrder( true );
XMLUnit.setNormalizeWhitespace( true );
XMLUnit.setNormalize( true );
}

private void assertXMLEqual(String expectedXml,
String resultXml) {
try {
Diff diff = new Diff( expectedXml,
resultXml );
diff.overrideElementQualifier( new RecursiveElementNameAndTextQualifier() );
XMLAssert.assertXMLEqual( diff,
true );
} catch ( Exception e ) {
throw new RuntimeException( "XML Assertion failure",
e );
}
}

Wednesday, March 18, 2009

Google Summer of code

Its the time of year for Google summer of code applications. JBoss (with Fedora) has been accepted again as a mentoring organisation. 

"Students" can apply to work on a project (Google pays them). The JBoss proposals (so far) are here on the wiki. (It is possible to suggest other ones). Obviously the ones on that wiki have mentors etc (keen for the drools one of course myself !). 

You can read more about Googles Summer of Code thing here

So if you are interested, please apply (or forward on to anyone else).


Drools Use Case: Telecom

by Luciano Morozowski Junior

Here at Auster Solutions, we have been working with Drools since its version 2.0, back in 2004. We were developing a product for Telecom Rating & Billing data processing and selected Drools for its rules processing module. Now that our product is out there and running Drools 4 for some time, it is time to give some feedback to the project.

Here in Brazil, especially inside big companies, we usually have a hard time to use or embed open source software, probably due to lack of support and formality they suffered years ago. On the other hand, since Drools was already demonstrating high technical quality, increasing adoption and constant evolution, it has turned out to be regarded as a serious open source tool, which eventually enabled its adoption inside our solution.

Now let me go through the technical info. Inside our rules processing module, the unit of work is called request. Several requests might be processed at the same time, within multiple threads.

Each working memory contains the facts related to a single request (actually we do it twice, but it will be explained later on). The size of the requests varies widely, so we can find tiny requests that generate less than a dozen facts, as well as we can have tens of thousands of facts being asserted from a single, yet very large, request. Just out of curiosity, the largest requests might even exceed 100,000 facts.

We have collected some data from one of our customer’s production environment. The table below divides requests into 5 differently-sized groups and presents the average amount of facts and processing time for each request. The solution consists of about 300 rules, running on a Fujitsu pw650 server with 8 CPUs, 16GB memory, Solaris 10, Java SE 5, and Drools 4.







Avg # of FactsAvg. Processing Time
Smallest90.3s
Small1150.4s
Mid-size5800.7s
Large61003.9s
Largest3600022.6s
Overall Avg7330.8s


Please note that, in an attempt to evaluate the tool by itself, both the application startup and I/O operations (file and DB access), which represents about 90% of the whole processing time, have been ignored, i.e., we have considered exclusively the time it took to allocate a working memory, set the globals, assert the facts (LHS), fire all rules (RHS), and dispose the session.

Now let me go back to explain the two-working-memory-per-request issue. The process for each request can be roughly divided in two steps: enrichment and validation. Enrichment consists of rules that basically add data to the facts in the working memory, in order to enrich them, while validation is the main processing phase that generates the final results of the processing.

At first, the implementation of the rules processing module had a single stateful working memory for each request and the rule base had all enrichment and validation rules together. Enrichment rules had, of course, a higher salience than the validation ones.

Some time later, while trying to improve the rules processing performance, we noticed that it would be faster if we created two working memories for each request, one (stateless) for enrichment, and other (stateful) for validation.

We found out that the overhead of allocating a working memory and asserting all facts twice was largely superseded by the fact that each update executed during enrichment step would no longer affect the agenda. Another important detail is that the actual performance gain was due to the sequential mode, which is available only on a stateless session.

Before we finish this article, we would like to illustrate the magnitude of the rules processing our system has performed so far only for one of our customers: more than 16 million requests have been processed during the past 4 years. Just last month, the application processed up to 390,000 requests, comprising nearly 200,000,000 facts evaluated by Drools.

In the final analysis, despite a few obstacles we have found along the way, Drools is working very well. We were able to achieve a good development throughput and short maintenance turnaround, a relatively short learning curve, and a fairly acceptable performance.

-------------

Luciano Morozowski Junior is a software architect at Auster Solutions. He has been working with application performance improvement since 2006 and specializing in Drools for more than one year now.

Auster Solutions is a Brazilian software company focused on the Telecom Industry, specially on the OSS & BSS processes chain. Auster has been running Drools inside its products since 2005.

Monday, March 16, 2009

Drools Boot Camp and San Francisco

Third Pillar just came through and offered to sponsor the meeting room. I'm just waiting on my managers approval, but hopefully tomorrow I can put up a blog confirming the Boot Camp will go ahead with dates and times. Now all we need are sponsors for sundries, like a constant stream of coffee and pizza :)


Initial dates look likely to co-incide with JavaOne, as I'm presenting there. So from the 1st to the 7th of June, inclusive, and the location is looking likely to be the Hilton, where rooms are currently $103 through Expedia.

The Hilton (SF)

Drools Boot Camp and San Francisco

I'm still trying to get budget approval. We are looking at the Hilton at the moment as expedia have rooms at $103 per day, which is very good value. I can get travel and room budget for my team, but not meeting room budget. The meeting room is $400 per day, I might be able to negotiate a little off that, and we will need "sponsors" to cover the price of the meeting room for the week, if this is to go ahead.

We are looking at a date range of:
arriving - 31 of May
departure - 8th of June

So we'll need a room from the 1st to the 7th, inclusive. If you are able to sponsor the meeting room for a day or two, or more :), please do let me know as soon as possible.

Thanks

Mark

Saturday, March 14, 2009

Marshalling with MVEL

Till now MVEL was only able to read in MVEL data structures, not marshall them out to a string, stopping MVEL from being used as a marshalling format. I needed something different to do for a day so I wrote a quick MVEL marshaller, that takes a pojo structure and writes it out as an MVEL statement. It's only a small bit of code and while spiking it I've left it in the main test class, so you can see everything currently here in MarshallingTest. testMVEL shows it working, there is a testXStream so you can compare it, there is a counter so you can do perf comparisons. We could probably speed it up some more if we use code generation instead of reflection.

This shows the code that generates the original pojo structure:
Pet pet = new Pet();
pet.setName( "rover" );
pet.setAge( 7 );
List list = new ArrayList();
list.add( "a" );
list.add( 12 );
list.add( new SomeNumers( 10.02f, 22.02, 5, 100l, new BigDecimal( 23.0234d, MathContext.DECIMAL128 ), new BigInteger( "1001" ) ) );
list.add( new Date() );
list.add( new Cheese( "cheddar", 6 ) );

pet.setList( list );
pet.setArray( new int[]{1, 2, 3} );

Map map = new HashMap();
map.put( "key1", 13 );
map.put( "key3", "value3" );
map.put( "key2", 15 );
map.put( "key4", new Cheese( "stilton", 11 ) );

Person person = new Person();
person.setName( "mark" );
person.setAge( 33 );
person.setPet( pet );
person.setSomeDate( new Date() );
person.setMap( map );
Calendar cal = Calendar.getInstance();
cal.setTime( new Date() );
person.setCal( cal );


Which generates the following MVEL statement:
new org.mvel2.marshalling.MarshallingTest$Person().{ age = 33, cal = 
with ( java.util.Calendar.getInstance() ) { time = new
java.util.Date(1237047853121)} , map = [ 'key1':13, 'key3':'value3',
'key2':15, 'key4':new org.mvel2.marshalling.MarshallingTest$Cheese().{
age = 11, edible = false, type = 'stilton' } ] , name = 'mark', nullTest
= null, pet = new org.mvel2.marshalling.MarshallingTest$Pet().{ age = 7,
array = { 1, 2, 3 } , list = [ 'a', 12, new
org.mvel2.marshalling.MarshallingTest$SomeNumers().{ ABigDecimal =
23.02339999999999875512912694830447, ABigInteger = 1001, ADouble =
22.02, AFloat = 10.02, AInt = 5, ALong = 100 }, new
java.util.Date(1237047853046), new
org.mvel2.marshalling.MarshallingTest$Cheese().{ age = 6, edible =
false, type = 'cheddar' } ] , name = 'rover' }, someDate = new
java.util.Date(1237047853121) }

Which is a lot nicer than XML or JSON, we can improve this further if we add imports to avoid fully qualifying every Class and add a nice pretty printer :)

Friday, March 13, 2009

Business Intelligence integration with Pentaho and Drools

Someone just pointed out that Pentaho now have some Drools integration. Here is a snippet from the introdouction:
"The aim of this project is to provide the Pentaho platform with a business rules engine to support business logic validation. The rules engine will be embedded in a Pentaho component, which will allow flexible deployment, interaction with other components and reuse throughout the system."

And the team:
James Dixon - Chief Geek at Pentaho, GSOC 'Mentor'
Alessio Tomasino - GSOC 'Student'
Role of Product Owner - James Dixon
Role of Unblocker - James Dixon

Would love to hear more about this project, whether anyone is using it, or if it provides value.

Thursday, March 12, 2009

Drools Boot Camp possibly in San Francisco for June

As I'm going to be at JavaOne we are thinking about combing this years boot camp at the same time, if we can get budget approval. We don't yet have a hotel booked or meeting room available, so if you can recommend a cheap but good hotel with conference facilities for approx 15-20 people, or better still can donate meeting room facilities then please let me know.

Thanks

Mark

Tuesday, March 10, 2009

Drools 5.0 CR1 New and Noteworthy Release Summary

Previous New and Noteworth release summary notes:
M1
M2
M3/M4
M5

Drools Guvnor

Guvnor of course has some fixes on usability, security and bugs. But the biggest new thing is that Guvnor can now be translated. Default language is English and currently you can also use Spanish. Date formats and default languages can be set with drools.dateformat, drools.defaultlanguage and drools.defaultcountry in preferences.properties file. These properties are then used when Guvnor builds packages.

Drools Flow

JPA

Firstly, we have improved our support for persistence (JPA) and transactions (JTA). An example on how to use persistence and transactions in combination with processes:
// create a new JPA-based session and specify the JPA entity manager factory
Environment env = KnowledgeBaseFactory.newEnvironment();
env.set( EnvironmentName.ENTITY_MANAGER_FACTORY, Persistence.createEntityManagerFactory( "emf-name" ) );
env.set( EnvironmentName.TRANSACTION_MANAGER, TransactionManagerServices.getTransactionManager() );

StatefulKnowledgeSession ksession = JPAKnowledgeService.newStatefulKnowledgeSession( kbase, null, env ); // KnowledgeSessionConfiguration may be null, and a default will be used
int sessionId = ksession.getId();

// if no transaction boundary is specified, the method invocation is executed in a new transaction automatically
ProcessInstance processInstance = ksession.startProcess( "org.drools.test.TestProcess" );

// the users can also specify the transaction boundary themselves
UserTransaction ut = (UserTransaction) new InitialContext().lookup( "java:comp/UserTransaction" );
ut.begin();
ksession.insert( new Person( "John Doe" ) );
ksession.startProcess( "org.drools.test.TestProcess" );
ksession.fireAllRules();
ut.commit();
Variable Injection

Secondly, we now support direct access to process variables in both MVEL and Java in code constraints and actions, so if you have a variable called "person" in your process, you can now describe constraints like:
  • [Java code constraint] return person.getAge() > 20;
  • [MVEL action] System.out.println(person.name);

Eclipse plugin

Some minor bug fixes were added related to the Eclipse plugin:
  • The audit log now also supports the old Drools4 format.
  • Drools runtimes now support the directory structure of the binary download.
  • Fixed a template to create a new RuleFlow process
  • Issue with changes to an action node not being detected fixed

Drools Core and Fusion

Type safe configuration

In addition to the ability of configuring options in drools through configuration files, system properties and by setting properties through the API setProperty() method, Drools-API now supports type safe configuration.

We didn't want to add specific methods for each possible configuration methods for two reasons: it polutes the API and every time a new option is added to Drools, the API would have to change. This way, we followed a modular, class based configuration, where a new Option class is added to the API for each possible configuration, keeping the API stable, but flexible at the same time.

So, in order to set configuration options now, you just need to use the enumerations or factories provided for each option. For instance, if you want to configure the knowledge base for assert behavior "equality" and to automatically remove identities from pattern matchings, you would just use the enums:
KnowledgeBaseConfiguration config = KnowledgeBaseFactory.newKnowledgeBaseConfiguration();
config.setOption( AssertBehaviorOption.EQUALITY );
config.setOption( RemoveIdentitiesOption.YES );
For options that don't have a predefined constant or can assume multiple values, a factory method is provided. For instance, to configure the alpha threshold to 5, just use the "get" factory method:
config.setOption( AlphaThresholdOption.get(5) );
As you can see, the same setOption() method is used for the different possible configurations, but they are still type safe. The following test cases show examples on how to configure KnowledgeBuilder options and KnowledgeBase options:

KnowledgeBaseConfigurationTest.java
KnowledgeBuilderConfigurationTest.java

New accumulate functions: collectSet and collectList

There are times when it is necessary to collect sets or lists of values that are derived from the facts attributes, but are not facts themselves. In such cases, it was not possible to use the collect CE. So, Drools now has two accumulate functions for such cases: collectSet for collecting sets of values (i.e., with no duplicate values) and collectList for collecting lists of values (i.e., allowing duplicate values):
# collect the set of unique names in the working memory
$names : Set() from accumulate( Person( $n : name, $s : surname ),
collectSet( $n + " " + $s ) )

# collect the list of alarm codes from the alarms in the working memory
$codes : List() from accumulate( Alarm( $c : code, $s : severity ),
collectList( $c + $s ) )
New metadata for type declarations: @propertyChangeSupport

Facts that implement support for property changes as defined in the Javabean(tm) spec, now can be annotated so that the engine register itself to listen for changes on fact properties. The boolean parameter that was used in the insert() method in the Drools 4 API is deprecated and does not exist in the drools-api module.
declare Person
@propertyChangeSupport
end
Batch Executor
No sooner had we introduced the parameters api in M3 than we have killed them off in favour of the new BatchExecutor interface and Command api. Batch Executor allows for the scripting of of a Knowledge session using Commands, which can also re, both the StatelessKnowledgeSession and StatefulKnowledgeSession implement this interface

Commands are created using the CommandFactory and executed using the "execute" method, such as the following insert Command:

ksession.execute( CommandFactory.newInsert( person ) ); 

Typically though you will want to execute a batch of commands, this can be achieved via the composite Command BatchExecution. BatchExecutionResults is now used to handle the results, some commands can specify "out" identifiers which it used to add the result to the BatchExecutionResult. Handily querries can now be executed and results added to the BatchExecutionResult. Further to this results are scoped to this execute call and return via the BatchExecutionResults:

List<Command> cmds = new ArrayList<Command>();
cmds.add( CommandFactory.newSetGlobal( "list1", new ArrayList(), true ) );
cmds.add( CommandFactory.newInsert( new Person( "jon", 102 ), "person" ) );
cmds.add( CommandFactory.newQuery( "Get People" "getPeople" );

BatchExecutionResults results = ksession.execute( CommandFactory.newBatchExecution( cmds ) );
results.getValue( "list1" ); // returns the ArrayList
results.getValue( "person" ); // returns the inserted fact Person
results.getValue( "Get People" );// returns the query as a QueryResults instance.
The CommandFactory details the supported commands, all of which can marshalled using XStream and the BatchExecutionHelper. This can be combined with the pipeline to automate the scripting of a session.
Action executeResultHandler = PipelineFactory.newExecuteResultHandler();
Action assignResult = PipelineFactory.newAssignObjectAsResult();
assignResult.setReceiver( executeResultHandler );
Transformer outTransformer = PipelineFactory.newXStreamToXmlTransformer( BatchExecutionHelper.newXStreamMarshaller() );
outTransformer.setReceiver( assignResult );
KnowledgeRuntimeCommand batchExecution = PipelineFactory.newBatchExecutor();
batchExecution.setReceiver( outTransformer );
Transformer inTransformer = PipelineFactory.newXStreamFromXmlTransformer( BatchExecutionHelper.newXStreamMarshaller() );
inTransformer.setReceiver( batchExecution );
Pipeline pipeline = PipelineFactory.newStatelessKnowledgeSessionPipeline( ksession );
pipeline.setReceiver( inTransformer );
Using the above for a rulset that updates the price of a Cheese fact, given the following xml to insert a Cheese instance using an out-identifier:
<batch-execution>
<insert out-identifier='outStilton'>
<org.drools.Cheese>
<type>stilton</type>
<price>25</price>
<oldPrice>0</oldPrice>
</org.drools.Cheese>
</insert>
</batch-execution>

We then get the following BatchExecutionResults:
<batch-execution-results>
<result identifier='outStilton'>
<org.drools.Cheese>
<type>stilton</type>
<oldPrice>0</oldPrice>
<price>30</price>
</org.drools.Cheese>
</result>
</batch-execution-results>
The following unit test XStreamBatchExecutionTest shows in much more detail how to use all the other commands, such as querries, and what the xml looks like. BatchExecutionHelper also documents the xml.

Marshalling

The MarshallerFactory is used to marshal and unmarshal StatefulKnowledgeSessions. At the simplest it can be used as follows:

// ksession is the StatefulKnowledgeSession
// kbase is the KnowledgeBase
ByteArrayOutputStream baos = new ByteArrayOutputStream();
Marshaller marshaller = MarshallerFactory.newMarshaller( kbase );
marshaller.marshall( baos, ksession );
baos.close();

However with marshalling you need more flexibility when dealing with referenced user data. To achieve this we have the ObjectMarshallingStrategy interface. Two implementations are provided, but the user can implement their own. The two supplied are IdentityMarshallingStrategy and SerializeMarshallingStrategy. SerializeMarshallingStrategy is the default, as used in the example above and it just calls the Serializable or Externalizable methods on a user instance. IdentityMarshallingStrategy instead creates an int id for each user object and stores them in a Map the id is written to the stream. When unmarshalling it simply looks to the IdentityMarshallingStrategy map to retrieve the instance. This means that if you use the IdentityMarshallingStrategy it's stateful for the life of the Marshaller instance and will create ids and keep references to all objects that it attempts to marshal. Here is he code to use a IdentityMarshallingStrategy.

ByteArrayOutputStream baos = new ByteArrayOutputStream();
Marshaller marshaller = MarshallerFactory.newMarshaller( kbase, new ObjectMarshallingStrategy[] { MarshallerFactory.newIdentityMarshallingStrategy() } );
marshaller.marshall( baos, ksession );
baos.close();

For added flexability we can't assume that a single strategy is suitable for this we have added the ObjectMarshallingStrategyAcceptor interface that each ObjectMarshallingStrategy has. The Marshaller has a chain of strategies and when it attempts to read or write a user object it iterates the strategies asking if they accept responsability for marshalling the user object. One one implementation is provided the ClassFilterAcceptor. This allows strings and wild cards to be used to match class names. The default is "*.*", so in the above the IdentityMarshallingStrategy is used which has a default "*.*" acceptor.

But lets say we want to serialise all classes except for one given package, where we will use identity lookup, we could do the following:

ByteArrayOutputStream baos = new ByteArrayOutputStream();
ObjectMarshallingStrategyAcceptor identityAceceptor = MarshallerFactory.newClassFilterAcceptor( new String[] { "org.domain.pkg1.*" } );
ObjectMarshallingStrategy identityStratetgy = MarshallerFactory.newIdentityMarshallingStrategy( identityAceceptor );
Marshaller marshaller = MarshallerFactory.newMarshaller( kbase, new ObjectMarshallingStrategy[] { identityStratetgy, MarshallerFactory.newSerializeMarshallingStrategy() } );
marshaller.marshall( baos, ksession );
baos.close();
Knowledge Agent

The KnowlegeAgent is created by the KnowlegeAgentFactory. The KnowlegeAgent provides automatic loading, caching and re-loading, of resources and is configured from a properties files. The KnowledgeAgent can update or rebuild this KnowlegeBase as the resources it uses are changed. The strategy for this is determined by the configuration given to the factory, but it is typically pull based using regular polling. We hope to add push based updates and rebuilds in future versions.

The Following example constructs an agent that will build a new KnowledgeBase from the files specified in the path String. It will poll those files every 30 seconds to see if they are updated. If new files are found it will construct a new KnowledgeBase, instead of updating the existing one, due to the "newInstance" set to "true" (however currently only the value of "true" is supported and is hard coded into the engine):

// Set the interval on the ResourceChangeScannerService if you are to use it and default of 60s is not desirable.
ResourceChangeScannerConfiguration sconf = ResourceFactory.getResourceChangeScannerService().newResourceChangeScannerConfiguration();
sconf.setProperty( "drools.resource.scanner.interval",
"30" ); // set the disk scanning interval to 30s, default is 60s
ResourceFactory.getResourceChangeScannerService().configure( sconf );
KnowledgeBase kbase = KnowledgeBaseFactory.newKnowledgeBase();
KnowledgeAgentConfiguration aconf = KnowledgeAgentFactory.newKnowledgeAgentConfiguration();
aconf.setProperty( "drools.agent.scanDirectories",
"true" ); // we want to scan directories, not just files, turning this on turns on file scanning
aconf.setProperty( "drools.agent.newInstance",
"true" ); // resource changes results in a new instance of the KnowledgeBase being built,
// this cannot currently be set to false for incremental building

KnowledgeAgent kagent = KnowledgeAgentFactory.newKnowledgeAgent( "test agent", // the name of the agent
kbase, // the KnowledgeBase to use, the Agent will also monitor any exist knowledge definitions
aconf );
kagent.applyChangeSet( ResourceFactory.newUrlResource( url ) ); // resource to the change-set xml for the resources to add
KnowledgeAgents can take a empty KnowledgeBase or a populated one. If a populated KnowledgeBase is provided, the KnowledgeAgent will iterate KnowledgeBase and subscribe to the Resource that it finds. While it is possible for the KnowledgeBuilder to build all resources found in a directory, that information is lost by the KnowledgeBuilder so those directories will not be continuously scanned. Only directories specified as part of the applyChangeSet(Resource) method are monitored.

Wednesday, March 04, 2009

A new testing tool

This is a new experimental library looking for early adopters and feedback before we include it in the main part of the project. Previously there was the Fit For Rules library which worked well, but for whatever reason fit and documents haven't proven super popular (also, it is GPL licenced).

So, based on some work from James Kerridge (Qantas) I have worked on a replacement that uses spreadsheets to set up test data:


You can run it and get pretty printed output like the following:

(and much more of course).

This is just a first cut, am looking for feedback and suggestions - the basic idea is that that is another way to capture test data (columns == scenarios).

The project is currently on github for the moment. You can read a bit more about it in the README and download it.

Most of the documentation is in the test/sample spreadsheet.

As a small curiosity, this is written entirely in scala (but when using the library people wouldn't notice as its just a java api).

Would love to hear about peoples experiences with it (michael dot neale at gmail dot com).

Spanish localisation

Thanks to Salaboy, we now have a Spanish (es_ES) localised version of Guvnor:

As mentioned previously, this can be accessed via adding ?locale=es_ES to the end of the URL, or if the browsers locale header says it wants spanish etc...

Thanks Salaboy !