Thursday, May 27, 2010

Proposal for jBPM5 Roadmap (incorporating Drools Flow)

Kris just did a blog on the jBPM5 roadmap, Drools Flow will be incorporated into jBPM5
http://kverlaen.blogspot.com/2010/05/proposal-for-jbpm5-roadmap.html
----
A while ago, we presented a request for comments on jBPM5, where we asked you, our community, to provide feedback on the vision, architecture and key features for (the next version of) the jBPM project. We would like to thank anyone who participated in the discussion and provided feedback or suggested additional requirements.

Based on these requirements we would like to present a proposal for what we would like to include in the first release of jBPM5. While we have not yet pinpointed an exact date for this release (but we will do this asap, stay tuned), we are planning to release this near the end of the year (preferably a few months before even). As many people have requested, this release will focus on delivering a solid core and simple tooling, while additional (more advanced) features will be rolled out in the subsequent releases.

The key features of this release include:
  • Native BPMN2 execution
  • Highly configurable, embeddable, lightweight process engine using a generic process engine (PVM) underneath
  • Domain-specific processes and rule / event integration
  • Independent, human tasks service (using WS-HT)
  • Eclipse and web tooling for things like process creation, deployment, management, reporting and human tasks.
  • Migration capabilities from jBPM 3 and 4
More details on the roadmap can be found here. Disclaimer: this is currently still a proposal, so the content might still change (slightly), but we're planning to put out a confirmed roadmap as soon as possible.

Feedback on the roadmap is welcome on the jbpm-dev@lists.jboss.org mailing list.

By the way, don't fear that we're only getting started now. A lot has been implemented already, building on top of jBPM4 and Drools Flow. Don't worry, we're not building from scratch. And if you'd like to help a hand, you're more than welcome to drop a note on the mailing list as well.

Drools Flow Overview and Previous Relevant Articles

InfoQ recently did an article on Activi and BPMN 2.0, so I thought it was worth collecting some of the previous releveant Drools Flow articles, including ones on BPMN 2.0. So that people can get a better idea of what's available in the OSS landscape.

Drools Flow is embeddable with small jar sizes, minimal dependencies and under the Apache Software License.

In 5.0 we had our own xml format, as BPMN2 was not ready. About a year ago we moved 5.1.0.SNAPSHOT to BPMN2 as the default.
http://blog.athico.com/2009/07/drools-flow-and-bpmn2.html

We also have commitment to other standards such as our WS-HumanTask implementation, which we have had for some time.
http://blog.athico.com/2008/09/drools-and-ws-humantask.html

Drools Flow comes fully integrated with our rules (Drools Expert) and cep (Drools Fusion) technology. This allows for built in declarative monitoring and interceptors, an important part of building both a dynamic and adaptive platform.
http://blog.athico.com/2009/11/monitoring-your-drools-flow-processes.html

We also provide easy extension points for domain specific workflows, we call these "Work Items". We provide example ones, such as google calendar integration, file listings, ftp, command line execution, email etc:
http://blog.athico.com/2009/02/drools-flow-work-items.html

Video's of Drools Flow in action:
http://blog.athico.com/2009/12/screencasts-on-some-faq.html
http://blog.athico.com/2009/08/drools-flow-videos.html

Rules and Processes share so much as declarative languages, and the Drools platform makes this a very natural fit, you can read more about things like common life cycle here:
http://people.redhat.com/kverlaen/BPM/

We are just about to release M2, where the fruits of this work can be seen, and GA should follow very shortly. Follow the blog for latest release news:
http://www.jboss.org/drools/downloads.html

Going forward Drools Flow will be incorporated into jBPM, with additional feedback from the jBPM community as part of jBPM5.
http://kverlaen.blogspot.com/2010/05/proposal-for-jbpm5-roadmap.html

Wednesday, May 26, 2010

Drools 5.1 M2 release notes

We're proud to announce the second milestone release of Drools 5.1. It includes a lot of new features, as described below. So try them out and let us know what you think.

Knowledge Agent

Incremental change-set processing

The new version of the Knowledge Agent supports newInstance = false in its configuration (incremental change-set build). When setting this property to false, the kbase instance of the agent is reused when changes in the monitored resources are detected.
Now Knowledge Agent can process monitored resources modifications in an incremental way.
Modified definitions are compiled and compared against the original version. According to definition's type, the behaves in different ways:
  • Rules: For rules, the Agent searches for modifications in its attributes, LHS and RHS.
  • Queries: queries are always replaced on kbase wether they are modified or not.
  • Other definitions: All other definitions are always replaced in kbase (like if they were modified). We expect to add better support for definition's modification detection in further versions.
The current implementation only supports the deletion of rules and functions definitions. For further information please take a look at:

Live Querries

Drools has always had query support, but the result was returned as an iterable set; this makes it hard to monitor changes over time. Live Querries have a listener attached instead of returning an iterable result set. These live querries stay open creating a view and publish change events for the contents of this view.
ViewChangedEventListener listener = new ViewChangedEventListener() {       
public void rowUpdated(Row row) {
updated.add( row.get( "$price" ) );
}

public void rowRemoved(Row row) {
removed.add( row.get( "$price" ) );
}

public void rowAdded(Row row) {
added.add( row.get( "$price" ) );
}
};

// Open the LiveQuery
LiveQuery query = ksession.openLiveQuery( "cheeses",
new Object[] { "cheddar", "stilton" },
listener );

Timers and Calendars


Rule's now suport both interval and cron based timers, which replace the now deprecated duration attribute.
timer ( int: <initial delay> <repeat interval>? )
timer ( int: 30s )
timer ( int: 30s 5m )

timer ( cron: <cron expression> )
timer ( cron:* 0/15 * * * ? )
Calendars can now controll when rules can fire. The Calendar api is modelled on Quartz :
public interface Calendar { 
boolean isTimeIncluded(long timestamp);
}
Quartz provides several good Calendar implementations, so it makes sense to leverage those, for this we we provide an adapter helper method:
Calendar QuartzHelper.quartzCalendarAdapter(org.quartz.Calendar quartzCal)
Calendars are registered with the StatefulKnowledgeSession:
ksession.getCalendars().set( "week day", weekDayCal );
Which means they can now be used in rules. They can be used in conjunction with normal rules and rules including timers.

The rule calendar attribute can have one or more calendar names.
calendars "calname"
calendars "calname1", "calname2"

Drools Spring integration

Drools Flow easy configuration

We keep improving the integration with Spring Framework. We added support for configuring a Drools Flow JPA session for using it with a local transaction manager.
(Now if you don't need to configure variable persisters the jpaSessionServiceFactory is only one line! :))
More info and usage examples in: http://blog.bauna.com.ar/index.php/2010/05/drools-flow-configuration-using-spring/

Drools Camel Integration:

The Apache Camel integration let us interact with a Drools stateless or stateful session. The interaction is achieved using a pipeline that transforms the input command (right now an XML representation of the command) into an executable one. Also Apache Camel let us implement more advanced enterprise integration patterns than a simple transformation pipeline.
So, you can use any of the out-of-the-box Camel Components as a Drools commands producer, which allow to create more powerful services interactions. At this moment, we provide out-of-the-box marshallers for XStream & JAXB.

If you want to read more about how to configure and use the Drools Camel endpoint take a look at the following link:
http://lucazamador.wordpress.com/2010/05/28/drools-apache-camel-integration/

Drools Execution Server:

A new version of Drools Server is included in this release. This version add the support for Stateful Knowledge Sessions. Internally, for this new version, we use drools-camel, drools-spring and drools-grid to provide the following list of new features:
  • Stateful/Stateless Knowledge session support
  • RestWS & SOAP services to interact with the sessions
  • Spring configuration
  • JAXB/XStream XML commands support (using the out-of-the-box provided marshallers in drools-camel)
You can read more about this in the following blog post: http://lucazamador.wordpress.com/2010/05/27/drools-server-spring-configuration/

Extensive BPMN 2.0 support

As we already announced earlier, the Drools team has decided to support the use of the upcoming BPMN 2.0 specification for specifying business processes using XML. This milestone includes a significant extension of the BPMN2 parser to support more of the BPMN2 features using Drools Flow. More specifically:
  • more extensive event support: much more combinations of event types (start, intermediate and end) and event triggers (including for example error, escalation, timer, conditional and signal events), have been included, as well as (interrupting and non-interrupting) boundary events
  • sub-process parameters
  • diverging inclusive gateway
  • etc.
BPMN2 processes have also been integrated in the entire Drools tool chain, to support the entire life cycle of the business process. This includes
  • the ability to use BPMN2 processes in combination with our Eclipse tooling
  • Guvnor as process repository
  • web-based management using the BPM console
  • auditing and debugging
  • domain-specific processes
  • etc.

As a result, Drools Flow is not only the first open-source process engine that supports such a significant set of BPMN2 constructs natively, our knowledge-oriented approach also allows you to easily combine your BPMN2 processes with business rules and complex event processing, all using the same APIs and tools.

Installation script

The Drools build now contains an install folder that simplifies installing the Eclipse plugin, Guvnor and the gwt-console. It creates and copies the necessary jars and wars and deploys them to the JBoss AS. It also includes a simple evaluation process example you can use to test your setup. For more info, take a look at the readme in the install folder.

JMX Monitoring improvements

The JMX monitoring framework was improved to include process related statistics. JOPR plugin was also updated to show such data.

Session Inspection and Reporting framework

A new API based framework for runtime session inspection and reporting was introduced, allowing for better data gathering during debugging or profiling of the application. This inspection framework will become the basis of the tooling features to help providing more detailed information about the contents of each session.

To inspect a session, one can use the following API calls:
        StatefulKnowledgeSession ksession = ...

// ... insert facts, fire rules, etc

SessionInspector inspector = new SessionInspector( ksession );
StatefulKnowledgeSessionInfo info = inspector.getSessionInfo();
The StatefulKnowledgeSessionInfo instance will contain a lot of relevant data gathered during the analysis of the session. A simple example report template is provided and can be generated with the following API call:
        String report = SessionReporter.generateReport( "simple", info, null );

Guvnor

Updated to GWT 2.0

GWT was updated, making Guvnor faster.

Build in selector

The built in selector allows user to choose what assets to build according to:
1. Status (eg, Dev, QA etc)
2. Category
3. Metadata

Single asset verification

It is possible to verify just the asset you are working on ( ruleflow, rule, decision table ). Verification finds issues like conflicting restrictions in a rule or redundant rows in decision tables.

Global area

Assets stored in Global area can be shared to all packages.

Diff check between snapshots

Lists the changes between two snapshots.

Viewing multiple assets in one tab

Makes it possible to open more than one asset into one view. All the assets can be saved and edited as a group.

Pattern order

Guided Editor supports Pattern reordering in LHS and RHS sections. New Patterns can be inserted in any order (and not always at the end). For further information: http://ilesteban.wordpress.com/2010/05/28/guvnor-patternaction-ordering-in-guided-editor/

From/Collect/Accumulate support

Guided Editor has basic support for From, Accumulate and Collect Patterns. You can add any of these structures as regular Patterns.
New expression builder component was created to add support for nested method calls of a variable.
The following link contains a more detailed explanation about these improvements:
http://ilesteban.wordpress.com/2010/05/28/guvnor-guided-editor-suuport-for-fromcollectaccumulate-elements/

Templates

There is a new knowledge you can add to the base. You can create rules based on a data grid attached to a rule template. This will allow the user to don't write repetitive rules who only changes the value of the facts.
A screencast showing Templates in action can be found here: http://locademiaz.wordpress.com/2010/05/28/new-guvnor-feature-rules-templates/

Working Sets

Working Sets are a mean for grouping Facts and then defining constraints on them. You can create groups of Facts and only those Facts will be visible when authoring rules using the Guided Editor.

Fact Constraints

Once you have grouped facts in a Working Set you can define restrictions on those facts' fields. The restrictions that you define will be used to validate what you can write on rules.
For further information you can read:

Wednesday, May 19, 2010

Live Querries

Drools has always had query support, but the result was returned as an iterable set; this makes it hard to monitor changes over time.

I did a little hacking this weekend and we have now complimented this with Live Querries, which has a listener attached instead of returning an iterable result set. These live querries stay open creating a view and publish change events for the contents of this view.

So now you can execute your query, with parameters and listen to changes in the resulting view.

There are many applications for this, but already I'm thinking about Glazed List integration, so it can be used to provide advanced filtering for table displays.

Hopefully this unit test is self explanatory, for those interested, the main part is the creating of the listener and the opening of the live query.


String str = "";
str += "package org.drools.test \n";
str += "import org.drools.Cheese \n";
str += "query cheeses(String $type1, String $type2) \n";
str += " stilton : Cheese(type == $type1, $price : price) \n";
str += " cheddar : Cheese(type == $type2, price == stilton.price) \n";
str += "end\n";

KnowledgeBuilder kbuilder = KnowledgeBuilderFactory.newKnowledgeBuilder();
kbuilder.add( ResourceFactory.newByteArrayResource( str.getBytes() ),
ResourceType.DRL );

if ( kbuilder.hasErrors() ) {
fail( kbuilder.getErrors().toString() );
}

KnowledgeBase kbase = KnowledgeBaseFactory.newKnowledgeBase();
kbase.addKnowledgePackages( kbuilder.getKnowledgePackages() );

StatefulKnowledgeSession ksession = kbase.newStatefulKnowledgeSession();
Cheese stilton1 = new Cheese( "stilton", 1 );
Cheese cheddar1 = new Cheese( "cheddar", 1 );
Cheese stilton2 = new Cheese( "stilton", 2 );
Cheese cheddar2 = new Cheese( "cheddar", 2 );
Cheese stilton3 = new Cheese( "stilton", 3 );
Cheese cheddar3 = new Cheese( "cheddar", 3 );

org.drools.runtime.rule.FactHandle s1Fh = ksession.insert( stilton1 );
ksession.insert( stilton2 );
ksession.insert( stilton3 );
ksession.insert( cheddar1 );
ksession.insert( cheddar2 );
org.drools.runtime.rule.FactHandle c3Fh = ksession.insert( cheddar3 );

final List updated = new ArrayList();
final List removed = new ArrayList();
final List added = new ArrayList();

ViewChangedEventListener listener = new ViewChangedEventListener() {
public void rowUpdated(Row row) {
updated.add( row.get( "$price" ) );
}

public void rowRemoved(Row row) {
removed.add( row.get( "$price" ) );
}

public void rowAdded(Row row) {
added.add( row.get( "$price" ) );
}
};

// Open the LiveQuery
LiveQuery query = ksession.openLiveQuery( "cheeses",
new Object[] { "cheddar", "stilton" },
listener );

// Assert that on opening we have three rows added
assertEquals( 3, added.size() );
assertEquals( 0, removed.size() );
assertEquals( 0, updated.size() );

// And that we have correct values from those rows
assertEquals( 1, added.get( 0 ) );
assertEquals( 2, added.get( 1 ) );
assertEquals( 3, added.get( 2 ) );

// Do an update that causes a match to become untrue, thus triggering a removed
cheddar3.setPrice( 4 );
ksession.update( c3Fh, cheddar3 );

assertEquals( 3, added.size() );
assertEquals( 1, removed.size() );
assertEquals( 0, updated.size() );

assertEquals( 4, removed.get( 0 ) );

// Now make that partial true again, and thus another added
cheddar3.setPrice( 3 );
ksession.update( c3Fh, cheddar3 );


assertEquals( 4, added.size() );
assertEquals( 1, removed.size() );
assertEquals( 0, updated.size() );

assertEquals( 3, added.get( 3 ) );

// check a standard update
cheddar3.setOldPrice( 0 );
ksession.update( c3Fh, cheddar3 );

assertEquals( 4, added.size() );
assertEquals( 1, removed.size() );
assertEquals( 1, updated.size() );

assertEquals( 3, updated.get( 0 ) );

// Check a standard retract
ksession.retract( s1Fh );

assertEquals( 4, added.size() );
assertEquals( 2, removed.size() );
assertEquals( 1, updated.size() );

assertEquals( 1, removed.get( 1 ) );

// Close the query, we should get removed events for each row
query.close();

assertEquals( 4, added.size() );
assertEquals( 4, removed.size() );
assertEquals( 1, updated.size() );

assertEquals( 2, removed.get( 2 ) );
assertEquals( 3, removed.get( 3 ) );

// Check that updates no longer have any impact.
ksession.update( c3Fh, cheddar3 );
assertEquals( 4, added.size() );
assertEquals( 4, removed.size() );
assertEquals( 1, updated.size() );

Tuesday, May 18, 2010

Seam 3 Drools Integration - First Review

As we are nearing an Alpha release of the Seam 3 Drools module I wanted to give an overview of what we are working on. Drools integration in Seam 3 is a portable extensions to CDI/Weld. Some of the main focus points of the integration are:
  • Strong integration will all features of Drools 5
  • Simple yet powerful configuration and ease of use
  • Enhanced integration of Drools with rich Internet applications based upon the Java EE environment
If you are not yet familiar with Seam 3 I would definitely encourage you to get on the bandwagon. Great places to get information are seamframework.org as well as Pete Muir's blog.

So let's get started!

The starting point of the integration is the XML configuration in beans.xml. Out-of-the box Seam 3 provides some "most commonly used" configuration templates, for example:
Using the CEPPseudoClockRuleResources configuration template allows you to only have to specify your rule resources and not have to deal with setting Drools-specific configurations needed, as in this case would be setting drools.eventProcessingMode to "stream" and drools.clockType to "pseudo". Of course users can have full control of the config as shown here:
Once you have set up the configuration, you are ready to go! Seam 2 users will be happy that there is no need to create separate configuration blocks for the RuleBase, and each of the creates stateful/stateles sessions, etc; in Seam 3 once you define your rule resources you just "use stuff"

So let's say in our ceptest.drl we define a rule where facts can come from two different streams, namely "FireDetection" and "SprinklerDetection". Right off the bat we can inject those into our beans:


@Inject @CEPPseudoClockConfig @EntryPoint("FireDetection") WorkingMemoryEntryPoint fds;
@Inject @CEPPseudoClockConfig @EntryPoint("SprinklerDetection") WorkingMemoryEntryPoint sds;

and use them in methods, for example:
public void fireDetected() { fds.insert(new FireDetected()); }

So in case of a fire let's say we need to produce a quick evacuation report and want to see how many of our employees are/were affected by it:

@Inject void EvacuationReport(@CEPPseudoClockConfig @Query("employees affected") QueryResults qr) { ... }

Here what we are doing is passing our QueryResults as parameter to the constructor of the EvacuationReport class, and there is no additional configuration required! As long as we have the "employeed affected" query in our rules we can inject it's QueryResults.

But that's not all, let say that our Sprinkler system is controller some service which produces batch execution XML, for example:

We want to be able to create an instance of this SprinklerSystem for our application when it's needed and we can do with for example:


@Produces @FirstFloor
public Sprinklers getSprinklerSystem(@CEPPseudoClockConfig @Stateful ExecutionResults er) {
return (Sprinklers) er.getValue("sprinklerSystem");
}


Now let's say that our evacuation procedures are controlled by a Rule Flow process. We can control this process from our application with a set of new annotations, for example:


@StartProcess(process="buildingEvac", fire=true)
@CEPPseudoClockConfig
public void evacuateBuilding() {
...
}

@SignalEvent(type="evacuateFloor")
@CEPPseudoClockConfig
public String evacuateFirstFloor() {
return "first";
}

@AbortProcess(process="buildingEvac")
@CEPPseudoClockConfig
public void falseAlarm() {
...
}


That's it for now. I hope to have sparked some questions and interest. Please note that the code shown here is not set in stone yet and is subject to some changes as we are progressing to the Alpha release.

Monday, May 17, 2010

Rules Fest 2010 website launched

Rules Fest 2010, previously October Rules Festival, has just launched it's new website. The Drools team will hopefully be there, so cya there.

Sparkling Logic

Carole-Ann and Carlos, who recently left FICO, have just setup their new venture - Sparkling Logic. Carlos and Carole-Ann have been actively involved in the October Rules Fest from the start, now called Rules Fest, and I have always enjoyed meeting them there, and I think they are just finally starting to get use to me :) so I wish them the best of luck in their new venture.

Saturday, May 15, 2010

Prova 3.0 Released

The Prova team have just release 3.0:
http://www.prova.ws/

Tuesday, May 11, 2010

Automated nurse rostering based on hard and soft constraints with Drools Planner

The optimal solution for a planning problem is the solution with the highest score. But how do we calculate and compare scores?

In a previous blog, I have shown that there's no easy way to find the optimal solution in for example the bin packaging use case. Today I 'll use the nurse rostering use case to explain scoring based on hard and soft constraints.

The use case

In the nurse rostering rostering use case, we assign nurses to work shifts in a hospital. We want to increase quality of service, minimize staff size and maximize staff contentment.


In the example above, on the Monday morning we need 1 maternity nurse. Both solutions assign nurse C to that shift. As you can see, the solution on the right is better, because it breaks less constraints.

Each solution has a score and Drools Planner will look for the solution with the highest score. In some use cases the score is a single integer (SimpleScore), but in other use cases, such as this one, there are both hard and soft constraints (HardAndSoftScore).

Hard constraints

Hard constraints need to be fulfilled. We always look for the solution with the least hard constraints broken.

The example above shows 2 hard constraints. The solution is feasible when none of the hard constraints are broken. The hospital can work with any feasible solution, but still they prefer some feasible solutions over others...

Soft constraints

Soft constraints should be fulfilled as much as possible. Of all the feasible solutions, we look for the solution with the least soft constraints broken.

The example above shows some of the soft constraints of the nurse rostering use case. But if you talk to a business analyst, you 'll discover that there are many more and you need something which makes adding extra constraints easy. Something like Drools Planner.

Drools Language Enhancements (Drools 6) updates

Davide has been working on the new parser for Drools 6, he's updated the language guide with some more ideas. Don't worry, core developers are still focusing on 5.1. The full language guide, aimed to provide the Drools 6.0 drl syntax can be found here:
http://community.jboss.org/wiki/DroolsLanguageEnhancements

The parts Davide has updated are:
Pipes - Pass through Filters
Unit support
Rising / Falling edges


Pipes - Pass through Filters

While accumulate performs map and fold operations it returns a derived result. Pipes allow us to filter a set of Tuple chains, that represent the actual join partial match results. In 5.0.x we had "over" as a temporal pass through filter, Person() over win:time(30s). We feel this concept should be more generalised to allow any pluggable filters, and for those filters to work on single Patterns of a group of patterns:
Person() | win:time(30s)
These pipes can be combined, so we only allow the Persons for the last 10 minuts to propagate forward (retracting anyone who existed more than 10 mins ago) but we throttle the network updates to every 30s:
Person() | win:time(10m) | throttle(30s)
This is important for combination with say accumulates, where we don't want every single change to be aggregated and the result propagated:
acc( Bus( $t: takings) | win:time(1h) | throttle(5m),
$avgTakings : avg( $t ) )
So th above calculates the average takings for the last hour, continously, but it only updates the results every 5 minutes. Throttle is just currently a proposed filter name, we might come up with something different for that behaviour later.

Pipes don't just work on single Patterns, they can work on multiple patterns. So for instance we could provide a "unique" filter that remove ambiguous cross product joins; which Charles Forgy mentioned at ORF09. In the following example if we instead insert 3 As we would get 1 resulting match, not 9.
(A() A() A()) | unique
Unit support

Groovy added unit support, based around leveraging JScience and JSR275. http://groovy.dzone.com/news/domain-specific-language-unit-. The dot notation is a bit too ambigous for Drools, but we already use # to cast patterns, so we can do the same for units.
3#km + 5#m
We can cast existing units to other units. So we can declare something as 3km, but have it returned as feet.
3#km#feet
If a unit is combined with a normal literal, then it's just a operator on that literal value, for instance the following is 6k, it is not executed in the same way as 3km * 2km would do.
3#km * 2
# can be used with methods and functions. If the method returns a literal, then it's the same as saying 3km. If the method returns a unit, then it's like a caste:
returnInt()#km
returnKm()#feet
This works for strings too, and can be used to provide date/time formatting:
"2012 12 04"#myDateFormat
Rising / Falling edges

Grindworks supports the idea of executing actions on the rising or falling edges of a rule. While we could do this on the actions too, we think initially this would be better on the LHS, as a special conditinal element.
when
rising Person( age == 30 )

when
falling Person( age == 30 )
Clealry rising is the default behaviour for a pattern. Whether we allow it's inclusion, for readability intent, or only support falling, is to be decided.

We could combine this with a branch
when
...
branch( rising Person(....),
[b1] falling Person(....)
This also has some ramifications for Logical Closures, as it solves some of the same problems.

Wednesday, May 05, 2010

Drools @ BeJUG 2010: slides and droolsphone example

Here are the slides from the Drools presentation of Kris and me at BeJUG:



The Drools Expert part shows Droolsphone, an example which compares Java to SQL to Drools. It can be downloaded here.