Tuesday, January 27, 2009

Drools Flow and OSWorkflow Migration

Hi, my name is Miguel Fossati, I am a developer at Synapsis Argentina, and I am working with Chief Enterprise Architect Diego Naya (author of Using OSWorkflow in your Application"), Maurticio Salatino (A.K.A. Salaboy) and the Drools team in improving Drools Flow. We work together on projects for Argentina's biggest healthcare provider, who are big OSWorkflow users. However with Drools offering a much more powerful and complete framework, that integrates rules, processes and event processing there is now a desire to move to this as the standard. As part of this effort we need a migration path for al the legacy OSWorkflow applications.

OSWorkflow is an open source workflow tool developed by OpenSymphony, widely used in open source, commercial systems and third party tools (like Jira, and others).

Since Drools Flow, part of up coming Drools 5 release, supports a pluggable node framework for process definitions it's possible for us to implement other execution models. We are developing a tool to migrate OSWorkFlows processes into Drools Flow process format. This now makes it possible to migrate OSWorkflow processes and definitions to Drools. We expect this tool to be very useful for developers and users that want to migrate their existing OSWorkflow based systems, to take advantage of the Drools Flow process capabilities.

We have extended the underlying nodes framework adding a Step node to fully support OSWorkflow's step concept. In OSWorkflow a Step node is a kind of a wait state that have some actions to take and move to another step. It supports conditional actions that implements OSWorkflow's Condition contract, and execution of functions via the FunctionProvider interface. It also supports Split and Join nodes using the native RuleFlow Split and Join nodes.

Lets see an example, starting with this simple OSWorkflow definition:
<workflow>
<initial-actions>
<action id="1" name="Start Workflow">
<results>
<unconditional-result old-status="Finished"
status="Queued" step="1" />
</results>
</action>
</initial-actions>
<steps>
<step id="1" name="First Draft">
<actions>
<action id="2" name="Start First Draft">
<restrict-to>
<conditions>
<condition type="class">
<arg name="class.name">
com.opensymphony.workflow.util.StatusCondition
</arg>
<arg name="status">Queued</arg>
</condition>
</conditions>
</restrict-to>
<pre-functions>
<function type="class">
<arg name="class.name">
com.opensymphony.workflow.util.Caller
</arg>
</function>
<function type="beanshell">
<arg name="script">
System.out.println("Before executing actionid 2");
</arg>
</function>
</pre-functions>
<results>
<unconditional-result old-status="Finished"
status="Underway" step="1" owner="${caller}" />
</results>
</action>
<action id="3" name="Finish First Draft">
<restrict-to>
<conditions type="AND">
<condition type="class">
<arg name="class.name">
com.opensymphony.workflow.util.StatusCondition
</arg>
<arg name="status">Underway</arg>
</condition>
<condition type="class">
<arg name="class.name">
com.opensymphony.workflow.util.AllowOwnerOnlyCondition
</arg>
</condition>
</conditions>
</restrict-to>
<results>
<unconditional-result old-status="Finished"
status="Queued" step="2" />
</results>
</action>
</actions>
</step>
<step id="2" name="finished" />
</steps>
</workflow>
Our Drools Flow definition matching the above would be like this:
<process xmlns="http://drools.org/drools-4.0/osworkflow"
xmlns:xs="http://www.w3.org/2001/XMLSchema-instance"
xs:schemaLocation="http://drools.org/drools-4.0/osworkflow drools-osworkflow-4.0.xsd"
type="OSWorkflow" name="simple" id="simple" package-name="org.drools.osworkflow" >

<header>
<initial-actions>
<action id="1" name="Start Workflow">
<results>
<unconditional-result old-status="Finished"
status="Queued" step="1" />
</results>
</action>
</initial-actions>
</header>

<nodes>
<step id="1" name="First Draft" >
<action id="2" name="Start First Draft">
<restrict-to>
<conditions>
<condition type="class">
<arg name="class.name">
com.opensymphony.workflow.util.StatusCondition
</arg>
<arg name="status">Queued</arg>
</condition>
</conditions>
</restrict-to>
<pre-functions>
<function type="class">
<arg name="class.name">
com.opensymphony.workflow.util.Caller
</arg>
</function>
<function type="beanshell">
<arg name="script"><![CDATA[
System.out.println("Before executing actionid 2");
]]></arg>
</function>
</pre-functions>
<results>
<unconditional-result old-status="Finished" status="Underway" step="1" owner="${caller}"/>
</results>
</action>
<action id="3" name="Finish First Draft">
<restrict-to>
<conditions type="AND">
<condition type="class">
<arg name="class.name">
com.opensymphony.workflow.util.StatusCondition
</arg>
<arg name="status">Underway</arg>
</condition>
<condition type="class">
<arg name="class.name">
com.opensymphony.workflow.util.AllowOwnerOnlyCondition
</arg>
</condition>
</conditions>
</restrict-to>
<results>
<unconditional-result old-status="Finished" status="Queued" step="2"/>
</results>
</action>
</step>
<step id="2" name="finished" >
</step>
</nodes>
<connections>
<connection from="1" fromType="3" to="2" toType="Queued" />
</connections>
</process>
Persistence is currently managed by a mix of JPA and a Serialization process that store the status of the process inside a relational schema that is updated every time the process reach a wait state, taking out of memory the current processInstance until someone else interacts with the process. This persistence strategy is beeing extended but at the moment you can use it with the SingleCommandSessionService.

To run our process, we first build our knowledge base, as any native Drools flow:
            // create a builder
PackageBuilder builder = new PackageBuilder();
// load the process
Reader source = new InputStreamReader(this.getClass().getResourceAsStream(resourceName));
builder.addProcessFromXml(source);
// create the knowledge base
Package pkg = builder.getPackage();
RuleBase ruleBase = RuleBaseFactory.newRuleBase();
ruleBase.addPackage(pkg);
As you can see here we load our process in the working memory, exactly in the same way that we load a RuleFlow process.

When we want to interact with the process we must use the SingleSessionCommandService to execute commands that influence our process to jump from one step to another.
This SingleSessionCommandService is configured with a JPA session that store the status of process when it reachs a wait state.

Here we start an instance of our process using the ID of the process, in this case "simple", this will execute the OSWorkflow initial actions and point the execution of the process to the first step node called "First Draft".
        SingleSessionCommandService service = new SingleSessionCommandService(ruleBase);
StartProcessCommand startProcessCommand = new StartProcessCommand();
startProcessCommand.setProcessId("simple");
ProcessInstance processInstance = (ProcessInstance) service.execute(startProcessCommand);
System.out.println("Started process instance " + processInstance.getId());

service = new SingleSessionCommandService(ruleBase);
GetProcessInstanceCommand getProcessInstanceCommand = new GetProcessInstanceCommand();
getProcessInstanceCommand.setProcessInstanceId(processInstance.getId());
processInstance = (ProcessInstance) service.execute(getProcessInstanceCommand);
System.out.println("Now working with processInstance " + processInstance.getId());
Now we want to execute an action to move to another step node in the workflow, so we use the command DoActionCommand to tell the process that we want to move to the next node. In this stage, the process will be retrieved from the database and a deserialization process of the processInstance status will take place. After this deserialization the action will be executed and the execution of the process will reach the next step. When the execution enter in the new step the process waits until someone execute another action, so the process will be persisted again.
        service = new SingleSessionCommandService(ruleBase);
DoActionCommand doActionCmd = new DoActionCommand();
doActionCmd.setProcessInstanceId(processInstance.getId());
doActionCmd.setActionId(2); //Action to be executed at current step
service.execute(doActionCmd);
Please, feel free to test this tool, and provide feedback that will help us improve it. This work is currently in a branch and will be merged into Drools trunk this week, we will announce in the blog once that merge is complete and ready for testing.

Sponsoring missing tooling features

If there are features missing in our Eclipse tooling or Guvnor tooling and you would like to sponsor the development of those to accelerate their adoption in the project you can speak to NexB who have done excellent work already on sponsored Eclipse development. So please feel free to contact Philippe Ombr├ędanne if you would like to work with them, or you can contact me (mproctor at redhat d0t com) to liase between yourselves and Nexb.

Drools 5 - A Holistic approach to problem modelling using Rules, Workflow and Event Processing

I've always stated that end business users struggle understanding the differences between rules and processes, and more recently rules and event processing. For them they have this problem in their mind and they just want to model it using some software. The traditional way of using two vendor offerings forces the business user to work with a process oriented or rules oriented approach which just gets in the way, often with great confusion over which tool they should be using to model which bit.

PegaSystems and Microsoft have done a great job of showing that the two can be combined and a behavioural modelling approach can be used. This allows the business user to work more naturally where the full range of approaches is available to them, without the tools getting in the way. From being process oriented to rule oriented or shades of grey in the middle - whatever suites the problem being modelled at that time. We are taking this one step further and also adding event processing with Drools Fusion creating a more holistic approach to software development. Where the term holistic is used for emphasizing the importance of the whole and the interdependence of its parts.

So it was nice to receive an email on the public mailing list today, from Ekkehard Gentz (http://www.gentz-software.de/) that said the same thing about the confusion over two separate products for rules and processes and his delight in finding Drools 5, here is a quote from that email:

"the problem for business people to understand the differences between business processes and rules, because from the point-of-view of business use-cases they are integrated. why are there two tools / frameworks to manage the same thing?
....
Now I read the documentation of Drools 5 and noticed that all is availabe in ONE product: Drools 5. Thats really great news - I hope I understood all well and I'll give it a try to do it with Drools only. This gives me a better feeling than my previous decision. "

Here is the link to an archive of the original email in full:
http://www.mail-archive.com/rules-users@lists.jboss.org/msg07530.html

Saturday, January 24, 2009

LGPL QT

With QT now going LGPL, I think it is now definitively the best way to do cross platform GUI development. It would also seem that QT would be the "vendor neutral" way for IBM and Sun to move forward on a single toolkit, to avoid the Swing vs SWT debate which neither side will back down on. Now would seem like a good time for Java3 on the JVM. A new cleaned up javaish language, with cleaned up api that is consistent throughout and standardising on QT as the GUI toolkit.

Friday, January 23, 2009

Drools Community Clinic 4th of February at 5pm GMT

We will host an informal Drools Community Clinic on the 4th of February at 5pm GMT.

The idea here isn't to do one big presentation, but really more of an interactive Q&A where we will use the desktop to assist in some explanations talking through code or examples. We can mix it up with a bit of general fun AI/Drools geek chat, depending on what people want to talk about. As much as it works, we will try and keep an open floor.

Please post potential questions, talking points or just general ideas as comments on this blog posting.

We will publish connection details nearer the time.

Wednesday, January 21, 2009

Drools 5.0 M5 New and Noteworthy Release Summary

Previous New and Noteworth release summary notes:
M1
M2
M3/M4

Drools Guvnor

Most of the work has been bug fixes and small visual improvements.
  • It is possible to add objects to global collections.
  • Jamming asset lists are fixed. They were causing problems with some locales and date formats.
  • To allow faster rule opening, loading of some widgets have been deferred. For example the images show how assets meta data can be reviewed. Image on the right shows the default view and the left one shows detailed view.
  • There is now a "About" selection under Admin panel to show the Guvnor version and svn revision. This will help with bug reporting and support.


Drools Expert

But Fixes
lots of bug fixes, see JIRA for M5 for details.

Pipeline
The Drools pipeline helps with the automation of getting information into and out of Drools, especially when using services, such as JMS, and non pojo data sources. Transformers for Smooks, JAXB, Xstream and Jxls are povided. Smooks is an ETL tooling and can work with a variety of data sources, JAXB is a Java standard aimed at working with XSDs, while XStream is a simple and fast xml serialisation framework and finally Jxls allows for loading of pojos from an excel decision table. See the Javadocs PipelineFactory for documentation. Below shows part of a unit test which illustrates part of the JmsMessenger in action:
// as this is a service, it's more likely the results will be logged or sent as a return message
Action resultHandlerStage = PipelineFactory.newExecuteResultHandler();

// Insert the transformed object into the session associated with the PipelineContext
KnowledgeRuntimeCommand insertStage = PipelineFactory.newStatefulKnowledgeSessionInsert();
insertStage.setReceiver( resultHandlerStage );

// Create the transformer instance and create the Transformer stage, where we are going from Xml to Pojo. Jaxb needs an array of the available classes
JAXBContext jaxbCtx = KnowledgeBuilderHelper.newJAXBContext( classNames,
kbase );
Unmarshaller unmarshaller = jaxbCtx.createUnmarshaller();
Transformer transformer = PipelineFactory.newJaxbFromXmlTransformer( unmarshaller );
transformer.setReceiver( insertStage );

// payloads for JMS arrive in a Message wrapper, we need to unwrap this object
Action unwrapObjectStage = PipelineFactory.newJmsUnwrapMessageObject();
unwrapObjectStage.setReceiver( transformer );

// Create the start adapter Pipeline for StatefulKnowledgeSessions
Pipeline pipeline = PipelineFactory.newStatefulKnowledgeSessionPipeline( ksession );
pipeline.setReceiver( unwrapObjectStage );

// Services, like JmsMessenger take a ResultHandlerFactory implementation, this is because a result handler must be created for each incoming message.
ResultHandleFactoryImpl factory = new ResultHandleFactoryImpl();
Service messenger = PipelineFactory.newJmsMessenger( pipeline,
props,
destinationName,
factory );
messenger.start();

Drools Flow

The biggest improvement by far is improvement of the documentation. We're not there yet, but we've added sections on getting started, human task management, persistence, etc. The latest Flow documentation can be found here:

https://hudson.jboss.org/hudson/job/drools/lastSuccessfulBuild/artifact/trunk/target/docs/index.html

Some features have also been extended, including:
  • The history log - that keeps the history of all executed process instances in a database - has been extended so it is now caable of storing more detailed information for one specfic process instance. It is now possible to find out exactly which nodes were triggered during the execution of the process instance.

  • A new type of join has been added, one that will wait until n of its m incoming connections have been completed. This n could either be hardcoded in the process or based on the value of a variable in the process.

  • Improvements have been made to make persistence easier to configure. The persistence approach is based on a command service that makes sure that all the client invocations are executed inside a transaction and that the state is stored in the database after successful execution of the command. While this was already possible in M4 using the commands directly, we have extended this so that people can simply use the normal StatefulKnowledgeSession interface but simply can configure the persistence using configuration files. For more details, check out the chapter on persistence in the Drools Flow documentation.

Eclipse IDE

Some small bug fixes were added to make the IDE more stable.

Drools Fusion

Support to event expiration policy
added the ability to define a per-type event expiration policy. In the example bellow, the StockTick events will expire 10 minutes after they enter the system:
declare StockTick @role( event ) @expires( 10m ) end

Support to temporal operations over arbitrary dates.
added the ability for point-in-time operators (before, after and coincides) to be used with any arbitrary date field:
rule "Allow access"
when

WorkHours( $s : start, $e : end )

LogIn( time after $s, time before $e )
then
// allow access
end


Bug Fixes
lots of bug fixes, see JIRA for M5 for details.

Drools Pipeline for Smooks, JAXB/XSD, jXLS (Excel) and XStream

The Drools pipeline helps with the automation of getting information into and out of Drools, especially when using services, such as JMS, and non pojo data sources. Transformers for Smooks, JAXB, Xstream and Jxls are povided. Smooks is an ETL tooling and can work with a variety of data sources, JAXB is a Java standard aimed at working with XSDs, while XStream is a simple and fast xml serialisation framework and finally Jxls allows for loading of pojos from an excel decision table.

Pipeline is not meant as a replacement for products like the more powerful Camel, but is aimed as a complimentary framework that ultimately can be integrated into more powerful pipeline frameworks. Instead it is a simple framework aimed at the specific Drools use cases.

In Drools a pipeline is a series of stages that operate on and propagate a given payload. Typically this starts with a Pipeline instance which is responsible for taking the payload, creating a PipelineContext for it and propagating that to the first Receiver stage. Two types of Pipelines are provided, both requiring a different PipelineContexts. StatefulKnowledgeSessionPipeline and StatelessKnowledgeSessionPipeline. Notice that both factory methods take the relevant session as an argument.
Pipeline pipeline = PipelineFactory.newStatefulKnowledgeSessionPipeline( ksession );
pipeline.setReceiver( receiver );
A pipeline is then made up of a chain of Stages that can implement both the Emitter and the Receiver interfaces. The Emitter interface means the stage can propagate a payload and the Receiver interface means it can receive a payload. This is why the Pipeline interface only implements Emitter and Stage and not Receiver, as it is the first instance in the chain. The Stage interface allows a custom exception handler to be set on the stage.
Transformer transformer = PipelineFactory.newXStreamFromXmlTransformer( xstream );
transformer.setStageExceptionHandler( new StageExceptionHandler() { .... } );
The Transformer interface above extends both Stage, Emitter and Receiver, other than providing those interface methods as a single type, it's other role is that of a marker interface that indicates the role of the instance that implements it. We have several other marker interfaces such as Expression and Action, both of which also extend Stage, Emitter and Receiver. One of the stages should be responsible for setting a result value on the PipelineContext. It is the role of the ResultHandler interface, that the user implements that is responsible for executing on these results or simply setting them an object that the user can retrieve them from.
ResultHandler resultHandler = new ResultHandlerImpl();
pipeline.insert( factHandle, resultHandler );
System.out.println( resultHandler );
...
public class ResultHandlerImpl implements ResultHandler {
Object result;

public void handleResult(Object result) {
this.result = result;
}

public Object getResult() {
return this.result;
}
}
while the above example shows a simple handler that simply assigns the result to a field that the user can access, it could do more complex work
like sending the object as a message.

Pipeline is provides an adapter to insert the payload and internally create the correct PipelineContext. Two types of Pipelines are provided, both requiring a different PipelineContext. StatefulKnowledgeSessionPipeline and StatelessKnowledgeSessionPipeline. Pipeline itself implements both Stage and Emitter, this means it's a Stage in a pipeline and emits the payload to a receiver. It does not implement Receiver itself, as it the start adapter for the pipeline. PipelineFactory provides methods to create both of the two Pipeline. StatefulKnowledgeSessionPipeline is constructed as below, with the receiver set

In general it easier to construct the pipelines in reverse, for example the following one handles loading xml data from disk, transforming it with xstream and then inserting the object:
// Make the results, in this case the FactHandles, available to the user
Action executeResultHandler = PipelineFactory.newExecuteResultHandler();

// Insert the transformed object into the session associated with the PipelineContext
KnowledgeRuntimeCommand insertStage = PipelineFactory.newStatefulKnowledgeSessionInsert();
insertStage.setReceiver( executeResultHandler );

// Create the transformer instance and create the Transformer stage, where we are going from Xml to Pojo.
XStream xstream = new XStream();
Transformer transformer = PipelineFactory.newXStreamFromXmlTransformer( xstream );
transformer.setReceiver( insertStage );

// Create the start adapter Pipeline for StatefulKnowledgeSessions
Pipeline pipeline = PipelineFactory.newStatefulKnowledgeSessionPipeline( ksession );
pipeline.setReceiver( transformer );

// Instantiate a simple result handler and load and insert the XML
ResultHandlerImpl resultHandler = new ResultHandlerImpl();
pipeline.insert( ResourceFactory.newClassPathResource( "path/facts.xml", getClass() ),
resultHandler );
See StatefullKnowledgeSessionPipeline, StatelessKnowledgeSessionPipeline for more specific information and capabilities on these pipelines.

While the above example is for loading a resource from disk it is also possible to work from a running messaging service. Drools currently provides a single Service for JMS, called JmsMessenger. Support for other Services will be added later. Below shows part of a unit test which illustrates part of the JmsMessenger in action:
// as this is a service, it's more likely the results will be logged or sent as a return message
Action resultHandlerStage = PipelineFactory.newExecuteResultHandler();

// Insert the transformed object into the session associated with the PipelineContext
KnowledgeRuntimeCommand insertStage = PipelineFactory.newStatefulKnowledgeSessionInsert();
insertStage.setReceiver( resultHandlerStage );

// Create the transformer instance and create the Transformer stage, where we are going from Xml to Pojo. Jaxb needs an array of the available classes
JAXBContext jaxbCtx = KnowledgeBuilderHelper.newJAXBContext( classNames,
kbase );
Unmarshaller unmarshaller = jaxbCtx.createUnmarshaller();
Transformer transformer = PipelineFactory.newJaxbFromXmlTransformer( unmarshaller );
transformer.setReceiver( insertStage );

// payloads for JMS arrive in a Message wrapper, we need to unwrap this object
Action unwrapObjectStage = PipelineFactory.newJmsUnwrapMessageObject();
unwrapObjectStage.setReceiver( transformer );

// Create the start adapter Pipeline for StatefulKnowledgeSessions
Pipeline pipeline = PipelineFactory.newStatefulKnowledgeSessionPipeline( ksession );
pipeline.setReceiver( unwrapObjectStage );

// Services, like JmsMessenger take a ResultHandlerFactory implementation, this is because a result handler must be created for each incoming message.
ResultHandleFactoryImpl factory = new ResultHandleFactoryImpl();
Service messenger = PipelineFactory.newJmsMessenger( pipeline,
props,
destinationName,
factory );
messenger.start();

Tuesday, January 20, 2009

Drools Boot Camp 2009

After the success of the last Drools Boot Camp we've been under pressure to do another one. Location and date are not yet set, but initially I'm thinking we would continue to do this in the USA and this time wound be towards the end of May.

Last time I left it very informal, as I really wasn't sure if people would turn up, and I didn't want to formalise anything so as not to disappoint. As it turned out many people where there for the duration of the entire event. Which has given me the confidence to run something a bit more formal this year and promote it better, so our community can get more out of the core development team. So please do leave comments on this, for your preferences and ideas.

Drools Boot Camp 2009

Please do use the poll on the top right side panel if you are able to go. Also if you want me to run a poll for any other questions, on possible ideas for the event, then let me know.

Monday, January 12, 2009

Lots and lots a rules.... why not script it?

You can do a lot with rules - often way more then any one person would need. This often brings up the case of when you should or shouldn't use rules (as in drools) opposed to some simple (if large) sequence of "if statements".

This is not something you can answer in general, but what can be shown is that using drools don't have to be any worse then doing something similar as a sequence of if statements in a scripting language (or probably even java) for a reasonable number of rules.

To try this out, I wrote a script to generate 5000 randomised rules, which were of the useless and contrived form: "when p: Person(age > 40, age < 42, name =...." and so on...

So no inferencing, no higher order logic and such, but a few propositions. Really just no more then what you would do with an if statement.

For about 5000 rules I would typically get a result in less than 1ms.
Generating it as lots of if statements, it took around 5 or 6ms in jruby (just simple if statements - but the same logic) - so certainly a scripting approach is fast enough for that sort of sequential case (well its a few times slower, but if you get a response in less than 5ms, thats always pretty good I think !). What you don't get with the scripting approach is inferencing, higher order logic and much more (I am sure others can think up some other advantages).

So in summary: the features are there when you want them, but in really you don't pay a runtime speed penalty to not use them (and in some cases can get a benefit).

Friday, January 09, 2009

Irish Java Technology Conference (IJTC) 2009 presentation

I was at the Irish Java Technology Conference (IJTC) to do a Drools talk. Here are the slides I used, this was started with a 15 minute demo of showing Drools Flow working with rules, tasks and cep and a 5 minute guvnor demo:
Declarative Programming with Rules, Workflow and Event Processing - Mark Proctor