Using drools-ant to migrate/back-up existing Jackrabbit repositories

After using Guvnor for a while you may want to create a back-up of your existing JCR repository or migrate to one stored in an external database. Often this becomes a daunting task, especially if you already have a large number of existing resources in Guvnor.

Drools-ant to the rescue!

This video shows an example on how to use a new drools-ant task to migrate from the default file-based Guvnor repository to one stored in MySQL. Same task can also be used to create back-ups of your existing repository.

Using drools-ant to migrate/back-up existing Jackrabbit repositories

JBoss Asylum on jBPM5

I recently did an interview for JBoss Asylum. They bring us regular updates on what's new at JBoss by doing podcasts and inviting guests to talk about special topics.

With the release of jBPM 5.0, now was my time to get attacked by Max Rydahl Andersen, Emmanual Bernard and Michael Neale ;)

This episode had some bad luck with respect to the audio recording and took a while to get edited down to listenable quality. The news are therefore a bit out of date but the interview with Kris Verlaenen about jBPM 5 is all filled with great and interesting content.

The interview part has good audio and you can skip to the 22m33s mark to hear about jBPM 5.

We are again sorry that Emmanuel Bernard messed up the recording, but we will back soon with a new interviewee and with better sound ;)

If you got comments/feedback send it via mail or on twitter.

Get the episode from here

JBoss Asylum on jBPM5

Dynamic (non type safe) Expressions in Rules

Drools is a statically typed language, which means it has to know the types of all fields up front. Sometimes, like in Collections, this just isn't possible.

Previously we hacked this such as if you used a map or array accessor in an expression it would switch dynamic execution in MVEL for this.

We have now cleaned up this feature and made it generally available. Using type declarations you can now declare a type as un-typesafe. That then means it will not validate the expressions or their types at compile time and it will execute those expressions in dynamic (non-strict type) mvel mode.

Here is the unit test for those that want to see it in action:

    @Test
    public void testNoneTypeSafeDeclarations() {
        // same namespace
        String str = "package org.drools\n" +
            "global java.util.List list\n" +
            "declare Person\n" +
            "    @typesafe(false)\n" +
            "end\n" +
            "rule testTypeSafe\n dialect \"mvel\" when\n" +
            "   $p : Person( object.street == 's1' )\n" +
            "then\n" +
            "   list.add( $p );\n" +
            "end\n";
        
        executeTypeSafeDeclarations( str, true );   
    
        // different namespace with import
        str = "package org.drools.test\n" +
            "import org.drools.Person\n" +
            "global java.util.List list\n" +
            "declare Person\n" +
            "    @typesafe(false)\n" +
            "end\n" +
            "rule testTypeSafe\n dialect \"mvel\" when\n" +
            "   $p : Person( object.street == 's1' )\n" +
            "then\n" +
            "   list.add( $p );\n" +
            "end\n";            
        executeTypeSafeDeclarations( str, true );  
        
        // different namespace without import using qualified name
        str = "package org.drools.test\n" +
            "global java.util.List list\n" +
            "declare org.drools.Person\n" +
            "    @typesafe(false)\n" +
            "end\n" +
            "rule testTypeSafe\n dialect \"mvel\" when\n" +
            "   $p : org.drools.Person( object.street == 's1' )\n" +
            "then\n" +
            "   list.add( $p );\n" +
            "end\n";        
        executeTypeSafeDeclarations( str, true );  
        
        // this should fail as it's not declared non typesafe 
        str = "package org.drools.test\n" +
            "global java.util.List list\n" +
            "declare org.drools.Person\n" +
            "    @typesafe(true)\n" +
            "end\n" +
            "rule testTypeSafe\n dialect \"mvel\" when\n" +
            "   $p : org.drools.Person( object.street == 's1' )\n" +
            "then\n" +
            "   list.add( $p );\n" +
            "end\n";        
        executeTypeSafeDeclarations( str, false );            
    }    
 
    private void executeTypeSafeDeclarations(String str, boolean mustSucceed) {            
        KnowledgeBuilder kbuilder = KnowledgeBuilderFactory.newKnowledgeBuilder();
        kbuilder.add( ResourceFactory.newByteArrayResource( str.getBytes() ),
                      ResourceType.DRL );

        if ( kbuilder.hasErrors() ) {
            if ( mustSucceed ) {
                fail( kbuilder.getErrors().toString() );
            } else {
                return;
            }
        } 

        KnowledgeBase kbase = KnowledgeBaseFactory.newKnowledgeBase();
        kbase.addKnowledgePackages( kbuilder.getKnowledgePackages() );

        StatefulKnowledgeSession ksession = kbase.newStatefulKnowledgeSession();
        List list = new ArrayList();
        ksession.setGlobal( "list",
                            list );

        Address a = new Address("s1" );
        Person p = new Person( "yoda" );
        p.setObject( a );
        
        ksession.insert( p );
        ksession.fireAllRules();
        assertEquals( p, list.get(0));        
    }     

Dynamic (non type safe) Expressions in Rules

AtomPub Interface for Guvnor

I have been using Drools since the spring of 2008, when I first began working as part of a research team in Mexico. Over that period, I have explored some of the functionality within the platform, taking advantage of a few different projects (drools-expert, drools-planner and drools-guvnor) to help with my work here; creating software games based upon the participatory modeling our group does with rural people. Over the last 6 months, I have been working to make our newer games more open to changes and modifications by players on the ground and in realtime. To do this, I have implemented our latest game so it’s rules and object models are hosted by the Guvnor.

Over this time, I noticed that the Guvnor wants to work as part of a centralized workflow, such as that in a team-based rule editing and publishing process. This has made it somewhat difficult to adapt for the type of work involved in our gaming. For example, we want users to input new rules which modify social behaviors in the game; a process which is handled well by the 5.1.0 Guvnor. However, we also want the rule package compiled directly and available after such edits, possibly through our own interface, so all players in the running game are effected. To do this, I wanted a way to remotely invoke package compilation.

At RulesFest, in October 2010, I spoke with Esteban Aliverti and Mauricio Salatino, about my problem. Over the next few days we discovered a few others who also had an interest remote compilation and snapshot creation. I offered to add the required functionality, and was asked to log a JIRA issue.

I wrote-up GUVNOR-1080 and attempted to resolve what I needed through an “actions” API. Jervis Liu folded in the changes into the 5.2.0.M1 branch which enabled the following GET methods on the Guvnor webstack:

• http://server:port/guvnor-webapp/actions/compile-package
• http://server:port/guvnor-webapp/actions/create-snapshot

Both require a “package-name” header (aka slug) to indicate which package to compile/snapshot. And both calls simply return a success report in the response (e.g., 204, 500). The compiled package or segregated snapshot are available through the customary Guvnor mechanism.

However, this was really not a satisfactory resolution of Guvnor-1080.

Further talk about the overall approach to REST access in the Guvnor project expanded from the JIRA issue, ad hoc discussions on the dev list to Jevis Liu’s “AtomPub Interface for Guvnor” proposal on the Drools community wikI. Taking off from where Michael Neale’s original “Guvnor AtomPub Interface” started in 2008 and expanding into categories, metadata and other return types, this page became a rough, but working implementation specification.

Implementation

Two main JAX-RS resources have been added to the Guvnor for use by REST based clients, these are the PackageResource and the CategoryResource.

All resources return and accept information in the following formats:

• Application/Atom+XML
• Application/JSON
• Application/XML

Packages and Assets

The PackageResource allows clients access to packages and assets. The following methods are exposed via GET in the new REST based API for packages:

• http://server:port/guvnor-webapp/rest/packages
• http://server:port/guvnor-webapp/rest/packages/PACKAGENAME
• http://server:port/guvnor-webapp/rest/packages/PACKAGENAME/source
• http://server:port/guvnor-webapp/rest/packages/PACKAGENAME/binary
• http://server:port/guvnor-webapp/rest/packages/PACKAGENAME/assets
• http://server:port/guvnor-webapp/rest/packages/PACKAGENAME/assets/ASSETNAME
• http://server:port/guvnor-webapp/rest/packages/PACKAGENAME/assets/ASSETNAME/binary
• http://server:port/guvnor-webapp/rest/packages/PACKAGENAME/assets/ASSETNAME/source

Where “PACKAGENAME” means the package to be looked at and “ASSETNAME” means the asset. The following method is exposed via POST, and allows users to create new packages or assets:

• http://server:port/guvnor-webapp/rest/packages/

Users need to push up either ATOM, JSON, or XML based content to create the package on the server, or use a DRL file in order to have the package created from that format.

The following methods are exposed via PUT, and allow users to update packages or asserts:

• http://server:port/guvnor-webapp/rest/packages/PACKAGENAME
• http://server:port/guvnor-webapp/rest/packages/PACKAGENAME/assets/ASSETNAME

And the following methods are supported via DELETE:

• http://server:port/guvnor-webapp/rest/packages/PACKAGENAME
• http://server:port/guvnor-webapp/rest/packages/PACKAGENAME/assets/ASSETNAME

Categories

Category access is only available for doing searches across the server based upon a specific category name. Results are returned in a paged format, and allow users to navigate across large datasets. The following GET methods are exposed for Categories:

• http://server:port/guvnor-webapp/rest/category/CATEGORYNAME
• http://server:port/guvnor-webapp/rest/category/CATEGORYNAME/page/PAGENUMBER

Where “CATEGORYNAME” refers to an existing category on the server, and “PAGENUMBER” to the page number in the result set.

Idempotency

REST has certain guarantees regarding the idempotency of assets in GET, PUT, OPTIONS and DELETE methods. AS RFC 2616 states:

Methods can also have the property of "idempotence" in that (aside from error or expiration issues) the side-effects of N > 0 identical requests is the same as for a single request. The methods GET, HEAD, PUT and DELETE share this property. [RFC 2616, page 50]

Idempotency in the Guvnor can only be guaranteed when working from a revision id perspective, not one based upon pure string based names, which is our current approach. As Guvnor front-ends a source control repository, queries to assets by name may only be IDEMPOTENT when said assets are addressed by their revision. Hence, although repetitive calls to GET, PUT, or DELETE are indempotent in context, actions by other users could update assets and packages so that calls will change over time.

Future

I’m anxious to hear from the community whether this implementation proves adequate. It could be a good idea to think about updating the GET methods with calls to Packages and Assets that are specific to a checked-in version in the Guvnor repository. It might also be a good idea to expand what gets serialized into the Package and Asset data from a given PUT or POST with an Atom/Json/XML entity. Finally, some more thought may be given to how metadata is handled, expanded and represented in the service.

AtomPub Interface for Guvnor

jBPM Oryx Integration - Gateway example

Per user request, here is a simple executable process example using Gateways and Rule Tasks in Oryx (click on image below to watch video):










This example also shows couple of improvements we made to the Oryx designer such as added support for process variable definition, and ability to edit your BPMN2 processes in a pop-out window which makes editing larger processes easier.

In order to recreate this example, you must use the latest version of the Oryx designer which you can download from here (rename it to designer.war and replace your existing one in $jbossHome/server/$config/deploy directory).

jBPM Oryx Integration - Gateway example

jBPM5 lightweight? Running on Android!

I recently bought myself a new Android phone, and as a developer, that obviously means I have to try and write my own application for it ! :)

And since we always claim jBPM is such a lightweight engine that you can embed it practically anywhere, I decided to give that a try: run the jBPM5 engine on Android. Some people might be thinking, why would I want to do that? Well, my primary goal was just as a proof of concept. But maybe, if we extend that proof of concept a little more and add more Android tasks, end users without any real development experience might be able to use it do model simple Android applications as well, who knows!

So after downloading the Eclipse tooling and following a hello world example, I decided to create a simple process that first asks the user for some keyword and then uses this keyword to search the jBPM community forum for entries containing that keyword (I know, not rocket science, you don't really need a workflow engine for that but it's just a demo example).


After adding the jBPM5 jars to the classpath, I updated my application to start this process when the application is started. I also created two domain-specific services for Android, one for requesting some input from the user (where I then bind this result to a process variable) and one for showing a web page (which shows a URL, which I created based on the input of the user).

And it worked almost[*] out of the box! So this is what it looks like (on the emulator, but it runs nicely on my phone as well). First you start the jBPM application ...



The first node in the process will ask you for a keyword you want to search for ...


And the last node will show the results of the query on the jBPM community forum in a browser ...


I'm pretty sure there are a lot of developers out there that try to play with new technologies like this, so if you're interested in playing with this and maybe extending this a little, let me know. Imagine we add a few more default services for showing an image, getting some file, getting GPS location, etc. Would be pretty cool if you could create more advanced applications like that.

I've uploaded the source code and the jbpm-android.apk so you can give it a try yourself !

[*] A small modification is needed in org.drools.util.CompositeClassLoader. Apparently, the Android JVM doesn't allow null as the parent classloader so I changed that to super(CompositeClassLoader.class.getClassLoader()) in the constructor, and that works fine.

jBPM5 lightweight? Running on Android!

Vote Salaboy!!! Community Award!!!

Mauricio has been a full time community worker thanks to PlugTree, who allow him so much freedom to work on Drools and jBPM.

Mauricio is the author of the "jBPM Developers Guide" book
https://www.packtpub.com/jboss-business-process-management-jbpm-developer-guide/book

He also blogs regular on his work here:
http://salaboy.wordpress.com/

Because of all his hard work, we've nominated him for two 2011 JBoss Community Recognition Awards categories. If you would like to show your appreciation for Mauricio (Salaboy) then please do sign in to your jboss.org account and vote for him.

• New Features
  • http://community.jboss.org/polls/1066
• Bug Fixes Poll
  • http://community.jboss.org/polls/1067
    

Mauricio(left) at a JUG

Vote Salaboy!!! Community Award!!!

Splitting up into smaller git repositories and smaller builds

Until recently, creating a change for Drools, jBPM, Guvnor, Planner or any of the other Drools modules could prove to be challenging. Not because of the code, but because the monolithic build was big and brittle.

It took quite a few minutes to build the whole thing, not to mention to test it, if it could build successfully at all. Hudson was red more often than it was blue/green and stayed red for weeks. Why? If a test in for example drools-core failed for a few days, hudson didn't build guvnor (or any of the dozen other modules). Meanwhile people committed changes that broke guvnor but hudson didn't complain. By the time that drools-core was fixed, multiple other modules were failing. So while we're fixing one test, other tests were broken and it became very hard to determine which change broke which test.

Who's to blame for Hudson being red so much? Nobody really. You can't blame a developer for not running all the tests before committing if running all the tests takes hours. And if some tests are already failing (and hudson is red) before his changes, it's hard for him to notice that his changes also break the build.

The real culprit was the monolithic build. There's no need to build drools (the rule engine) from source and test it if you're just working on guvnor. And we are in luck, because our code isn't monolithic at all: every module is lightweight (especially just-drools-the-rule-engine or just-jbpm-the-workflow-engine). There are just a lot of modules adding all kinds of functionality on top of those lightweight engines. So, we have split up the build, the git repository and the hudson jobs.

We used have 1 big, monolithic git repository:

• droolsjbpm:
• Contained everything (flow/jBPM was already removed some time ago)

Now we have several smaller repositories, each with it's own build and it's own Hudson job:

• droolsjbpm-build-bootstrap
• The parent pom (common stuff for the build)
• droolsjbpm-knowledge
• The public API and the internal API common to drools and jpbm
• drools
• The rule engine modules
• jbpm
• The workflow engine modules
• guvnor
• The web app to manage Drools & jBPM repositories
• drools-planner
• The planning optimizer modules
• droolsjbpm-integration
• Integrating Drools & jBPM with Seam, Spring, Camel, etc
• droolsjbpm-tools
• The eclipse, maven and ant plugins for drools & jbpm
• droolsjbpm-build-distribution
• OSGi bundles, top level assemblies, etc

Every repository can build alone: you don't need to clone the other repositories or build them from source (but you can of course). Some repositories do depend on SNAPSHOT's build by other repositories, but those SNAPSHOT's are deployed by Hudson and therefor available from Nexus.

Project	Git size	Locally (2th build): mvn install -DskipTests	Hudson (*): mvn install -Dfull
droolsjbpm-build-bootstrap	1 MB	3s	55s
droolsjbpm-knowledge	35 MB	6s	1m 53s
drools	160 MB	38s	? (red)
drools-planner	142 MB	9s	4m 34s
jbpm	21 MB	31s	11m 7s
droolsjbpm-integration	38 MB	23s	11m 0s
guvnor	131 MB	58s	21m 40s
droolsjbpm-tools	24 MB	?m ?s	? (red)
droolsjbpm-build-distribution	10 MB	? (todo)	? (red)
New total	562 MB	2m 48s (without tools, dist)	?
Old droolsjbpm	1200 MB	3m 18s in JAN 2011 5m 29s in DEC 2010 (without jbpm, tools, dist)	2h 41m 57s (without jbpm)

The git history has been successfully migrated and split up across the new git repositories (except for jbpm). Generated files (such as jars and GWT binaries) are removed from the Git history. Therefor, the new total git size is over 50% smaller.
Branches and tags have not been migrated because their monolithic build would not work. The canonical source for branches and tags up to and including 5.2.0.M1 is subversion. The old git droolsjbpm repository will be deleted to avoid confusion.

We're still ironing some things out (most notably some major refactors on drools) and we're also working on speeding up the tests for some projects (such as guvnor).
(*) The Hudson times are not reliable, because it is the result of a random build which may or may not needed to download dependencies. However 1 reason why the monolithic full build was so slow, was that adding the eclipse p2 repo in 1 module (droolsjbpm-tools/drools-eclipse) made all the other modules build a lot slower (and now it only affects the modules in droolsjbpm-tools). Wierd.

Summary: Cloning a repository, building it, testing it and sending us a pull request (or patch) now takes a fraction of the time it used to. So try it out! More info in the README file.

Splitting up into smaller git repositories and smaller builds