Drools-ant to the rescue!
This video shows an example on how to use a new drools-ant task to migrate from the default file-based Guvnor repository to one stored in MySQL. Same task can also be used to create back-ups of your existing repository.
Thursday, March 24, 2011
Using drools-ant to migrate/back-up existing Jackrabbit repositories
After using Guvnor for a while you may want to create a back-up of your existing JCR repository or migrate to one stored in an external database. Often this becomes a daunting task, especially if you already have a large number of existing resources in Guvnor.
Drools-ant to the rescue!
This video shows an example on how to use a new drools-ant task to migrate from the default file-based Guvnor repository to one stored in MySQL. Same task can also be used to create back-ups of your existing repository.
Drools-ant to the rescue!
This video shows an example on how to use a new drools-ant task to migrate from the default file-based Guvnor repository to one stored in MySQL. Same task can also be used to create back-ups of your existing repository.
Friday, March 18, 2011
JBoss Asylum on jBPM5
I recently did an interview for JBoss Asylum. They bring us regular updates on what's new at JBoss by doing podcasts and inviting guests to talk about special topics.
With the release of jBPM 5.0, now was my time to get attacked by Max Rydahl Andersen, Emmanual Bernard and Michael Neale ;)
With the release of jBPM 5.0, now was my time to get attacked by Max Rydahl Andersen, Emmanual Bernard and Michael Neale ;)
This episode had some bad luck with respect to the audio recording and took a while to get edited down to listenable quality. The news are therefore a bit out of date but the interview with Kris Verlaenen about jBPM 5 is all filled with great and interesting content.
The interview part has good audio and you can skip to the 22m33s mark to hear about jBPM 5.
We are again sorry that Emmanuel Bernard messed up the recording, but we will back soon with a new interviewee and with better sound ;)
If you got comments/feedback send it via mail or on twitter.
Get the episode from here
Tuesday, March 15, 2011
Dynamic (non type safe) Expressions in Rules
Drools is a statically typed language, which means it has to know the types of all fields up front. Sometimes, like in Collections, this just isn't possible.
Previously we hacked this such as if you used a map or array accessor in an expression it would switch dynamic execution in MVEL for this.
We have now cleaned up this feature and made it generally available. Using type declarations you can now declare a type as un-typesafe. That then means it will not validate the expressions or their types at compile time and it will execute those expressions in dynamic (non-strict type) mvel mode.
Here is the unit test for those that want to see it in action:
Previously we hacked this such as if you used a map or array accessor in an expression it would switch dynamic execution in MVEL for this.
We have now cleaned up this feature and made it generally available. Using type declarations you can now declare a type as un-typesafe. That then means it will not validate the expressions or their types at compile time and it will execute those expressions in dynamic (non-strict type) mvel mode.
Here is the unit test for those that want to see it in action:
@Test
public void testNoneTypeSafeDeclarations() {
// same namespace
String str = "package org.drools\n" +
"global java.util.List list\n" +
"declare Person\n" +
" @typesafe(false)\n" +
"end\n" +
"rule testTypeSafe\n dialect \"mvel\" when\n" +
" $p : Person( object.street == 's1' )\n" +
"then\n" +
" list.add( $p );\n" +
"end\n";
executeTypeSafeDeclarations( str, true );
// different namespace with import
str = "package org.drools.test\n" +
"import org.drools.Person\n" +
"global java.util.List list\n" +
"declare Person\n" +
" @typesafe(false)\n" +
"end\n" +
"rule testTypeSafe\n dialect \"mvel\" when\n" +
" $p : Person( object.street == 's1' )\n" +
"then\n" +
" list.add( $p );\n" +
"end\n";
executeTypeSafeDeclarations( str, true );
// different namespace without import using qualified name
str = "package org.drools.test\n" +
"global java.util.List list\n" +
"declare org.drools.Person\n" +
" @typesafe(false)\n" +
"end\n" +
"rule testTypeSafe\n dialect \"mvel\" when\n" +
" $p : org.drools.Person( object.street == 's1' )\n" +
"then\n" +
" list.add( $p );\n" +
"end\n";
executeTypeSafeDeclarations( str, true );
// this should fail as it's not declared non typesafe
str = "package org.drools.test\n" +
"global java.util.List list\n" +
"declare org.drools.Person\n" +
" @typesafe(true)\n" +
"end\n" +
"rule testTypeSafe\n dialect \"mvel\" when\n" +
" $p : org.drools.Person( object.street == 's1' )\n" +
"then\n" +
" list.add( $p );\n" +
"end\n";
executeTypeSafeDeclarations( str, false );
}
private void executeTypeSafeDeclarations(String str, boolean mustSucceed) {
KnowledgeBuilder kbuilder = KnowledgeBuilderFactory.newKnowledgeBuilder();
kbuilder.add( ResourceFactory.newByteArrayResource( str.getBytes() ),
ResourceType.DRL );
if ( kbuilder.hasErrors() ) {
if ( mustSucceed ) {
fail( kbuilder.getErrors().toString() );
} else {
return;
}
}
KnowledgeBase kbase = KnowledgeBaseFactory.newKnowledgeBase();
kbase.addKnowledgePackages( kbuilder.getKnowledgePackages() );
StatefulKnowledgeSession ksession = kbase.newStatefulKnowledgeSession();
List list = new ArrayList();
ksession.setGlobal( "list",
list );
Address a = new Address("s1" );
Person p = new Person( "yoda" );
p.setObject( a );
ksession.insert( p );
ksession.fireAllRules();
assertEquals( p, list.get(0));
}
Thursday, March 10, 2011
AtomPub Interface for Guvnor
I have been using Drools since the spring of 2008, when I first began working as part of a research team in Mexico. Over that period, I have explored some of the functionality within the platform, taking advantage of a few different projects (drools-expert, drools-planner and drools-guvnor) to help with my work here; creating software games based upon the participatory modeling our group does with rural people. Over the last 6 months, I have been working to make our newer games more open to changes and modifications by players on the ground and in realtime. To do this, I have implemented our latest game so it’s rules and object models are hosted by the Guvnor.
Over this time, I noticed that the Guvnor wants to work as part of a centralized workflow, such as that in a team-based rule editing and publishing process. This has made it somewhat difficult to adapt for the type of work involved in our gaming. For example, we want users to input new rules which modify social behaviors in the game; a process which is handled well by the 5.1.0 Guvnor. However, we also want the rule package compiled directly and available after such edits, possibly through our own interface, so all players in the running game are effected. To do this, I wanted a way to remotely invoke package compilation.
At RulesFest, in October 2010, I spoke with Esteban Aliverti and Mauricio Salatino, about my problem. Over the next few days we discovered a few others who also had an interest remote compilation and snapshot creation. I offered to add the required functionality, and was asked to log a JIRA issue.
I wrote-up GUVNOR-1080 and attempted to resolve what I needed through an “actions” API. Jervis Liu folded in the changes into the 5.2.0.M1 branch which enabled the following GET methods on the Guvnor webstack:
However, this was really not a satisfactory resolution of Guvnor-1080.
Further talk about the overall approach to REST access in the Guvnor project expanded from the JIRA issue, ad hoc discussions on the dev list to Jevis Liu’s “AtomPub Interface for Guvnor” proposal on the Drools community wikI. Taking off from where Michael Neale’s original “Guvnor AtomPub Interface” started in 2008 and expanding into categories, metadata and other return types, this page became a rough, but working implementation specification.
Two main JAX-RS resources have been added to the Guvnor for use by REST based clients, these are the PackageResource and the CategoryResource.
All resources return and accept information in the following formats:
Packages and Assets
The PackageResource allows clients access to packages and assets. The following methods are exposed via GET in the new REST based API for packages:
Users need to push up either ATOM, JSON, or XML based content to create the package on the server, or use a DRL file in order to have the package created from that format.
The following methods are exposed via PUT, and allow users to update packages or asserts:
Categories
Category access is only available for doing searches across the server based upon a specific category name. Results are returned in a paged format, and allow users to navigate across large datasets. The following GET methods are exposed for Categories:
REST has certain guarantees regarding the idempotency of assets in GET, PUT, OPTIONS and DELETE methods. AS RFC 2616 states:
I’m anxious to hear from the community whether this implementation proves adequate. It could be a good idea to think about updating the GET methods with calls to Packages and Assets that are specific to a checked-in version in the Guvnor repository. It might also be a good idea to expand what gets serialized into the Package and Asset data from a given PUT or POST with an Atom/Json/XML entity. Finally, some more thought may be given to how metadata is handled, expanded and represented in the service.
Over this time, I noticed that the Guvnor wants to work as part of a centralized workflow, such as that in a team-based rule editing and publishing process. This has made it somewhat difficult to adapt for the type of work involved in our gaming. For example, we want users to input new rules which modify social behaviors in the game; a process which is handled well by the 5.1.0 Guvnor. However, we also want the rule package compiled directly and available after such edits, possibly through our own interface, so all players in the running game are effected. To do this, I wanted a way to remotely invoke package compilation.
At RulesFest, in October 2010, I spoke with Esteban Aliverti and Mauricio Salatino, about my problem. Over the next few days we discovered a few others who also had an interest remote compilation and snapshot creation. I offered to add the required functionality, and was asked to log a JIRA issue.
I wrote-up GUVNOR-1080 and attempted to resolve what I needed through an “actions” API. Jervis Liu folded in the changes into the 5.2.0.M1 branch which enabled the following GET methods on the Guvnor webstack:
- http://server:port/guvnor-webapp/actions/compile-package
- http://server:port/guvnor-webapp/actions/create-snapshot
However, this was really not a satisfactory resolution of Guvnor-1080.
Further talk about the overall approach to REST access in the Guvnor project expanded from the JIRA issue, ad hoc discussions on the dev list to Jevis Liu’s “AtomPub Interface for Guvnor” proposal on the Drools community wikI. Taking off from where Michael Neale’s original “Guvnor AtomPub Interface” started in 2008 and expanding into categories, metadata and other return types, this page became a rough, but working implementation specification.
Implementation
Two main JAX-RS resources have been added to the Guvnor for use by REST based clients, these are the PackageResource and the CategoryResource.
All resources return and accept information in the following formats:
- Application/Atom+XML
- Application/JSON
- Application/XML
Packages and Assets
The PackageResource allows clients access to packages and assets. The following methods are exposed via GET in the new REST based API for packages:
- http://server:port/guvnor-webapp/rest/packages
- http://server:port/guvnor-webapp/rest/packages/PACKAGENAME
- http://server:port/guvnor-webapp/rest/packages/PACKAGENAME/source
- http://server:port/guvnor-webapp/rest/packages/PACKAGENAME/binary
- http://server:port/guvnor-webapp/rest/packages/PACKAGENAME/assets
- http://server:port/guvnor-webapp/rest/packages/PACKAGENAME/assets/ASSETNAME
- http://server:port/guvnor-webapp/rest/packages/PACKAGENAME/assets/ASSETNAME/binary
- http://server:port/guvnor-webapp/rest/packages/PACKAGENAME/assets/ASSETNAME/source
Users need to push up either ATOM, JSON, or XML based content to create the package on the server, or use a DRL file in order to have the package created from that format.
The following methods are exposed via PUT, and allow users to update packages or asserts:
- http://server:port/guvnor-webapp/rest/packages/PACKAGENAME
- http://server:port/guvnor-webapp/rest/packages/PACKAGENAME/assets/ASSETNAME
- http://server:port/guvnor-webapp/rest/packages/PACKAGENAME
- http://server:port/guvnor-webapp/rest/packages/PACKAGENAME/assets/ASSETNAME
Categories
Category access is only available for doing searches across the server based upon a specific category name. Results are returned in a paged format, and allow users to navigate across large datasets. The following GET methods are exposed for Categories:
- http://server:port/guvnor-webapp/rest/category/CATEGORYNAME
- http://server:port/guvnor-webapp/rest/category/CATEGORYNAME/page/PAGENUMBER
Idempotency
REST has certain guarantees regarding the idempotency of assets in GET, PUT, OPTIONS and DELETE methods. AS RFC 2616 states:
Methods can also have the property of "idempotence" in that (aside from error or expiration issues) the side-effects of N > 0 identical requests is the same as for a single request. The methods GET, HEAD, PUT and DELETE share this property. [RFC 2616, page 50]Idempotency in the Guvnor can only be guaranteed when working from a revision id perspective, not one based upon pure string based names, which is our current approach. As Guvnor front-ends a source control repository, queries to assets by name may only be IDEMPOTENT when said assets are addressed by their revision. Hence, although repetitive calls to GET, PUT, or DELETE are indempotent in context, actions by other users could update assets and packages so that calls will change over time.
Future
I’m anxious to hear from the community whether this implementation proves adequate. It could be a good idea to think about updating the GET methods with calls to Packages and Assets that are specific to a checked-in version in the Guvnor repository. It might also be a good idea to expand what gets serialized into the Package and Asset data from a given PUT or POST with an Atom/Json/XML entity. Finally, some more thought may be given to how metadata is handled, expanded and represented in the service.
Tuesday, March 08, 2011
jBPM Oryx Integration - Gateway example
Per user request, here is a simple executable process example using Gateways and Rule Tasks in Oryx (click on image below to watch video):
This example also shows couple of improvements we made to the Oryx designer such as added support for process variable definition, and ability to edit your BPMN2 processes in a pop-out window which makes editing larger processes easier.
In order to recreate this example, you must use the latest version of the Oryx designer which you can download from here (rename it to designer.war and replace your existing one in $jbossHome/server/$config/deploy directory).
This example also shows couple of improvements we made to the Oryx designer such as added support for process variable definition, and ability to edit your BPMN2 processes in a pop-out window which makes editing larger processes easier.
In order to recreate this example, you must use the latest version of the Oryx designer which you can download from here (rename it to designer.war and replace your existing one in $jbossHome/server/$config/deploy directory).
jBPM5 lightweight? Running on Android!
I recently bought myself a new Android phone, and as a developer, that obviously means I have to try and write my own application for it ! :)
And since we always claim jBPM is such a lightweight engine that you can embed it practically anywhere, I decided to give that a try: run the jBPM5 engine on Android. Some people might be thinking, why would I want to do that? Well, my primary goal was just as a proof of concept. But maybe, if we extend that proof of concept a little more and add more Android tasks, end users without any real development experience might be able to use it do model simple Android applications as well, who knows!
So after downloading the Eclipse tooling and following a hello world example, I decided to create a simple process that first asks the user for some keyword and then uses this keyword to search the jBPM community forum for entries containing that keyword (I know, not rocket science, you don't really need a workflow engine for that but it's just a demo example).
After adding the jBPM5 jars to the classpath, I updated my application to start this process when the application is started. I also created two domain-specific services for Android, one for requesting some input from the user (where I then bind this result to a process variable) and one for showing a web page (which shows a URL, which I created based on the input of the user).
And it worked almost[*] out of the box! So this is what it looks like (on the emulator, but it runs nicely on my phone as well). First you start the jBPM application ...
The first node in the process will ask you for a keyword you want to search for ...
And the last node will show the results of the query on the jBPM community forum in a browser ...
I'm pretty sure there are a lot of developers out there that try to play with new technologies like this, so if you're interested in playing with this and maybe extending this a little, let me know. Imagine we add a few more default services for showing an image, getting some file, getting GPS location, etc. Would be pretty cool if you could create more advanced applications like that.
I've uploaded the source code and the jbpm-android.apk so you can give it a try yourself !
And since we always claim jBPM is such a lightweight engine that you can embed it practically anywhere, I decided to give that a try: run the jBPM5 engine on Android. Some people might be thinking, why would I want to do that? Well, my primary goal was just as a proof of concept. But maybe, if we extend that proof of concept a little more and add more Android tasks, end users without any real development experience might be able to use it do model simple Android applications as well, who knows!
So after downloading the Eclipse tooling and following a hello world example, I decided to create a simple process that first asks the user for some keyword and then uses this keyword to search the jBPM community forum for entries containing that keyword (I know, not rocket science, you don't really need a workflow engine for that but it's just a demo example).
After adding the jBPM5 jars to the classpath, I updated my application to start this process when the application is started. I also created two domain-specific services for Android, one for requesting some input from the user (where I then bind this result to a process variable) and one for showing a web page (which shows a URL, which I created based on the input of the user).
And it worked almost[*] out of the box! So this is what it looks like (on the emulator, but it runs nicely on my phone as well). First you start the jBPM application ...
The first node in the process will ask you for a keyword you want to search for ...
And the last node will show the results of the query on the jBPM community forum in a browser ...
I'm pretty sure there are a lot of developers out there that try to play with new technologies like this, so if you're interested in playing with this and maybe extending this a little, let me know. Imagine we add a few more default services for showing an image, getting some file, getting GPS location, etc. Would be pretty cool if you could create more advanced applications like that.
I've uploaded the source code and the jbpm-android.apk so you can give it a try yourself !
[*] A small modification is needed in org.drools.util.CompositeClassLoader. Apparently, the Android JVM doesn't allow null as the parent classloader so I changed that to super(CompositeClassLoader.class.getClassLoader()) in the constructor, and that works fine.
Thursday, March 03, 2011
Vote Salaboy!!! Community Award!!!
Mauricio has been a full time community worker thanks to PlugTree, who allow him so much freedom to work on Drools and jBPM.
Mauricio is the author of the "jBPM Developers Guide" book
https://www.packtpub.com/jboss-business-process-management-jbpm-developer-guide/book
He also blogs regular on his work here:
http://salaboy.wordpress.com/
Because of all his hard work, we've nominated him for two 2011 JBoss Community Recognition Awards categories. If you would like to show your appreciation for Mauricio (Salaboy) then please do sign in to your jboss.org account and vote for him.
Mauricio(left) at a JUG
Mauricio is the author of the "jBPM Developers Guide" book
https://www.packtpub.com/jboss-business-process-management-jbpm-developer-guide/book
He also blogs regular on his work here:
http://salaboy.wordpress.com/
Because of all his hard work, we've nominated him for two 2011 JBoss Community Recognition Awards categories. If you would like to show your appreciation for Mauricio (Salaboy) then please do sign in to your jboss.org account and vote for him.
- New Features
- Bug Fixes Poll
Mauricio(left) at a JUG
Tuesday, March 01, 2011
Splitting up into smaller git repositories and smaller builds
Until recently, creating a change for Drools, jBPM, Guvnor, Planner or any of the other Drools modules could prove to be challenging. Not because of the code, but because the monolithic build was big and brittle.
It took quite a few minutes to build the whole thing, not to mention to test it, if it could build successfully at all. Hudson was red more often than it was blue/green and stayed red for weeks. Why? If a test in for example drools-core failed for a few days, hudson didn't build guvnor (or any of the dozen other modules). Meanwhile people committed changes that broke guvnor but hudson didn't complain. By the time that drools-core was fixed, multiple other modules were failing. So while we're fixing one test, other tests were broken and it became very hard to determine which change broke which test.
Who's to blame for Hudson being red so much? Nobody really. You can't blame a developer for not running all the tests before committing if running all the tests takes hours. And if some tests are already failing (and hudson is red) before his changes, it's hard for him to notice that his changes also break the build.
The real culprit was the monolithic build. There's no need to build drools (the rule engine) from source and test it if you're just working on guvnor. And we are in luck, because our code isn't monolithic at all: every module is lightweight (especially just-drools-the-rule-engine or just-jbpm-the-workflow-engine). There are just a lot of modules adding all kinds of functionality on top of those lightweight engines. So, we have split up the build, the git repository and the hudson jobs.
We used have 1 big, monolithic git repository:
Now we have several smaller repositories, each with it's own build and it's own Hudson job:
The git history has been successfully migrated and split up across the new git repositories (except for jbpm). Generated files (such as jars and GWT binaries) are removed from the Git history. Therefor, the new total git size is over 50% smaller.
Branches and tags have not been migrated because their monolithic build would not work. The canonical source for branches and tags up to and including 5.2.0.M1 is subversion. The old git droolsjbpm repository will be deleted to avoid confusion.
We're still ironing some things out (most notably some major refactors on drools) and we're also working on speeding up the tests for some projects (such as guvnor).
(*) The Hudson times are not reliable, because it is the result of a random build which may or may not needed to download dependencies. However 1 reason why the monolithic full build was so slow, was that adding the eclipse p2 repo in 1 module (droolsjbpm-tools/drools-eclipse) made all the other modules build a lot slower (and now it only affects the modules in droolsjbpm-tools). Wierd.
Summary: Cloning a repository, building it, testing it and sending us a pull request (or patch) now takes a fraction of the time it used to. So try it out! More info in the README file.
It took quite a few minutes to build the whole thing, not to mention to test it, if it could build successfully at all. Hudson was red more often than it was blue/green and stayed red for weeks. Why? If a test in for example drools-core failed for a few days, hudson didn't build guvnor (or any of the dozen other modules). Meanwhile people committed changes that broke guvnor but hudson didn't complain. By the time that drools-core was fixed, multiple other modules were failing. So while we're fixing one test, other tests were broken and it became very hard to determine which change broke which test.
Who's to blame for Hudson being red so much? Nobody really. You can't blame a developer for not running all the tests before committing if running all the tests takes hours. And if some tests are already failing (and hudson is red) before his changes, it's hard for him to notice that his changes also break the build.
The real culprit was the monolithic build. There's no need to build drools (the rule engine) from source and test it if you're just working on guvnor. And we are in luck, because our code isn't monolithic at all: every module is lightweight (especially just-drools-the-rule-engine or just-jbpm-the-workflow-engine). There are just a lot of modules adding all kinds of functionality on top of those lightweight engines. So, we have split up the build, the git repository and the hudson jobs.
We used have 1 big, monolithic git repository:
- droolsjbpm:
- Contained everything (flow/jBPM was already removed some time ago)
Now we have several smaller repositories, each with it's own build and it's own Hudson job:
- droolsjbpm-build-bootstrap
- The parent pom (common stuff for the build)
- droolsjbpm-knowledge
- The public API and the internal API common to drools and jpbm
- drools
- The rule engine modules
- jbpm
- The workflow engine modules
- guvnor
- The web app to manage Drools & jBPM repositories
- drools-planner
- The planning optimizer modules
- droolsjbpm-integration
- Integrating Drools & jBPM with Seam, Spring, Camel, etc
- droolsjbpm-tools
- The eclipse, maven and ant plugins for drools & jbpm
- droolsjbpm-build-distribution
- OSGi bundles, top level assemblies, etc
| Project | Git size | Locally (2th build): mvn install -DskipTests | Hudson (*): mvn install -Dfull |
|---|---|---|---|
| droolsjbpm-build-bootstrap | 1 MB | 3s | 55s |
| droolsjbpm-knowledge | 35 MB | 6s | 1m 53s |
| drools | 160 MB | 38s | ? (red) |
| drools-planner | 142 MB | 9s | 4m 34s |
| jbpm | 21 MB | 31s | 11m 7s |
| droolsjbpm-integration | 38 MB | 23s | 11m 0s |
| guvnor | 131 MB | 58s | 21m 40s |
| droolsjbpm-tools | 24 MB | ?m ?s | ? (red) |
| droolsjbpm-build-distribution | 10 MB | ? (todo) | ? (red) |
| New total | 562 MB | 2m 48s (without tools, dist) | ? |
| Old droolsjbpm | 1200 MB | 3m 18s in JAN 2011 5m 29s in DEC 2010 (without jbpm, tools, dist) | 2h 41m 57s (without jbpm) |
The git history has been successfully migrated and split up across the new git repositories (except for jbpm). Generated files (such as jars and GWT binaries) are removed from the Git history. Therefor, the new total git size is over 50% smaller.
Branches and tags have not been migrated because their monolithic build would not work. The canonical source for branches and tags up to and including 5.2.0.M1 is subversion. The old git droolsjbpm repository will be deleted to avoid confusion.
We're still ironing some things out (most notably some major refactors on drools) and we're also working on speeding up the tests for some projects (such as guvnor).
(*) The Hudson times are not reliable, because it is the result of a random build which may or may not needed to download dependencies. However 1 reason why the monolithic full build was so slow, was that adding the eclipse p2 repo in 1 module (droolsjbpm-tools/drools-eclipse) made all the other modules build a lot slower (and now it only affects the modules in droolsjbpm-tools). Wierd.
Summary: Cloning a repository, building it, testing it and sending us a pull request (or patch) now takes a fraction of the time it used to. So try it out! More info in the README file.
Subscribe to:
Posts (Atom)