Friday, May 31, 2013

Stress testing the Phreak algorithm

Some time ago, Mark blogged about his new rule engine algorithm. This new algorithm, an alternative to RETE, has been named Phreak. It's a drop-in replacement deep inside Drools, so from a user's perspective, nothing changes in your Drools related code.

Such a new algorithm is a big change, so regression testing is critical. Luckily we have a battery of tests.

Drools's tests


First up were the 3305 tests in Drools itself. A large portion of these tests are small DRL integration tests which insert/modify/remove a few facts and call fireAllRules() a few times.

Mark's initial Phreak implementation passed all these tests a few weeks ago. Given the test coverage size and the complexity of some of these tests, one would think this enough to label Phreak as production ready. Luckily, we test more.

OptaPlanner's stress tests


Next up, are OptaPlanner's stress tests. These tests solve an OptaPlanner example, while continuously modifying a fact and calling fireAllRules() on a single stateful Drools session. They also continuously assert the incremental results of the stateful Drools session with a fresh Drools session. Any bug in the fast incremental delta's of a stateful session quickly distorts the results slightly and fails the assertions.

Each test of the most notorious of these tests - named the TurtleTests - call fireAllRules() about 5000 times per second for a duration of 10 minutes on a dataset of thousands of facts. There are 153 TurtleTests, each using a separate dataset, across 11 realistic use cases. This results in pounding the Drools rule engine for over 25 hours by calling over 459 million times fireAllRules().

Yes, we did find issues :) About 20 of them... and they were infamously complex too. But Mario, Mark and I have isolated and fixed each one for 6.0.0.Beta3. Except one exotic bug, which will be fixed for 6.0.0.Beta4.

Try Phreak yourself


To use Phreak instead of RETE for yourself in 6.0.0.Beta3, enable it like this in Drools:

KieBaseConfiguration conf
    = KnowledgeBaseFactory.newKnowledgeBaseConfiguration();
conf.setOption(PhreakOption.ENABLED);
KieBase kieBase = kieContainer.newKieBase(conf);

In OptaPlanner 6.0.0.Beta3, configure it in the *SolverConfig.xml like this instead:

<scoreDirectorFactory>
  ...
  <kieBaseConfigurationProperties>
    <drools.phreakEnabled>true</drools.phreakEnabled>

  </kieBaseConfigurationProperties>
</scoreDirectorFactory>

Benchmarking


Phreak is considerably faster than RETE. Stay tuned for the benchmark reports...

Tuesday, May 28, 2013

What's Coming in Drools 6.0 Slides

Recently I gave a talk at the London JBUG, on what's coming in Drools 6.0. Those slides are now available online.

HTML export from Keynote (apparently some firefox issues, but fine in Safari and Chrome):
http://docs.jboss.org/drools/blog/LondonJBUG2013rulesmarkp/assets/player/KeynoteDHTMLPlayer.html

Here are some relevant videos:
Drools and Guvnor demo
http://blog.athico.com/2013/05/drools-and-guvnor-beta3-video.html 

Backward Chaining and Reactive Transitive Closures
http://blog.athico.com/2013/05/backward-chaining-and-reactive.html

How to build and run the Drools and Guvnor demo:
http://blog.athico.com/2013/05/how-to-build-and-run-drools-and-guvnor.html

Sunday, May 26, 2013

How to build and run the Drools and Guvnor Beta3 Demo

After the release of the Drools and Guvnor beta3 video see here, I'm being asked how people can build and run it themselves. Luckily it's very easy, so here are some quick instructions :)
  • Make sure you set your MAVEN_OPTS:
    • MAVEN_OPTS="-Xms512m -Xmx1024m -XX:MaxPermSize=128m"
  • Git clone:
    • https://github.com/droolsjbpm/guvnor
  • From clone root, built the main application:
    • mvn clean install -DskipTests
  • Change to the WAR module:
    • cd drools-wb/drools-wb-distribution-wars
  • Build the WAR
    • mvn clean install -DskipTests
    • Builds Tomcat and JBoss AS WARs, with the firefox profile.
      • WAR only works with Firefox, but quick to compile, for testing.
  • Download unzip and start JBoss AS7.1
    • http://download.jboss.org/jbossas/7.1/jboss-as-7.1.1.Final/jboss-as-7.1.1.Final.zip
    • ./standalone.sh
    • Once started, don't forget to create a management user with add-user.sh
  • Install the WAR via the AS web console
    • Find the WAR and upload it, AS automatically installs it
    • drools-workbench-6.0.0-SNAPSHOT-jboss-as7.0.war
  • Go to URL and login
    • http://localhost:8080/drools-workbench-6.0.0-SNAPSHOT-jboss-as7.0
    • login/pass admin/admin
    • Must use fireFox, as maven build defaults to reduced number of profile sets, for fast building. 
      • Use -PfullProfile with mvn install, to build all profiles, with longer build times.
  • There is a sample repo, but mostly has rubbish in it, stick to the uberfire playground.

Things to notice, or try:
  • Maven POM editor, when the file explorer is inside a maven project
  • As per my video, try building and installing the poject, and viewing in the maven manager
  • Clone some external projects, see if you can build and deploy them
  • Have a play with some of the custom editors, especially the guided editors and decision tables.

Friday, May 24, 2013

Backward Chaining and Reactive Transitive Closures with Drools

I've made a small tutorial on Backward Chaining and Reactive Transitive Closures with Drools. The slides. The video (Don't forget to turn on Youtube HD settings):






Drools and Guvnor Beta3 Video

I've made a small video to show the UI improvements for Drools and Guvnor. It's recorded in 1024x768, so it's slightly more cramped, people typically have their UI's open larger than that these days. Don't forget to turn on Youtube HD settings:
https://www.youtube.com/watch?v=ckAznbOOV-4


Tuesday, May 14, 2013

Creating your own Drools and jBPM Persistence with Infinispan


Original post here by me:

Hello and welcome to a post in which I intend to show you how to create your own implementation of drools and jBPM persistence. I’ve worked on an infinispan based persistence scheme for drools objects and I learnt a lot in the process. It’s my intention to give you a few pointers if you wish to do something of the sort.

Why?

If you’re reading this, you probably already have a “why” to redefine the persistence scheme that drools uses, but it’s good to go over some good reasons to do something like this. The most important thing is that you might consider the JPA persistence scheme designed for drools doesn’t meet your needs for one or more reasons. Some of the most common I’ve found are these:
The given model is not enough for my design: Current objects created to persist the drools components (sessions, process instances, work items and so on) currently are as small as possible to allow the best performance on the database, and most of the operational data is stored in byte arrays mapped to blob objects. This scheme is enough for the drools and jBPM runtime to function, but it might not be enough for your domain. You might want to keep the runtime information in a scheme that is easier to query from outside tools, and to do that you would need to enrich the data model, and even create one of your own.
The persistence I’m using is not compatible with JPA: There are a lot of persistence implementations out there that no longer use databases as we once knew (distributed caches, key value storages, NoSQL databases) and the model usually needs extra mappings and special treatment when persisting in such storages. To do so, sometimes JPA is not our cup of tea
I need to load special entities from different sources every time a drools component is loaded: When we have complex objects and/or external databases, sometimes we want new models to relate in a special way to the objects we have. Maybe we want to make sure our sessions are binded to our model in a special way because it makes sense to our business model. To do so we would have to alter the model

How?

In order to make our own persistence scheme for our sessions, we need to understand clearly how the JPA scheme is built, to use it as a template to build our own. This class diagram shows how the JPA persistence scheme for the knowledge session is implemented:

Looks complicated, right? Don’t worry. We’ll go step by step to understand how it works.
First of all, you can see that we have two implementations of the StatefulKnowledgeSession (or KieSession, if you’re using Drools 6). The one that does all the “drools magic” is StatefulKnoweldgeSessionImpl, and the one we will be using is CommandBasedStatefulKnowledgeSession. It has nothing to do with persistence, but it helps a lot with it by surrounding every method call with a command object and deriving its execution to a command service. So, for example, if you call the fireAllRules method to this type of session, it will create a FireAllRulesCommand object and give it to another class to execute.
This command based implementation allows us to do exactly the thing we need to implement persistence on a drools environment: It lets us implement actions before and after every method call done to the session. That’s where the SingleSessionCommandService class comes in handy: This command service contains a StatefulKnowledgeSessionImpl and a PersistenceContextManager. Every time a command has to be executed, this class creates or loads a SessionInfo object and tells the persistence context to save it with all the state of the StatefulKnowledgeSessionImpl.
That’s the most complicated part: the one that implements the session persistence. Persistence of pretty much everything else is done easily through a set of given interfaces that provide methods to implement how to load everything else related to a session (process instances, work items and signals). As long as you create a proper manager and its factory, you can delegate on them to store anything to anywhere (or do anything you want, for that matter).

So, after seeing all the components, it’s a good time to start thinking of how to create our own implementation. For this example, we’ve created an Infinispan based persistence scheme and we will show you all the steps we took to do it.

Step 1: (re)define the model
Most of the time when we want to persist drools objects in our way, we might want to do it with a gist of your own. Even if we don’t wish to change the model, we might need to add special annotations to the model to work with your storage framework. Another reason might be that you want to store all facts in a special way to cross-query them with some other legacy system. You can literally do this redefinition any way you want, as long as you understand that whatever model you create, the persistence scheme will serialize and deserialize it every time you call a method on the knowledge session, so always try to keep it simple.
Here’s the model we created for this case:
Nothing too fancy, just a flattened model for all things drools related. We weren’t too imaginative with this model, because we just wanted to show you that you can change it if you want to.
One thing to notice in this model is that we are still saving all the internal data of these objects pretty much the same way as it is stored for the JPA persistence. The only difference is that  JPA stores it in a Blob, and we store it in a Base64 encrypted string. If you wish to change the way that byte array is generated and read, you have to create your own implementations of these interfaces:
  • org.kie.api.marshalling.Marshaller for knowledge sessions
  • org.jbpm.marshalling.impl.ProcessInstanceMarshaller for process instances
But providing an example of that would take way too much time and perhaps even a whole book to explain, so we’ll skip it :)

Step 2: Implementing the PersistenceContext
For some cases, redefining the PersistenceContext and the PersistenceContextManager would be enough to implement all your persistence requirements. The PersistenceContext is an object in charge of persisting work items and session objects by implementing methods to persist them, query them by ID and removing them from a particular storage implementation. The PersistenceContextManager is in charge of creating the PersistenceContext, either once for all the application or on a per-command basis. The comand service will use it to persist the session and its objects when needed.
In our case we implemented a PersistenceContext and a PersistenceContextManager using an Infinispan cache as storage. The different PersistenceContextManager instances will have access to all configuration objects through the Environment variable. We’ve used the already defined keys in Environment to store our Infinispan related objects:
  • EnvironmentName.ENTITY_MANAGER_FACTORY is used to store an Infinispan based CacheManager
  • EnvironmentName.APP_SCOPED_ENTITY_MANAGER and EnvironmentName.CMD_SCOPED_ENTITY_MANAGER will point to an Infinispan Cache object.
You can see that code here:


At this point we have some very important steps to redefining our drools persistence. Now we need to know how to configure our knowledge sessions to work with this components.

Step 3: Creating managers for our work items, process instances and signals

Now that we have our persistence contexts, we need to teach the session how to use them properly. The knowledge session has a few managers that can be configured that allow you to modify or alter the default behaviour. These managers are:
org.kie.api.runtime.process.WorkItemManager: It manages when a work item is executed, connects it with the proper handler, and notifies the process instance when the work item is completed.
org.jbpm.process.instance.event.SignalManager: It manages when a signal is sent to or from a process. Since process instances might be passivated, it needs to
org.jbpm.process.instance.ProcessInstanceManager: It manages the actions to be taken when a process instance is created, started, modified or completed.

JPA implementation of these interfaces already work with a persistence context manager, so most of the times you won’t need to extend them. However, with Infinispan, we have to make sure the process instance is persisted more often than with JPA, so we had to implement them differently.
Once you have these instances, you will need to create a factory for each type of manager.The interface names are the same, except with the suffix “Factory”. Each receives a knowledge session as parameter, from which you can get the Environment object and all other configurations.

Step 4: Configuring the knowledge session

Now that we have our different managers created, we will need to tell our knowledge sessions to use them. To do so you need to create a CommandBasedStatefulKnowledgeSession instance with a SingleSessionCommandService instance. The SingleSessionCommandService, as its name describes, is a class to execute commands against one session at a time. SingleSessionCommandService’s constructor receives all parameters needed to create a proper session and execute commands against it in a way that it becomes persistent. Those parameters are:

  • KieBase: the knowledge base with the knowledge definitions for our session runtime.
  • KieSessionConfiguration: Where we configure the manager factories to create and dispose of work items, process instances and signals.
  • Environment: A bag of variables for any other purpose, where we will configure our persistence context mananager objects.
  • sessionId (optional): If present, this parameter looks for an already existing session in the storage. Otherwise, it creates a new one.

Also, in our example, we’re using Infinispan, which is not a reference based storage, but a value based storage. This means that once you say to infinispan to store a value, it will store a copy of it and not the actual object. Some things in drools persistence are managed to be stored through reference based storages, meaning you can tell the framework to persist an object, change its attributes, and see those changes stored in the database after committing the transaction. With infinispan, this wouldn’t happen, so you have to implement an update of the cache values after the command execution is finished. Luckily for us, the SingleSessionCommandService allows us to do this by implementing an Interceptor.
Interceptors are basically your own command service to wrap the default one. You can tell each command to add more behaviour before or after each execution. Here’s a couple of diagrams to explain how it works:

As you can see, the SingleSessionCommandService allows for a command service instance to actually invoke the command’s execute method. And thanks to the interceptor extension of the command service, we can add as many as we want in chain, allowing us to have something like the next sequence diagram executing every time a command needs execution:

In our case, we created a couple of these interceptors and added them to the SingleSessionCommandService. One makes sure any changes done to a session object are stored after finishing the command. The other one allows us to do the same with process instance objects.

Overall, this is how we need to create our knowledge sessions at this point to actually use infinispan as a persistence scheme:

Complicated, right? Don’t worry. There’s yet another couple of classes to make it easier to configure.

Step 4: Creating our own initiation service

Yes, we could write that ton of code every time we want to create our own customized persistent knowledge sessions. It’s a free world (for the most part). But you can also wrap this implementation in a single class with two exposed methods:
  • One to create a new session
  • One to load a previously existing session
And creates all the configuration internally, merging it whenever you wish to change one or more things. Drools provides an interface to serve as a contract for this called org.kie.api.persistence.jpa.KieStoreServices
We created our own implementation of this interface and also a static class to access it, called InfinispanKnowledgeService. This allows us to be able to create the session like this:


Conclusion

Drools persistence can seem complicated to understand and to get working, let alone to implement it in your own way. However, I hope this shows a bit of demystification to those who need to implement drools persistence in a special way, or were even wondering if it is possible to do so in any other way than JPA.
Also, if you wish to see the modifications done to make it work, see these three pull requests:
A feature request to add this features to Drools is specified in this JIRA ticket. Feel free to upvote it if you wish to have it as part of the core drools project!

Tuesday, May 07, 2013

London JBUG May Event - What's new in Drools 6.0 (22nd May 2013)

http://www.c2b2.co.uk/london_jbug_may_2013


London JBUG May Event - What's new in Drools 6.0

WHENWHEREHOW TO REGISTER?
Wednesday
22nd of May 2013
6pm-8:30pm
Skills Matter eXchange
116-120 Goswell Road
London, EC1V 7DP
Fill in the form at the bottom 
of the page!
If you are interested in attending the future JBUG events please join JBUG on Meetup

Presentation

Drools 6.0 introduces new approaches for authoring, building, deploying and utilising rules. Using convention and configuration, over programmatic apis, in an effort to simplify the experience for end users.
This talk, presented by Mark Proctor - Platform Architect at Red Hat - will take users through the new web based IDE (built with UberFire) for authoring and deploying rules, and then utilising them on the client side. It will also introduce the other new 6.0 features, including our new lazy rule engine algorithm. Ideally attendees will have a basic understanding of Drools and rule engines.

Speaker 

Mark Proctor received his B.Eng in Engineer Science and Technology and then his M.Sc. in Business and Information Systems; both from Brunel University, West London. His M.Sc. thesis was in the field of Genetic Algorithms; which is where he discovered his interest for anything AI related.
For the last 10 years Mark's focus has been on developing knowledge engineering systems in java, while working for Red Hat as Platform Architect. Mark is the co-founder of the Drools project, the leading Java expert system tool, and as Platform Architect he is responsible for all technologies related to rules, processes, events, ontologies and intelligent agents at Red Hat.

Agenda

18:00 - 18:15 - Coffee, Welcome and Networking 
18:15 - 19:30 - 'What's new in Drools 6.0' by Mark Proctor 
19:30 - 19:50 - Lightning Talks / Problem Solving Sessions
'Hands off my data! How to use Off-Heap Memory from Java and keep the latencies down' lightning talk will be presented by Jaromir Hamala
19:50 - 20:30 - Beer, Pizza and Networking
‘Problem Solving’ Sessions
Following your suggestions, we’re introducing the new Problem Solving panel sessions which give you an opportunity to discuss various JBoss-related problems you may want to share and discuss with the rest of the JBUG members. If you want to ask others for help and advice, discuss the issues, listen to suggestions and find the solutions – let us know! Email your topic to jbug@c2b2.co.uk and we will be more than happy to add your session to the agenda.
 

Lightning Talks

We are also opening up the floor to anyone using JBoss who has a tale to share. We are looking for a number of lightning talks 5 - 10 minutes in length where you can share your experiences, problems or wonderful solutions with the rest of the community. This is a huge opportunity to develop new or hone existing speaking skills!
If you are interested, please send the title of your talk to jbug@c2b2.co.uk .

Monday, May 06, 2013

Drools and jBPM talks at JUDCon:2013 United States


In Boston in June?  Meet some of the leaders of the Drools and jBPM teams!   Mark Proctor and Edson Tirelli will be be speaking this June at JBoss User and Developer Conference.  Take advantage of meeting Mark and Edson there.  It is a great opportunity to share your ideas and implementations.

Admission tickets get you into three separate events: Red Hat Developer Exchange, JBoss User Developer Conference (JUDCon) and CamelOne.  The events will be held in Boston Sunday June 9th - Tuesday June 11th.  



The activities start Sunday evening June 9th with the JUDCon, CamelOne and Red Hat Exchange reception. Then Monday and Tuesday, there will be 3 tracks of sessions, and 2 workshop tracks as well. The evening will also include a JBoss Core Developer panel, a live recording of the JBoss Community Asylum, and yes, beer.

The JBoss Users and Developers Conferences are developer gatherings held around the Globe to give JBoss users the chance to talk to and collaborate with Java contributors and core developers. The core JBoss developers, along with the open source community, create and support the projects that drive innovation and help lead development in standards bodies like the Java Community Process, the Apache Software Foundation, OASIS, W3C and other open standards organizations. Many of these projects become the upstream for Red Hat JBoss products.

3 tracks and 33 sessions will cover topics including:
  • Java and Java EE App Development
  • Mobile Development
  • Drools, jBPM, Fuse, ActiveMQ, Infinispan 
  • and many more
6 workshops providing hands-on labs covering:
  • Ceylon taught by Gavin King and Stephane Epardaud
  • Infinispan and JBoss Data Grid cross-datacenter replication
  • CDI, Forge and Errai
  • and many more.
JUDCon:2013 Boston Agenda



CamelOne is designed specifically for professionals using open source integration and messaging solutions in the enterprise and will feature a combination of keynote presentations, educational sessions and networking events that will enable attendees to meet, learn and share ideas with open source integration experts.

Founders, committers and users of Apache Camel, ServiceMix, ActiveMQ and CXF enjoyed a great Meet and Greet in the Exhibit Hall from 6:30 to 8:30. Stop by and mingle with your community over hors d’oeuvres and drinks!

CamelOne Agenda


Red Hat Connect Developer Exchange is heading back to Boston. You won't want to miss this one-day event where you can learn more about Red Hat developer tools and technologies. From gcc to Java to scripting languages, from traditional models to devops--You'll get the chance to connect with fellow developers, share real-world challenges, and solve mutual problems through collaboration.

5 tracks and 25 sessions will cover important topics, including:
• Programming on OpenShift Online PaaS
• OpenShift Enterprise and Java
• Languages and tools for mission-critical development
• Get more out of Red Hat Enterprise Linux tools

Red Hat Developer Exchange Agenda:


Friday, May 03, 2013

IntelliFest Oct 2013 - San Diego


Keynote Speakers:
  • Dr. Ernest Friedman-Hill (Jess) 
  • Dr. Daniel Miranker (Treat, Leaps, Venus)
  • Dr. Ellen Voorhees (Information Retrieval, National Institute of Standards and Technology, NIST)

International Conference
on Reasoning Technologies

October 7-11 • Bahia Resort Hotel

San Diego, CA • USA

IntelliFest returns to the charming Bahia Hotel for 2013! Join us for a full program featuring:

  • Single track technical conference for software and IT developers, programmers, engineers, and architects in the applied AI and rule-based domains.
  • Welcoming Reception on Sunday, October 6.
  • Technical Vertical Day on Monday, October 7.
  • Two days of boot camps on Monday, October 7 and Friday October 11!
  • Main Conference, Tuesday, October 8 through Thursday, October 10
  • AM plenary sessions with guest keynotes each day!
  • Hands-on, mini-camps all during the PM sessions of the main conference!

Topics include but are not limited to:

  • Reasoning About Big Data AI and Cloud Computing
  • AI and the Semantic Web
  • Emerging Standards
  • Latest News in Rule Engines
  • … and much more!

Wednesday, May 01, 2013

Google Summer of Code 2013

JBoss is participating in the Google Summer of Code 2013 program, which means that students can work on their favourite open-source project during the summer and get ultimate glory and a nice paycheck in return.
Google Summer of Code is a global program that offers students stipends to write code for open source projects. We have worked with the open source community to identify and fund exciting projects for the upcoming summer.
JBoss has created a list of possible ideas you can take a look at, but you can always propose us your own ideas as well !

For Drools & jBPM, we've added a few ideas to the list, but there is a Wiki page with over 10 possible jBPM proposals available here, including:
  • jBPM on android
  • Integrating jBPM with your own preferred project(s)
  • Realtime Collaborative Editor for Drools Decision Tables using GWT and Errai OT/EC
  • jBPM performance on steroids
  • Document management system
  • Mobile client(s) for jBPM
  • From BPEL to BPMN2
  • Social BPM using jBPM
  • Process mining for jBPM
  • jBPM and Drools for access control
  • jBPM and Drools for clinical decision support

The deadline for student applications is May 3rd, 19:00 UTC, so if you're interested but didn't submit anything yet, you'll need to be fast!