Tuesday, June 30, 2015

Uberfire, Drools and jBPM High Level Roadmap Slides and Videos from Red Hat Summit 2015

I did a short, 20 minute, high level presentation on some road map items for Uberfire, Drools and jBPM. I have provided the slides here, and it includes some early POC videos.

You can see the ongoing L&F work in the link below. The design is sleeker and more minimal. It now has a compact mode, that collapses the perspective switcher and sub-perspective menus -  see the “simple perspective”. Click the user name to switch modes. It automatically goes into compact mode when a panel is enlarged. 
user : admin
pass : admin

Mark

Share/Bookmark

Friday, June 19, 2015

Improved multi-threading behaviour with Drools 6.3 SNAPSHOT

We’ve rewritten the internal parts of our code that deal with multi-threading to remove a large number of synchronisation points and to improve stability and predictability. We believe that what we have done is now far more robust for the interaction of the User, Timer and Engine threads. Our initial benchmarking is showing that this has led to mild performance improvements too. We’d really like to get this hardened, before we do 6.3 final, so if you have an application that users Timers or Time Windows, especially when using FireUntilHalt, could you give it a good hammering? Especially those using the TimedRuleExecutionFilter, which allows a timer to fire reactively when the engine is in passive mode (not fireUntilHalt).

For this iteration we just focused on the engine internals, we have not yet touched the outer lock and sync points, i.e. the ksession and kbase locks that threads go through when they do an insert/update/delete action. These apparently can create contention for lots of small lived ksessions. We believe with the latest work we've been doing we can soon improve this area too.

You should find all this work in the latest snapshot, for drools-core and drools-compiler.
https://repository.jboss.org/nexus/content/repositories/snapshots/org/drools/drools-core/6.3.0-SNAPSHOT/
https://repository.jboss.org/nexus/content/repositories/snapshots/org/drools/drools-compiler/6.3.0-SNAPSHOT/

For those interest, we have done two things. The first part was to properly separate the User insert/update/delete thread actions with he Engine network evaluations thread.  The second part is to remove most of the internal sync points and replace with a state machine.

The User/Engine thread separation has been made possible by our move away from Rete to Phreak. With Rete the network evaluation is done during the User insert/update/delete action, meaning each user action locks the entire engine. With phreak the insert/update/delete is separated and network evaluation happens when fireAllRules or fireUntilHalt is called. We've added a queue, SynchronizedPropagationList, that stores up the user actions as commands, in a thread safe queue. The engine thread then takes all the entries on each of its iterations. We found our custom queue outperformed the JDK concurrent queues, but I think that is due to our specialist implementation. Instead of the engine taking just the HEAD entry, it does a takeAll and the processes that returned linked list as a batch. This reduces the amount of times the Engine thread hits the queue for each of the elements it processes. We can also efficiently handle when to park and when to notify the engine to spin up again, which was alway a bit hit and miss before. Now it simply parks when takeAll returns null, and it notifies if a Timer or User adds work be done and the engine is known to be parked.

The second part introduces a state machine for the User, Timer and Engine thread interactions. This now provides us with a system that we can ca be documented, due to it's simplification, and also this will help explain the various thread interactions and behaviours. This was missing before, and understanding the behaviour could be a be bit confusing for users. It also means we now have a better behaviour for the interactions of calling fireAllRules and fireUntilHalt and when they overlap, or are called twice. i..e what happens if you call fireUntilHalt while fireAllRules is currently operating? or you call fireAllRules twice, or fall fireAllRules when fireUntilHalt is operating? Our state machine now more cleanly handles this with describable behaviour.

The bulk of the work is contained within the DefaultAgenda:
https://github.com/droolsjbpm/drools/blob/master/drools-core/src/main/java/org/drools/core/common/DefaultAgenda.java

There are three threads that can interact. A User thread doing an insert/update/delete, the  Timer thread, for timers and time windows and the engine thread for network evaluations.  We have now changed this so that the timer thread no longer does network evaluations, blocking other threads, instead it submits a job and notifies the Engine thread (if it's not already running) to process it. You can see this in PhreakTimerNode. When the Timer now triggers it'll submit a job tot he queue that I introduced in the previous paragraph.
public void execute(JobContext ctx) {
    TimerNodeJobContext timerJobCtx = (TimerNodeJobContext) ctx;
    InternalWorkingMemory wm = timerJobCtx.getWorkingMemory();
    wm.addPropagation( new TimerAction( timerJobCtx ) );
}

When a timer thread is kicked off it has no idea if the engine thread is evaluating or parked. It could be parked because fireAllRules has returned and it's waiting for the next fireAllRules. Or it could be parked because fireUntilHalt currently has no work to do. If for instance the engine is parked in fireUntilHalt it needs to notify the engine thread to unpark and process the timer work. If however engine thread is working (be it fireUntilRules or fireUntilHalt) it should just put it into the queue for the engine thread to process and not do the notification. These interactions are subtle, but they must be solid and avoid contention or excessing syncing. The behaviour is complicated further by the TimedRuleExecutionFilter.

To handle this we introduced the following enum to represent the available states of the engine:
private enum ExecutionState {  // fireAllRule | fireUntilHalt | executeTask <-- action="" br="" required="">    INACTIVE( false ),         // fire        | fire          | exec    FIRING_ALL_RULES( true ),  // do nothing  | wait + fire   | enqueue    FIRING_UNTIL_HALT( true ), // do nothing  | do nothing    | enqueue    REST_HALTING( false ),     // wait + fire | wait + fire   | enqueue    FORCE_HALTING( false ),    // wait + fire | wait + fire   | wait + exec    EXECUTING_TASK( false );   // wait + fire | wait + fire   | wait + exec
    private final boolean firing;

    ExecutionState( boolean firing ) {
        this.firing = firing;
    }

    public boolean isFiring() {
        return firing;
    }
}

You can now see this state machine being used by fireAllRules and fireUntilHalt. Notice the new method waitAndEnterExecutionState. This allows threads to either park or return straight away - i.e. if you call fireAllRules and fireUntilHalt is running, just return straight away. If you call fireUntilHalt while fireAllRules is running, wait until fireAllRules finishes, then start fireUntilHalt.

public int fireAllRules(AgendaFilter agendaFilter,
                        int fireLimit) {
    synchronized (this) {
        if (currentState.isFiring()) {
            return 0;
        }
        waitAndEnterExecutionState( ExecutionState.FIRING_ALL_RULES );
    }

public void fireUntilHalt(final AgendaFilter agendaFilter) {
    synchronized (this) {
        if (currentState == ExecutionState.FIRING_UNTIL_HALT) {
            return;
        }
        waitAndEnterExecutionState( ExecutionState.FIRING_UNTIL_HALT );
    }

private void waitAndEnterExecutionState( ExecutionState newState ) {
    if (currentState != ExecutionState.INACTIVE) {
        try {
            wait();
        } catch (InterruptedException e) {
            throw new RuntimeException( e );
        }
    }
    currentState = newState;
}

Previously you saw the Timer thread submitted a job into a queue, this is also handled by the state machine.
public void executeTask( ExecutableEntry executable ) {
    synchronized (this) {
        if (isFiring() || currentState == ExecutionState.REST_HALTING) {
            executable.enqueue();
            return;
        }
        waitAndEnterExecutionState( ExecutionState.EXECUTING_TASK );
    }

    try {
        executable.execute();
    } finally {
        immediateHalt();
    }
}

A key aspect we had to support here was what if a Timer thread triggers some work while the Engine thread is just returning. You end up with gaps, so that's work that doesn't fire, that the user was expecting. This is a problem people have seen in previous Drools releases. The combination of this task system halting statuses, allow the engine to restart again before properly halting. You can think of it as a two phase halting system. You an see that with the main do loop and then the second while loop, ensuring we get a clean shut down - i.e. the engine cannot park, unless there are no timer actions, before it returns and sets the state machine to INACTIVE.
this.workingMemory.flushPropagations();
int returnedFireCount;
do {
    returnedFireCount = fireNextItem( agendaFilter, fireCount, fireLimit );
    fireCount += returnedFireCount;
    this.workingMemory.flushPropagations();
} while ( ( isFiring() && returnedFireCount != 0 && (fireLimit == -1 || fireCount < fireLimit) ) );

PropagationEntry head = tryHalt();
while (head != null) {
    fireCount += fireNextItem( agendaFilter, fireCount, fireLimit );
    SynchronizedPropagationList.flush(workingMemory, head);
    head = workingMemory.takeAllPropagations();
}

private PropagationEntry tryHalt() {
    synchronized (this) {
        PropagationEntry head = workingMemory.takeAllPropagations();
        if (head == null) {
            currentState = ExecutionState.INACTIVE;
            notify();
        } else if (currentState != ExecutionState.FORCE_HALTING) {
            currentState = ExecutionState.REST_HALTING;
        }
        return head;
    }
}

One of the key aspects here is the takeAll action. We can use this to atomically both check if there is work to do, and return that work within a sync point. But process the work outside of the sync point. So you can see it it will only finally halt, if takeAll returns null. Note the Timer thread would have to go through this sync point to add more work - ensuring there are no gaps.

There is a lot to take in here, and it's a bit of a brain dump. But I hope it proves useful to those wanting to understand how we are improving our engine, and how the prior work we did with the Phreak algorithm has enabled this.

Share/Bookmark

Thursday, June 18, 2015

Drools & jBPM get Dockerized

Docker is becoming a reference to build, ship and run container-based applications. It provides an standard, easy and automated way to deploy your applications.

Since latest 6.2.0.Final community release you can use Docker to deploy and run your Drools & jBPM applications in an easy and friendly way. Do not worry about operation system, environment and/or application server provisioning and deployments ... just use the applications!

The images are already available at Docker Hub:

Please refer to next "Drools & jBPM community Docker images" section for more information about what's contained in each image.

Why are these images helpful for me and my company?

To understand the advantages of using these Docker images, let's do a quick comparison with the deployment process for a manual installation of a Drools Workbench application.

If you do it by yourself:
  1. Install and prepare a Java runtime environment
  2. Download the workbench war (and other resources if necessary), from the official home page or from JBoss Nexus
  3. Download and prepare a JBoss WildFly server instance
  4. Configure the WildFly instance, including for example configuring the security subsystem etc.
  5. Deploy Drools into the WildFly instance
  6. Start the application server and run your Drools application
As you can notice, manual installation already takes quite a few steps.  While this process can be automated in some way (as the jbpm-installer for example does), some questions arise at this point ... What if I need a more complex environment? Are other colleagues using the same software versions and configuration? Can I replicate exact same environment? Could someone else easily run my local example easily during a customer demo? And if I need to deploy several identical runtime environments? What about removing my local installation from my computer? ...

Software containers & Docker are a possible solution and help providing an answer to some of these questions.

Both Drools & jBPM community Docker images include:
  • The OpenJDK JRE 1.7 environment 
  • A JBoss WildFly 8.1.0.Final application server
  • Our web-based applications (Drools Workbench, KIE server and/or jBPM Workbench) ready to run (configurations and deployments already present)
You don't have to worry about the Java environment, the application server, the web applications or configuration ... just run the application using a single command:

  docker run -p 8080:8080 -d --name drools-wb jboss/drools-workbench-showcase:6.2.0.Final

Once finished, just remove it:

   docker stop ...

At this point, you can customize, replicate and distribute the applications! Learn more about Docker, its advantages and how to use it at the offical site.

The environment you need

Do not worry about Java environments, application servers or database management systems, just install Docker:

   # For RHEL/Fedora based distributions:
   sudo yum -y install docker

More installation information at the official Docker documentation.

Are you using Windows? 

For windows users, in order to use Docker, you have to install Boot2Docker. It provides a Linux basic environment where Docker can run. Please refer to the official  documentation for the Docker installation on Windows platforms.

You are ready to run!

Drools & jBPM community Docker images

For the 6.2.0.Final community release six Docker images have been released.  They can be categorized in two main groups: Base images provide the base software with no custom configurations. They are intended to be extended and customized by Docker users.   Showcase images provide applications that are ready to run out-of-the-box (including for example some standard configuration).  Just run and use it!  Ideal for demos or evaluations / getting started.
  • Base images 
    • Drools Workbench
    • KIE Execution Server
    • jBPM Workbench
       
  • Showcase images
    • Drools Workbench Showcase
    • KIE Execution Server Showcase
    • jBPM Workbench Showcase


Let's dive into a detailed description of each image in the following sections.


Drools Workbench

This image provides the standalone Drools web authoring and rules management application for version 6.2.0.Final.  It does not include any custom configuration, it just provides a clean Drools Workbench application running in JBoss WildFly 8.1.  The goal of this image is to provide the base software and allow users to extend it, and apply custom configurations and build custom images.

Fetch the image into your Docker host:

   docker pull jboss/drools-workbench:6.2.0.Final

Customize the image by creating your Dockerfiles:

   FROM jboss/drools-workbench:6.2.0.Final
   ...

Please refer to Appendix C for extending this image.

Run a Drools Workbench container:

docker run -p 8080:8080 -d --name drools-wb jboss/drools-workbench:6.2.0.Final

Navigate to your Drools Workbench at:

   http://localhost:8080/drools-wb # Linux users
   http://<boot2docker_ip>:8080/drools-wb # Windows users

Refer to Appendix A for more information about IP address and port bindings.

Drools Workbench Showcase

See it in Docker Hub

This image provides the standalone Drools web authoring and rules management application for version 6.2.0.Final plus security configuration and some examples.
Tip: This image inherits from the Drools Workbench one and adds custom configurations for WildFly security subsystem (security realms) and system properties for enabling the use of the examples repository. 
The goal for this image is to provide a ready to run Drools Workbench application: just pull, run and use the Workbench.

1. Pull the image:

  docker pull jboss/drools-workbench-showcase:6.2.0.Final

2. Run the image:

  docker run -p 8080:8080 -d --name drools-wb-showcase jboss/drools-workbench-showcase:6.2.0.Final

3. Navigate to the workbench at:

   http://localhost:8080/drools-wb # Linux users
   http://<boot2docker_ip>:8080/drools-wb # Windows users

Refer to Appendix A for more information about IP address and port bindings.

You can use admin/admin for default logging in - Refer to Appendix B for default users and roles included

KIE Execution server

This image provides the standalone rules execution component for version 6.2.0.Final, to handle rules via remote interfaces.
More information for the KIE Execution Server can be found at the official documentation.
This image does not include any custom configuration, it just provides a clean KIE Execution Server application running in JBoss WildFly 8.1.  The goal for this image is to provide the base software and let the users to extend it, and apply custom configurations and build custom images.

Fetch the image into your Docker host:

   docker pull jboss/kie-server:6.2.0.Final

Customize the image by creating your Dockerfiles:

   FROM jboss/kie-server:6.2.0.Final
   ...

Please refer to Appendix C for extending this image.
Run a KIE Execution Server container:

   docker run -p 8080:8080 -d --name kie-server jboss/kie-server:6.2.0.Final

The KIE Execution Server is located at:

   http://localhost:8080/kie-server # Linux users
   http://<boot2docker_ip>:8080/kie-server # Windows users


Refer to Appendix A for more information about IP address and port bindings.

Example: use the remote REST API to perform server requests :

 http://localhost:8080/kie-server/services/rest/server # Linux
 http://<boot2docker_ip>:8080/kie-server/services/rest/server # Win

KIE Execution Server Showcase

See it in Docker Hub

This image provides the standalone rules execution component version 6.2.0.Final to handle rules via remote interfaces plus a basic security configuration (include a default user and role).
More information for the KIE Execution Server can be found at the official documentation. 
Tip: This image inherits from the KIE Execution Server one and adds custom configuration for WildFly security subsystem (security realms).

The goal of this image is to provide a ready to run KIE Execution Server: just pull, run and use the remote services.

1. Pull the image:

   docker pull jboss/kie-server-showcase:6.2.0.Final

2. Run the image:

   docker run -p 8080:8080 -d --name kie-server-showcase jboss/kie-server-showcase:6.2.0.Final

3. The server is located at:

   http://localhost:8080/kie-server # Linux users
   http://<boot2docker_ip>:8080/kie-server # Windows users


    The REST API service is located at:
  
 http://localhost:8080/kie-server/services/rest/server # Linux  
 http://<boot2docker_ip>:8080/kie-server/services/rest/server # Win  

Refer to Appendix A for more information about IP address and port bindings.

You can use kie-server/kie-server for default logging - Refer to Appendix B for default users and roles included
  

jBPM Workbench


This image provides the standalone version 6.2.0.Final of the jBPM Workbench: web-based authoring and management of your processes.  It does not include any custom configuration, it just provides a clean jBPM Workbench application running in JBoss WildFly 8.1.  The goal of this image is to provide the base software and let the users to extend it, and apply custom configurations and build custom images.

Fetch the image into your Docker host:

   docker pull jboss/jbpm-workbench:6.2.0.Final

Customize the image by creating your Dockerfiles:

   FROM jboss/jbpm-workbench:6.2.0.Final
   ...

Please refer to Appendix C for extending this image.
Run a jBPM Workbench container:

   docker run -p 8080:8080 -d --name jbpm-wb jboss/jbpm-workbench:6.2.0.Final

Navigate to your jBPM Workbench at:

   http://localhost:8080/jbpm-console # Linux users
   http://<boot2docker_ip>:8080/jbpm-console # Windows users

Refer to Appendix A for more information about IP address and port bindings.

jBPM Workbench Showcase


This image provides the standalone version 6.2.0.Final of the jBPM Workbench: web-based authoring and management of your processes. It includes the security and persistence configurations and some examples too.
Tip: This image inherits from the jBPM Workbench one and adds custom configurations for WildFly security subsystem (security realms) and system properties for enabling the use of the examples repository. 
The goal of this image is to provide a ready to run jBPM Workbench application: just pull, run and use the Workbench:

1. Pull the image:

   docker pull jboss/jbpm-workbench-showcase:6.2.0.Final

2. Run the image:

   docker run -p 8080:8080 -d --name jbpm-wb-showcase jboss/jbpm-workbench-showcase:6.2.0.Final

3. Navigate into the workbench at:

   http://localhost:8080/jbpm-console # Linux users  
   http://<boot2docker_ip>:8080/jbpm-console # Windows users

Refer to Appendix A for more information about IP address and port bindings.

You can use admin/admin for default logging - Refer to Appendix B for default users and roles included

Appendix

 

Appendix A - IP address and ports bindings for Docker containers

 
Port bindings
By default, when using any of the Drools & jBPM Docker images, the port 8080 is exposed for the use of the HTTP connector. This port is not exposed to the Docker host by default, so in order to expose it and be able to navigate through the applications please read the following instructions.

The recommended use for running containers is specifying in the docker client the -p argument as:

  docker run -p 8080:8080 -d ....

Doing this way, the docker daemon binds the internal container's port 8080 to the Docker host machine's port 8080. So you can navigate into the applications at:

   http://<docker_host>:8080/jbpm-console
   http://<docker_host>:8080/kie-server
   http://<docker_host>:8080/drools-wb

If your Docker host machine's port 8080 is not available, run the containers with the -P command line argument. Docker binds the internal 8080 port to an available free exposed port in the Docker host, so in order to access the application you have to discover the bind port number.

To discover running container's ports type the following command:

   docker ps -a

This command will output the processes and the port mappings for each running container:

CONTAINERID  IMAGE             COMMAND  CREATED STATUS PORTS                     NAMES
2a55fb....   jboss/drools-w..  ...      ...     ..     0.0.0.0:49159->8080/tcp.. drools-wb
The PORTS column shows that the internal container's port 8080 is bound to port 49159 on the Docker host, so you can navigate into the applications at:

   http://<docker_host>:49159/jbpm-console
   http://<docker_host>:49159/kie-server
   http://<docker_host>:49159/drools-wb


Docker hostname & IP address
The Docker hostname or IP address have to be specified in order to navigate through the container's applications.

If you are running Docker in your localhost and using Linux based OS, it defaults to localhost:

   http://localhost:8080/jbpm-console
   http://localhost:8080/kie-server
   http://localhost:8080/drools-wb

If you are running Docker on another machine or in Windows environments, where Boot2Docker is required,  you have to specify the host name (if DNS available for it) or the IP address for it:

   http://192.168.1.156:8080/jbpm-console
   http://192.168.1.156:8080/kie-server
   http://192.168.1.156:8080/drools-wb

Appendix B - Default applications users & roles

The Showcase images Drools Workbench Showcase and jBPM Workbench Showcase include default users & roles:

Drools & jBPM Workbench Showcase roles
Role Description
admin The administrator
analyst The analyst
developer The developer
manager The manager
user The end user
kiemgmt KIE management user
Accounting Accounting role
PM Project manager role
HR Human resources role
sales Sales role
IT IT role

Drools & jBPM Workbench Showcase users
Username Password Roles
admin admin admin,analyst,kiemgmt
krisv krisv admin,analyst
john john analyst,Accounting,PM
mary mary analyst,HR
sales-rep sales-rep analyst,sales
katy katy analyst,HR
jack jack analyst,IT
salaboy salaboy admin,analyst,IT,HR,Accounting

For KIE Execution Server Showcase there is a single user and role:
Username Password Roles
kie-server kie-server kie-server

Appendix C - Extending base images

The Base images are intended to be inherited from, for adding your custom configurations or deployments.

In order to extend the images, the Dockerfile must start with: 

    FROM jboss/drools-workbench:6.2.0.Final 
    FROM jboss/kie-server:6.2.0.Final
    FROM jboss/jbpm-workbench:6.2.0.Final

At this point, custom configurations and deployments can be added. Some notes:
  • JBoss WildFly is located at the path given by $JBOSS_HOME environment variable
  • $JBOSS_HOME points to /opt/jboss/wildfly/
  • Applications are using the server in standalone mode:
    • Configurations located at $JBOSS_HOME/standalone/configuration/
    • Configuration files for the standalone-full profile are used
    • Deployments are located at $JBOSS_HOME/standalone/deployments/
You can find more information at each official image page at Docker Hub:


Share/Bookmark

Wednesday, June 17, 2015

Drools & jBPM meeting space needed in Barcelona

We are looking to try and organise a team meeting in Barcelona, towards the end of this year. We have limited to no budget for meeting space :( So I thought I'd see if anyone out there would like to volunteer this space - in return you'll have all the core Drools and jBPM developers on hand for a week :) We need a large room suitable for around 25-30 people sitting at tables, and then one or two break out rooms with around 10 people or so per room.

Mark

Share/Bookmark