Wednesday, May 27, 2015

More Eclipse Tooling enhancements

The biggest complaint from our customers about the eclipse tooling for B*MS is that the cost of entry is too high; not only must a user be familiar with several different technologies, such as Git, maven, REST services and how these technologies are exposed by the eclipse tooling, but s/he must also understand the various Drools and jBPM configuration and definition files. Since there are only a few user-friendly/graphical editors that hide underlying file details, the user must become familiar with most of these file formats, and where in the Project or Repository hierarchy the file resides.

One of the enhancements I have been working on will hopefully ease some of this burden by providing a "navigator" similar to the Eclipse Project Explorer, but designed specifically for Drools/jBPM projects (see below).
At the root of this tree viewer are the app servers that have Drools/jBPM installed. Servers are managed (start, stop, debug) from the WST Servers view. At the next level is the Organizational Unit, then Repositories and finally Projects. Essentially, this viewer mimics the web console with the addition of multiple servers.

The tree structure is cached whenever a connection to the server can be established. This allows the view to be used in "offline" mode if the server is down or network connection is unavailable. When the server is available again, the viewer synchronizes its cache with the server.

Repositories are automatically cloned, and Projects are imported as they are requested by the user with a context menu action.

I'm still in the design/experimenting phase right now, so if there's a feature you'd like to see, or if you have suggestions for improving this interface please post your comments here.

You can also see a related post, showing my work on improving the wizards and runtime generation and configuration.

Improved Drools & jBPM Eclipse wizard

Bob has been working on improving our Drools & jBPM Eclipse wizards.

  • The user no longer needs to create runtimes. They can now be created automatically on the fly by the new project wizard.
  • The project wizard will now list examples from the github repository and allow them to be selected and dowloaded as part of the wizard.
You cans see a video for this here:

Currently all the downloadable examples are jBPM, we still need to migrate the Drools examples over to this repository format.

Mark

Wednesday, May 20, 2015

A Comparative Study of Correlation Engines for Security Event Management

This just paper came up on my google alerts, you can download the full text from ResearchGate.
"A Comparative Study of Correlation Engines for Security Event Management"

 It's an academic paper, published in the peer reviewed journal.
"10th International Conference on Cyber Warfare and Security (ICCWS-2015)"

Th paper is evaluating the correlation performance for large rule sets and large data sets in different open source engines. I was very pleased to see how well Drools scaled at the top end. I'll quote this from the conclusion and copy the results charts.
"As for the comparison study, it must be said that if the sole criteria was raw performance Drools would be considered the best correlation engine, for several reasons: its consistent behaviour and superior performance in the most demanding test cases."

In Table 2 (first image) we scale form 200 rules to 500 rules, with 1mil events with almost no speed loss - 67s vs 70s.

In Table 1 (second image) our throughput increases as the event sets become much larger.

I suspect the reason why our performance is less for for the lower rule and event set numbers, is due to the engine initialisation time for all the functionality we provide and for all the indexing we do. As the matching time becomes large enough, due to larger rule and data sets, this startup time becomes much less significant on the over all figure.




Tuesday, May 12, 2015

Validation and Verification for Decision Tables

The decision tables are getting even more improvements than the UI work Michael has been working on.
Zooming and Panning between Multiple Huge Interconnected Decision Tables
Cell Merging, Collapsing and Sorting with Multiple Large Interconnected Decision Tables

I am currently working on improving the validation and verification of the decision tables. Making it real time and improving the existing V&V checks.

Validation and verification are used to determine if the given rules are complete and to look for any bugs in the dtable authors logic. More about this subject.

Features coming in the next release


Real time Verification & Validation

Previously the user had to press a button to know if the dtable was valid or not. Now the editor does the check in real time, removing the need to constantly hit the Validate-button. This also makes the V&V faster, since there is no need to validate the entire table, just check how the change of a field affected the rest of the table.




Finding Redundancy 

To put it simple: two rows that are equal are redundant, but redundancy can be more complicated. The longer explanation is: redundancy exists when two rows do the same actions when they are given the same set of facts.

Redundancy might not be a problem if the redundant rules are setting a value on an existing fact, this just sets the value twice. Problems occur when the two rules increase a counter or add more facts into the working memory. In both cases the other row is not needed.




 

 

Finding Subsumption

Subsumption exists when one row does the same thing as another, with a sub set of the values/facts of another rule. In the simple example below I have a case where a fact that has the max deposit below 2000 fires both rows.

The problems with subsumption are similar to the case with redundancy.






Finding Conflicts

Conflicts can exists either on a single row or between rows.
A single row conflict prevent the row actions from ever being executed.

Single row conflict - second row checks that amount is greater than 10000 and below 1






Conflict between two rows exists when the conditions of two rules are met with a same set of facts, but the actions set existing fact fields to  different values. The conditions might be redundant or just subsumptant.

The problem here is, how do we know what action is made last? In the example below: Will the rate be set to 2 or 4 in the end? Without going into the details, the end result may be different on each run and with each software version. 
Two conflicting rows - both rows change the same fact to a different value

 

Reporting Missing Columns

In some cases, usually by accident, the user can delete all the condition or action columns.

When the conditions are removed all the actions are executed and when the actions columns are missing the rows do nothing.
The action columns are missing
The condition columns are missing










What to expect in the future releases?


Better reporting

As seen on the examples above. Reporting the issues is currently poor.
The report should let the user know how serious the issue is, why it is happening and how to fix it.

The different issue levels will be:
  • Error - Serious fault. It is clear that the author is doing something wrong. Conflicts are a good example of errors.
  • Warning - These are most likely serious faults. They do not prevent the dtable from working, but need to be double checked by the dtable author. Redundant/subsumptant rules for example, maybe the actions need to happen twice in some cases.
  • Info - The author might not want to have any conditions in the dtable. If the conditions are missing each action gets executed. This can be used to insert a set of facts into the working memory. Still it is good to inform that the conditions might have been deleted by accident.  

 

Finding Deficiency

Deficiency gives the same kind of trouble that conflicts did. The conditions are too loose and the actions conflict.

For example:
If the loan amount is less than 2000 we do not accept it.
If the person has a job we approve the loan.
The problem is, we might have people with jobs asking for loans that are under 2000. Sometimes they get them, sometimes they do not.


 

Finding Missing Ranges and Rows

Is the table complete? In our previous examples we used the dtable to see if the loan application gets approved. One row in the dtable should always activate, no matter how the user fills out his loan application. Either rejecting or approving the loan or else the applicant does not get a loan decision.
The goal of the V&V tool will be to find these gaps for the dtable author.

 

Finding Cycles

The actions can insert new facts and the conditions trigger the actions when new facts are inserted. This can cause an infinite number of activations.
This issue is a common mistake that the goal is to pick it up in the authoring phase with the V&V tool.

Monday, April 27, 2015

Cell Merging, Collapsing and Sorting with Multiple Large Interconnected Decision Tables

Last month I showed you videos for our proof of concept work, using Lienzo, to see how viable HTML5 canvas is for multiple large interconnected decision tables.

Michael's made more progress adding cell merging and collapsing as well are sorting columns. All still working on truly massive interconnected tables.

We plan to make the generic core of this work available as a Lienzo grid component in the future. Although we still need to figure out different data types for cells and how to do seamless in-cell editing rather than a popup.

(click to turn on 720p HD and full screen)

Domain Extensions for Data Modeller

Walter is working on adding domain extensions to the Data Modeller. This will allow different domains to augment the model - such as custom annotations for JPA or OptaPlanner. Each domain is pluggable via a "facet" extension system. Currently, as a temporary solution, each domain extension is added as an item in the toolbar, but this will change soon. In parallel to this Eder will be working on something similar to Intellij's Tool Windows for side bars. Once that is ready those domain extensions plugged in as facets and exposed via this tool window capability. Here is a video showing JPA and it's annotations being used with the Data Modeller.

(Click to  turn on 720p HD and full screen)

Friday, March 13, 2015

Reactive Incremental Graph Reasoning with Drools

Today Mario got a first working version for incremental reactive graphs with Drools. This means people no longer need to flatten their models to a purely relational representation to get reactivity. It provides a hybrid language and engine for both relational and graph based reasoning. To differentiate between relational joins and reference traversal a new XPath-like was introduced, that can be used inside of patterns. Like XPath it supports collection iteration.

Here is a simple example, that finds all men in the working memory:
Man( $toy: /wife/children[age > 10]/toys )

For each man it navigates the wife reference then the children reference; the children reference is a list. For each child in the list that is over ten it will navigate to its toy's list. With the XPath notation if the leaf property is collection it will iterate it, and the variable binds to each iteration value. If there are two children over the age of 10, who have 3 toys each, it would execute 6 times.

As it traverses each reference a hook is injected to support incremental reactivity. If a new child is added or removed, or if an age changes, it will propagate the incremental changes. The incremental nature means these hooks are added and removed as needed, which keeps it efficient and light.

You can follow some of the unit tests here:
https://github.com/mariofusco/drools/blob/xpath/drools-compiler/src/test/java/org/drools/compiler/xpath/XpathTest.java

It's still very early pre-alpha stuff, but I think this is exciting stuff.

Tuesday, March 10, 2015

UF Dashbuilder - Activity monitoring in jBPM

syndicated from http://dashbuilder.blogspot.com.es/2015/03/uf-dashbuilder-in-jbpm-for-activity.html

Last week, the jBPM team announced the 6.2.0.Final release (announcement here). In this release (like in previous ones) you can author processes, rules, data models, forms and all the assets of a BPM project. You can also create or clone existing projects from remote GIT repositories and group such repositories into different organizational units. Everything can be done from the jBPM authoring console (aka KIE Workbench), a unified UI built using the Uberfire framework & GWT.

   In this latest release, they have also added a new perspective to monitor the activity of the source GIT repositories and organizational units managed by the tooling (see screenshot below). The perspective itself it's just a dashboard displaying several indicators about the commit activity. From the dashboard controls it is possible to:

  • Show the overall activity on our repositories
  • Select a single organizational unit or repository
  • List the top contributors
  • Show only the activity for an specific time frame

  In this video you can see the dashboard in action (do not forget to select HD).

Contributors Perspective

  Organizational units can be managed from the menu Authoring>Administration>Organizational Units. Every time an organizational unit is added or removed the dashboard is updated.

Administration - Organizational Units 

   Likewise, from the Authoring>Administration>Repositories view we can create, clone or delete repositories. The dashboard will always feed from the list of repositories available.

Administration - Repositories



   As shown, activity monitoring in jBPM can be applied to not only to the processes business domain but also to the authoring lifecycle in order the get a detailed view of the ongoing development activities.

How it's made


The following diagram shows the overall design of the dashboard architecture. Components in grey are platform components, blue ones are specific to the contributors dashboard.

Contributors dashboard architecture

  These are the steps the backend components take to build the contributors data set:

  • The ContributorsManager asks the platform services for the set of available org. units & repos. 
  • Once it has such information, it builds a data set containing the commit activity.
  • The contributors dataset is registered into the Dashbuilder's DataSetManager.

   All the steps above are executed on application start up time. Once running, the ContributorsManager also receives notifications form the platform services about any changes on the org. units & repositories registered, so that the contributors data set is synced up accordingly. 


   From the UI perspective, the jBPM's contributors dashboard is an example of hard-coded dashboard built using the Dashbuilder Displayer API which was introduced in this blog entry. The ContributorsDashboard component is just a GWT composite widget containing several Displayer instances feeding from the contributors data set.

   (The source code of the contributors perspective can be found here)

    This has been a good example of how to leverage the Dashbuilder technology to build activity monitoring dashboards. In the future, we plan for applying the technology in other areas within jBPM, like, for instance, an improved version of the jBPM process dashboard. We will keep you posted!

Monday, March 09, 2015

Zooming and Panning between Multiple Huge Interconnected Decision Tables

Michael has started the work on revamping our web based decision tables. We've been experimenting with HMTL5 canvas with great results, using the excellent Lienzo tool. First we needed to ensure we could scale to really large decision tables, with thousands of rows. Secondly we wanted to be able to pan and zoom between related or interconnected decision tables. We'll be working towards Decision Model and Notation support, that allows networked diagrams of Decision Tables.

You can watch the video here, don't forget to select HD:
https://www.youtube.com/watch?v=WgZTdfLis0Q

Notice in the video that while you can manually pan and zoom it also has links between tables. When you select the link it animates the pan and zoom to the linked location. 25s to 47s in is showing  that we can have really large number of rows and keep excellent performance, while 55s is showing the pan speed with these large tables. Initially the example starts with 50% of cells populated, at 1m in we change that to 100% populated and demonstrate that we still have excellent performance.




jBPM 6.2.0.Final released

The bits for the jBPM 6.2 release are now available for you to download and try out !  

Version 6.2 comes with a few new features and a lot of bug fixes !  New features include a.o. EJB, (improved) OSGi and Camel endpoints support, a new asset management feature (to introduce a development and release branch and promote assets between both), social profiles and feeds and the ability to extend the workbench with your own plugins!

More details below, but if you want to jump right in:

Downloads
Documentation
Release Notes

Ready to give it a try but not sure how to start?  Take a look at the jbpm-installer chapter.

jBPM 6.2 is released alongside Drools (for business rules) and Optaplanner (for planning and constraint solving), check out the new features in the Drools release blog, including a brand new rules execution server and the Optaplanner release blog as well.

A big thank you to everyone who contributed to this release!

Some highlights from the release notes.

Core services

  • EJB: the jBPM execution server (that is for example embedded in our web-based workbench) now also comes with an EJB interface.  A refactoring of the underlying jbpm-services now makes the execution services accessible using pure Java, CDI, EJB and Spring. Remote interfaces using REST and JMS are still available as well of course !  A lot more details are described in Maciej's blog here.
  • Deployments (defining which versions of which projects are currently active in the execution server) are now by default stored in the database.  This greatly simplifies the architecture in a clustered environment in case you are only using our runtime side of our web tooling (for example by having dedicated execution servers in production).
  • Our asynchronous job executor has improved support for requeuing failed jobs and for recurring jobs (e.g. daily tasks).
  • OSGi: Full core engine functionality is now available on top of OSGi.  A significant number of additional jars (including for example the human task service, the runtime managers, full persistence, etc.) were "OSGi-fied". Specific extensions and tests showing it in action are available for Apache Karaf and Aries Blueprint (in the droolsjbpm-integration repository).
  • Camel endpoint URIs: A new out-of-the-box service task has been implemented for using Apache Camel to connect a process to the outside world using some of the numerous Camel endpoint URIs. The service task allows you to for example specify how to pass data to an FTP endpoint by configuring properties such as hostname, port, username, payload, etc. for some common endpoints like (S)FTP, File, JMS, XSLT, etc. but you can use virtually any of the available endpoints by defining the URI yourself (http://camel.apache.org/uris.html).

Workbench
  • Form Modeler comes with improved support for adding custom logic to your forms using JavaScript on changes, and support for configurable ComboBox and RadioGroup fields, and simple List types.
  • Asset management: It is now possible to make a repository a "managed repository".  This allows you to split up a repository in multiple branches, one for doing development and on for releasing.  Users can then request various assets to be promoted to the resource branch when ready.  This promotion process, and the linked build and deploy processes, are defined using a BPMN2 process as well and include approval and build tasks.  Check the documentation for more details.

  • Social features, like user profiles (including gravatar pictures), and various event feeds like the most recent assets you worked on, on recent changes by other users.


  • Contributors perspective is a new out-of-the-box report (using the new dashbuilder technology) that gives high-level insight in who is changing what in your repositories.
  • Pluggable workbench:  you can now extend the workbench with your own views, menus, etc. using workbench plugins. Available features includes creation of perspectives via a programmable or a drag and drop interface, create new screens, editors, splashscreens and dynamic menus.