Friday, November 10, 2017

Building Business Applications with DMN and BPMN

A couple weeks ago our own Matteo Mortari delivered a joint presentation and live demo with Denis Gagné from Trisotech at the BPM.com virtual event.

During the presentation, Matteo live demo'd a BPMN process and a couple DMN decision models created using the Trisotech tooling and exported to Red Hat BPM Suite for seamless execution.

Please note that no glue code was necessary for this demo. The BPMN process and the DMN models are natively executed in the platform, no Java knowledge needed.

Enough talking, hit play to watch the presentation... :)



Share/Bookmark

Thursday, October 12, 2017

5 Pillars of a Successful Java Web Application

Last week, Alex Porcelli and I had the opportunity to present at JavaOne San Francisco 2017 two talks related to our work: "5 Pillars of a Successful Java Web Application” and The Hidden Secret of Java Open Source Projects.

It was great to share our cumulative experience over the years building the workbench and the web tooling for the Drools and jBPM platform and both talks had great attendance (250+ people in the room).


In this series of posts, we’ll detail our "5 Pillars of a Successful Java Web Application”, trying to give you an overview of our research and also a taste of participating in a great event like Java One.
There are a lot of challenges related to building and architecting a web application, especially if you want to keep your codebase updated with modern techniques without throwing away a lot of your code every two years in favor of the latest trendy JS framework.
In our team we are able to successfully keep a 7+ year old Java application up-to-date, combining modern techniques with a legacy codebase of more than 1 million LOC, with an agile, sustainable, and evolutionary web approach.
More than just choosing and applying any web framework as the foundation of our web application, we based our web application architecture on 5 architectural pillars that proved crucial for our platform’s success. Let's talk about them:

1st Pillar: Large Scale Applications

The first pillar is that every web application architecture should be concerned about the potential of becoming a long-lived and mission-critical application, or in other words, a large-scale application. Even if your web application is not exactly big like ours (1mi+ lines of web code, 150 sub-projects, +7 years old) you should be concerned about the possibility that your small web app will become a big and important codebase for your business. What if your startup becomes an overnight success? What if your enterprise application needs to integrate with several external systems?
Every web application should be built as a large-scale application because it is part of a distributed system and it is hard to anticipate what will happen to your application and company in two to five years.
And for us, a critical tool for building these kinds of distributed and large-scale applications throughout the years has been static typing.

Static Typing

The debate of static vs. dynamic typing is very controversial. People who advocate in favor of dynamic typing usually argue that it makes the developer's job easier. This is true for certain problems.
However, static typing and a strong type system, among other advantages, simplify identifying errors that can generate failures in production and, especially for large-scale systems, make refactoring more effective.
Every application demands constant refactoring and cleaning. It’s a natural need. For large-scale ones, with codebases spread across multiple modules/projects, this task is even more complex. The confidence when refactoring is related to two factors: test coverage and the tooling that only a static type system is able to provide.
For instance, we need a static type system in order to find all usages of a method, in order to extract classes, and most importantly to figure out at compile time if we accidentally broke something.
But we are in web development and JavaScript is the language of the web. How can we have static typing in order to refactor effectively in the browser?

Using a transpiler

A transpiler is a type of compiler that takes the source code of a program written in one programming language as its input and produces equivalent source code in another programming language.
This is a well-known Computer Science problem and there are a lot of transpilers that output JavaScript. In a sense, JavaScript is the assembly of the web: the common ground across all the web ecosystems. We, as engineers, need to figure out what is the best approach to deal with JavaScript’s dynamic nature.
A Java transpiler, for instance, takes the Java code and transpiles it to JavaScript at compile time. So we have all the advantages of a statically-typed language, and its tooling, targeting the browser.

Java-to-JavaScript Transpilation

The transpiler that we use in our architecture, is GWT. This choice is a bit controversial, especially because the GWT framework was launched in 2006, when the web was a very different place.
But keep in mind that every piece of technology has its own good parts and bad parts. For sure there are some bad parts in GWT (like the Swing Style Widgets, multiple permutations per browser/language), but keep in mind that for our architecture what we are trying to achieve is static typing on the web, and for this purpose the GWT compiler is amazing.
Our group is part of GWT steering committee, and the next generation of GWT is all about JUST these good parts. Basically removing or decoupling the early 2000 legacy and keeping only the good parts. In our opinion the best parts of GWT are:
  • Java to JavaScript transpiler: extreme JavaScript performance due to compiling optimizations and static typing in the web;
  • java.* emulation: excellent emulation of the main java libraries, providing runtime behavior/consistency;
  • JS Interop: almost transparent interoperability between Java <-> Javascript. This is a key aspect of the next generation of GWT and the Drools/jBPM platform: embrace and interop (two way) with JS ecosystem.

Google is currently working on a new transpiler called J2CL (short for Java-to-Closure, using the Google Closure Compiler) that will be the compiler used in GWT 3, the next major GWT release. The J2CL transpiler has a different architecture and scope, allowing it to overcome many of the disadvantages of the previous GWT 2 compiler.

Whereas the GWT 2 compiler must load the entire AST of all sources (including dependencies), J2CL is not a monolithic compiler. Much like javac, it is able to individually compile source files, using class files to resolve external dependencies, leaving greater potential for incremental compilation.
These three good parts are great and in our opinion, you should really consider using GWT as a transpiler in your web applications. But keep in mind that the most important point here is that GWT is just our first pillar implementation. You can consider using other transpilers like Typescript, Dart, Elm, ScalaJS, PureScript, or TeaVM.
The key point is that every web application should be handled as a large-scale application, and every large-scale application should be concerned about effective refactoring. The best way to achieve this is using statically-typed languages.
This is the first of three posts about our 5 pillars of successful web applications. Stay tuned for the next ones.

[I would like to thank Max Barkley and Alexandre Porcelli for kindly reviewing this article before publication, contribute with the final text and provided great feedback.]



Share/Bookmark

Wednesday, September 20, 2017

Watch all the sessions from Red Hat Drools Day LIVE from your desktop or mobile, Sept 26th

We will be streaming all the sessions of the Drools Day in NYC, on Sep 26th, live!

Use the following link to watch:

https://www.facebook.com/RedHatDeveloperProgram/videos/1421743381254509/

Or watch it here:




Share/Bookmark

Wednesday, September 13, 2017

Drools, jBPM and Optaplanner Day: September 26 / 28, 2017 (NY / Washington)

Red Hat is organizing a Drools, jBPM and Optaplanner Day in New York and Washington DC later this year to show how business experts and citizen developers can use business processes, decisions and other models to develop modern business applications.
This free full day event will focus on some key aspects and several of the community experts will be there to showcase some of the more recent enhancements, for example:
  • Using the DMN standard (Decision Model and Notation) to define and execute decisions
  • Moving from traditional business processes to more flexible and dynamic case management
  • The rise of cloud for modeling, execution and monitoring
IT executives, architects, software developers, and business analysts who want to learn about the latest open source, low-code application development technologies.

Detailed agenda and list of speakers can be found on each of the event pages.

Places are limited, so make sure to register asap !

Share/Bookmark

Wednesday, September 06, 2017

Is Optimization AI or OR?

With the renewed interest in AI the same conversations are starting to come up again, about what is or isn't AI.  My recent discussion was on whether optimisation products, such as OptaPlanner, are considered AI as some considered it more Operations Research (OR). For some background, OptaPlanner started out as a Tabu Solver implementation, but has since added other techniques like Simulated Annealing.

Although I'd like to add that no single technique is AI, they are all tools and techniques that are quite typically used together in a blended, hybrid or integrated AI solution. So it's the right tool or tools for the job.

The answer is that Optimisation is both an AI and an OR problem. It is a technique used and researched by both groups,  the two different disciplines tend to take different approaches to the problem, having differing use cases and have historically used different techniques, with a lot of cross pollination from both sides.

I'll start with a consumer oriented answer to the question. StaffJoy has a nice blog article on the overlap of OR and AI, and I'll quote from that below:
https://blog.staffjoy.com/how-operations-research-and-artificial-intelligence-overlap-b128a3efee2e
"Startups are using OR techniques in products like OnFleet, Instacart, and Lyft Line. However, when similar techniques are being exposed externally as services, they are often described as AI — e.g. x.ai, Atomwise, and Sentient. Very few companies describe algorithms that they sell as optimization (with the exception of SigOpt) because the end goal of customers is automating decisions. With StaffJoy, we have found that customers better understand our product when we describe it as an “artificial intelligence” tool rather than an “optimization” or “operations” tool. We think that this is because customers care more about what a product achieves, rather than the means it uses to achieve it."

In short consumers do not see the difference between OR and AI, when applied to real world problems and it is commonly marketed as AI.

I'll go a little more technical now, to further demonstrate it's more than just marketing - as that side is only touched on in the above blog post.

While the two groups (OR and AI) may have once been distinct, it's been well established that the OR and AI groups overlap in this space and have collaborated for years. Glover (1986) states them as "the recent remarriage of two disciplines that were once united, having issued from a common origin, but which became separated" - see final paper link at end.

A cursory google with terms "operations research" and "artificial intelligence" will more than prove this. Some techniques, like Linear Programming, are strongly on the OR side, others like Local Search (which OptaPlanner falls under) are shared. Optimisation, and local search (along with other techniques), is a core fundamental taught in every AI course without fail, and will be covered in every general AI book, used in schools - such as "AI:  A Modern Approach"- see chapter 4, page 120
http://aima.cs.berkeley.edu/contents.html

The book "Artificial Intelligence Methods and Applications" also makes it clear the two (OR and AI) are linked:
"Local search, or local optimisation, is one of the primitive forms of continuous optimisation in a discrete problem space. It was one of the early techniques proposed during the mid sixties to cope with the overwhelming computational intractability of NP-hard combinatorial optimisation problems. Unlike continuous optimisation techniques, local search has often been used in AI research and has established a strong link between AI and the operational research area."
https://books.google.co.uk/books?id=0a_j0R0qh1EC&pg=PA67&lpg=PA67&dq=%22local+search%22+operations+research+artificial+intelligence&source=bl&ots=h2jGquBm4d&sig=2V7CKRIs3ZL-NKzqL3Dnkx33_NI&hl=en&sa=X&redir_esc=y#v=onepage&q=%22local%20search%22%20operations%20research%20artificial%20intelligence&f=false

Lastly I'll quote directly from the original Tabu Solver paper "These developments may be usefully viewed as a synthesis of the perspectives of operations research and artificial intelligence... ... The foundation for this prediction derives, perhaps surprisingly, from the recent remarriage of two disciplines that were once united, having issued from a common origin, but which became separated and maintained only loose ties for several decades: operations research and artificial intelligence. This renewed reunion is highlighting limitations in the frameworks of each (as commonly applied, in contrast to advocated), and promises fertile elaborations to the strategies each has believed fruitful or approaching combinatorial complexity." Glover (1986). Note the paper is from "The Center of Applied Artificial Intelligence". 
http://leeds-faculty.colorado.edu/glover/TS%20-%20Future%20Paths%20for%20Integer%20Programming.pdf

So I hope that clears that up - AI is a very broad church :)



Share/Bookmark

Friday, August 18, 2017

Capture your decisions with DMN

Donato Marrazzo published a very nice and comprehensive introduction to DMN with Drools and Trisotech.

It is worth a read for anyone interested in DMN:

Capture your decisions with DMN

Happy Drooling!


Share/Bookmark

Wednesday, August 09, 2017

Talking about Rule Engines at Software Engineering Radio

I had the pleasure of talking to Robert Blumen, at Software Engineering Radio, about Drools and Rule Engines in general.

http://www.se-radio.net/2017/08/se-radio-episode-299-edson-tirelli-on-rules-engines/

If you don't know this podcast, I highly recommend their previous episodes as well. Very informative, technically oriented podcast.

Hope you enjoy,
Edson
Share/Bookmark

Tuesday, August 08, 2017

Drools Canonical Model - Pure Java Rules

Rule engines, like Drools, typically  make use of a custom languages to define a set of rules. For example, he Drools compiler translates a drl file to an internal representation (the KiePackages) that is subsequently used to generate the ReteOO/Phreak network that will perform the rules evaluation.

This internal representation was never really intended to be generated or consumed by end users. This complication makes it difficult to write rules programmatically and the suggestion is instead to generate text rules at the DRL level.  This means that drl itself is currently the only practical formal notation to define a set of rules in Drools.

Drools internals were developed with several assumptions at the time, that are no longer true or desirable. Prior to Java 8, perm gen was a concern, so various solutions were utilized to address this - such as MVEL for reflective based evaluation. Java 8 now puts code on the heap, so this is no longer necessary.  A the engine level it also inspected and produced indexes, which tied it to the expressions produced by DRL - this makes polyglot impractical. Lastly it liberally uses classloaders and reflection, which makes it difficult to transpiler for execution on different environments.

An engine independent rule model


To overcome this limitation and offer the possibility of programmatically define a set of rule in pure Java we developed a model aimed to provide a canonical representation of a rule set and a fluent DSL to conveniently create an instance of this model. The model itself is totally independent from Drools and could in theory be re-used by other engines, It also introduces layers that fully separates the engine from having to be aware of any language. For example it will not inspect and generate indexes, instead it expects to be provided those indexes, from the layers above. Another advantage is it now means Drools has a developer friendly view of what it's executing, as it's all just pure low-level pojo rules.

This model, other than giving to all Java developers a clean way to write rules in plain Java, will also enable our team to experiment with new features faster, freeing us from the burden of also implementing the corresponding parser and compiler parts that integrate the new feature with the drl notation.
Let's see a practical example of the model of a rule defined using the before mentioned DSL:
 
Rule rule = rule( "Persons older than Mark" )
    .view(
        expr("exprA", markV, p -> p.getName().equals("Mark"))
            .indexedBy( String.class, ConstraintType.EQUAL, Person::getName, "Mark" )
            .reactOn( "name", "age" ), 
        expr("exprB", olderV, p -> !p.getName().equals("Mark"))
            .indexedBy( String.class, ConstraintType.NOT_EQUAL, Person::getName, "Mark" )
            .reactOn( "name" ),
        expr("exprC", olderV, markV, (p1, p2) -> p1.getAge() > p2.getAge())
            .indexedBy( int.class, ConstraintType.GREATER_THAN, Person::getAge, Person::getAge )
            .reactOn( "age" )
    )
    .then(on(olderV, markV)
         .execute((p1, p2) -> System.out.println( p1.getName() + " is older than " + p2.getName())));


This is the equivalent of the following rule expressed in drl:
 
rule "Persons older than Mark" when
    $p1 : Person(name == \"Mark\")
    $p2 : Person(name != \"Mark\", age > $p1.age)
then
    System.out.println($p2.getName() + \" is older than \" + $p1.getName());
end

It is evident that the DSL version is much more verbose and requires to specify many more details that conversely are inferred by the drools compiler when it parses the rule written in drl.
As anticipated this has been done on purpose because the model has to explicitly contain all the information required to generate the section of ReteOO/Phreak network intended to evaluate that rule, without the need of performing any bytecode inspection or using any other complex, brittle and non-portable introspection technique. In those simple examples the index contains the same logic as the expr, as per DRL expressions can contain more than just the index. The index would be inferred and implicitly added by the upper language layers.
To clarify this aspect let's analyze a single LHS constraint in a bit more detail:

expr("exprA",                                                                 [1] 
     markV,                                                                   [2]
     p -> p.getName().equals("Mark") )                                        [3]  
    .indexedBy( String.class, ConstraintType.EQUAL, Person::getName, "Mark" ) [4]
    .reactOn( "name", "age" )                                                 [5]


In this statement you can notice the following parts
[1] This is a label for the constraint used to determine its identity. Two identical constraint must have the same label and two different ones must have different labels in order to let the engine properly implement the node sharing where possible. This label can be optionally omitted and in this case it will be generated from an automatic introspection of the bytecode of the lambda expression implementing the constraint. However, as anticipated, it's preferable to avoid any introspection mechanism and then to explicitly provide a constraint label whenever is possible.
[2] This is the Variable defined before creating the rule and used to bind an actual fact with the formal parameter of the lambda expression used to evaluate a condition on it. It is the equivalent of the variable declaration in rule written with the drl notation.
[3] The lambda expression performing the constraint evaluation.
[4] The specification of the type of this constraint and how it has to be indexed by the engine.
[5] The name of the properties of the Person object for which this constraint has to be reactive.

Building and executing the model


You can programmatically add this and other rules of your rule base to a model:

Model model = new ModelImpl().addRule( rule );


Once you have this model, which again is totally independent from Drools algorithms and data structures, you can use a second project, this time dependent both on Drools and on the model itself, to create a KieBase out of it.

KieBase kieBase = KieBaseBuilder.createKieBaseFromModel( model );


What this builder internally does is recreating the KiePackages out of the model and then building them using the existing drools compiler. The resulting KieBase is identical to the one you may obtain by compiling the corresponding drl and then you can use it exactly in the same way, creating a KieSession out of it and populating it with the facts of your domain. Here you can find more test cases showing the Drools features currently supported by the model and the DSL constructs allowing to use them, while here you can find a more complete example.

Embedding the executable model inside the kjar


At the moment a kjar contains some pregenerated classes implementing the constraints and the consequences. However all of the drl files need to be parsed and compiled from scratch, making limited savings on building it from source only.The executable model will be also useful to speed this process up. Embedding the classes implementing the model of the rule base inside the kjar, makes possible to quickly recreate the KiePackages, instead of having to restart the whole compilation process from the drl files. A proof concept of how this could work is available here.

DRL compiler to Executable Model


We are working on a new compiler that will take existing DRL and directly output the executable model, which is stored in the kjar. This will result in much faster loading and building times. We are also looking into other ways to speed this process up, by pre-caching more information in the kjar, which can be determined during the initial kjar build.

Pojo, Polyglot and DRLX Rules


The executable model is low level and because you have to provide all the information it needs, this makes it a little verbose. This is done by design, to support innovation in the language level. While it is technically still pojo-rules, it is not desirable for everyday use. In the near future we will work on a less verbose pojo rules layer, that will use an annotation processor to inject things like indexes and reactOn. We also hope that it will result in several rule languages, not just one - scala rules, closure rules, mini-dsl rules etc. Nor is it necessary for a language to expose all of the engine capabilities, people can write mini-dsls exposing the subset.

Finally we are also working on a next generation DRL language (DRLX), that will be a superset of Java. This is still in the design phases and will not be available for some time, but we will publish some of the proposals for this, once an early draft spec is ready.



Share/Bookmark

Wednesday, August 02, 2017

Drools, jBPM and Optaplanner Day: September 26 / 28, 2017 (NY / Washington)

Red Hat is organizing a Drools, jBPM and Optaplanner Day in New York and Washington DC later this year to show how business experts and citizen developers can use business processes, decisions and other models to develop modern business applications.
This free full day event will focus on some key aspects and several of the community experts will be there to showcase some of the more recent enhancements, for example:
  • Using the DMN standard (Decision Model and Notation) to define and execute decisions
  • Moving from traditional business processes to more flexible and dynamic case management
  • The rise of cloud for modeling, execution and monitoring
IT executives, architects, software developers, and business analysts who want to learn about the latest open source, low-code application development technologies.

Detailed agenda and list of speakers can be found on each of the event pages.

Places are limited, so make sure to register asap !
Share/Bookmark

Monday, July 03, 2017

Drools, jBPM and Optaplanner are switching to agile delivery!

Today we would like to give everyone in the community a heads up at some upcoming changes that we believe will be extremely beneficial to the community as a whole.

The release of Drools, jBPM and Optaplanner version 7.0 a few weeks ago brought more than just a new major release of these projects.

About a year ago, the core team and Red Hat started investing on improving a number of processes related to the development of the projects. One of the goals was to move from an upfront planning, waterfall-like development process into a more iterative agile development.

The desire to deliver features earlier and more often to the community, as well as to better adapt to devops-managed cloud environments, required changes from the ground up. From how the team manages branches to how it automates builds and how it delivers releases. A challenge for any development team, but even more so to a team that is essentially remote with developers spread all over the world.

Historically, Drools, jBPM and Optaplanner aimed for a cadence of 2 releases per year. Some versions with a larger scope took a bit longer, some were a bit faster, but on average that was the norm.

With version 7.0 we started a new phase in the project. We are now working with 2-week sprints, and with an overall goal of releasing one minor version every 2 sprints. That is correct, one minor version per month on average.

We are currently in a transition phase, but we intend to release version 7.1 at the end of the next sprint (~6 weeks after 7.0), and then we are aiming to release a new version every ~4 weeks after that.

Reducing the release timeframe brings a number of advantages, including:
  • More frequent releases gives the community earlier access to new features, allowing users to try them and provide valuable feedback to the core team. 
  • Reducing the scope of each release allows us to do more predictable releases and to improve our testing coverage, maintaining a more stable release stream.
  • Bug fixes as usual are included in each release, allowing users more frequent access to them as well. 
It is important to note that we will continue to maintain backward compatibility between minor releases (as much as possible - this is even more important in the context of managed cloud deployments as well where seamless upgrades are the norm) and the scope of features is expected to remain similar to what was before. That has two implications:
  • If before, we would release version 7.1 around ~6 months after 7.0, we now will release roughly 6 new versions in those 6 months (7.1, 7.2, ..., 7.6 ), but the amount of feature will be relatively equivalent. I.e., the old version 7.1 is roughly equivalent in terms of features as the scope of the new versions 7.1,..., 7.6 combined. It just splits the scope in smaller chunks and delivers earlier and more often.
  • Users that prefer to not update so often will not lose anything. For instance, a user that updated every 6 months can continue to do so, but instead of jumping from one minor version to the next, he will jump 5-6 minor versions. This is not a problem, again, because the scope is roughly the same as before and the backward compatibility between versions is the same.
This is of course work in progress and we will continue to evolve and adapt the process to better fit the community's and user's needs. We strongly believe, though, that this is a huge step forward and a milestone on the project maturity level.

Share/Bookmark