Monday, November 19, 2007

A Vision for Unified Rules and Processes

Since Drools 4.0 I've been demonstrating our ruleflow stuff, which includes a graphical designer and basic rules and process integration for stateful rule orchestration. What is ruleflow? Ruleflow is the integration of rules and processes, which can predominantly be used to orchestrate the execution of rules. Additionally what 4.0 also provides is a prototype to prove that we don't need to have a process oriented or a rule oriented view of the world. I believe that any company that isn't able to truly unify rules and processes into a single modelling system, how PegaSystems have done, will not be in a suitable position for the future - actually Microsoft get this too with their Workflow Foundations offering, although their rule offering is still very weak. Typically the market has a strong process company with weak rules, or strong rules with weak processes - rules and processes must exist as first class citizens within the modelling and execution environment. The current "modus operandi" for the rule industry is a focus on stateless decision services, where the separate workflow engine at some point calls out to the separate stateless rule engine to assist in some decision making - after 30 years of research and development is that the best we have to offer, multi-million pound license deals that effectively boil down to glorified spreadsheets called via a stateless web services from some workflow engine. Not to mock this model, as it is actually quite useful, but this level of integration is superficial at most and we need to make sure that we unify these models and allow it to go to much greater depths. Once rules and processes are fully integrated that modelling environment will benefit from any other declarative systems added, such as our plans to add Complex Event Processing (CEP) to the engine and language - this means we can have rules monitoring streams of data and triggering process milestones.


Drools 4.0 ruleflow diagram

Partly the reason for the rule industry flocking to this model of stateless decision services is their existing tech is hard to understand and implement and thus more difficult to sell, decision services are simple to understand and thus easier toimplement and sell - it's the rule engine industry's stab at trying to grow their measly market share, compared to the workflow industry.

The jBPM team have put out their vision of the "Process Virtial Machine" (PVM), but it is a process centric vision. The PVM was a term used by Tom Baeyens, in the linked paper, to present the idea of a generic engine for executing different process models, the "virtual machine" term may not be totally appropriate, and already irks some purists, but we have continued with this terminology for mean while - so we can get apples to oranges comparisons, instead of apples to giraffes :) Mike Brock suggests that Virtual Process Engine, or Generic Process Engine - I prefer something away from the terms process and rules, to something that focuses on the unified modelling concepts, so hopefully someone out there can put in an argument for something more appropriate :)

What we lay out here is what we have started to put in place with Drools, our vision of a PVM+ with rules and processes as first class citizens, tightly integrated modelling GUIs, single unified engine and apis for compilation/building, deployment and runtime execution.

So with the base PVM in place what we are working on now? We'll we still have a lot to do, I've found our compilation framework is too coupled to rules, so I'm busy trying to refactor this so it can compile and build both rules and actions. The engine variables are currently only scoped at two levels, globals and rules; we need to make sure that we can scope variables by both process and sub process, and have the rules executed in those processes also scoped to that level. I need to extend our concept of a rule "duration", which is basically a simple timer, to allow for cron type definitions and allow rules to execute each time, if its still true - this will allow for rich conditional timers. I have plans for stateful high availability, via JBoss Cache, and also I need to put in an optimal framework for persistence and restoring - ideally I want all this done, and more by Q1 :) We do not plan to do the BPEL, BPM etc layers and instead hope the jBPM team will become consumers of our tech, and also core developers (a joining of the two teams), and work on these parts of the domain.

The rest of this blog is a small paper put together by our ruleflow lead Kris Verlaenen, but exemplifies the whole Drool's team vision and commitment to declarative programming, via multiple modelling paradigms - no one tool fits all solutions.

The Process Virtual Machine (PVM)
This is an attempt to clarify our vision on an integrated approach for modelling business logic using rules and processes on top of the Drools Platform. It is intended to serve as a glossary, to create a common set of terms that might help in simplifying future discussions and creating a combined vision regarding this matter.

Figure 1 shows an overview of our approach to unify rules and processes by integrating a powerful process virtual machine (PVM+) into the Drools Platform. This allows us to support the execution of rules as well as the execution of processes based on this PVM+ within the Drools Platform. We believe that creating a unified approach for handling rules and processes (for the end user) will result in a much more powerful business logic system than what can be achieved by simply linking separate rules and workflow products. It will also allow us to create a lot of additional services on top of this unified platform (IDE, web-based management system, etc.), which can then be applied easily for both rules and processes, giving a much more unified experience for the end users of the platform. Each of the terms used in the figure will be explained in more detail in the subsequent sections.

Figure 1
PVM
The Process Virtual Machine defines a common model that supports multiple process models. It is the basis for implementing workflow process models, and their implementation. It represents a state machine that can be embedded into any software application. Therefore it defines:
  • A process (definition) model: Defines concepts like a process, variables, nodes, connections, work definitions, etc.
  • A runtime model: Runtime instances corresponding to each of the elements in the process model, like process instance, variable instance, node instance, work item, etc.
  • API: Process instances can be started, aborted, suspended, the value of variable instances can be retrieved, work items can be completed or aborted, etc.
  • Services: The PVM also implements (non-functional) services which are useful for most process language implementations, like persistence, transaction management, asynchronous continuations, etc. These services should all be pluggable (do not have to be used, minimal overhead if not used) and configurable (different strategies could be used for each of these services, this should be configurable and extensible so people can plug in their own implementation).
On top of this process model, the PVM also defines/shows how to use the concepts of process (instance), node (instance), connection, etc. to implement common workflow patterns (in control flow, data, resource, exceptions) like a sequence of nodes, parallelism, choice, synchronization, state, subprocess, scoped variables, etc. These node implementations can be used as a basis for implementing different process languages.

PVM+
Extends the PVM and integrates it into the Drools Platform. This allows:
  • Integration of rules and processes: Processes can include (the power of) rules in their process model whenever appropriate, e.g. split decisions, assignment of actors to work items, rules as expression language, etc. vice-versa, rules can start processes during their execution.
  • Processes and rules share one common data contextl, no need to integrate two (or more) different systems, continuously pass information between those two systems, synchronize data, etc.
  • Processes (and rules) can use other functionality that is offered by the Drools Platform: a unified audit system, unified API to start processes / rules, single build and deployment infrastructure etc.
  • One engine session can execute multiple different process instances in parallel, where each process can interact with the other processes and rules via changes to the shared variable context.
This PVM+ also defines additional node implementations that show the power of integrating rules and processes and how that power can be used inside a process model, e.g. choice using rules to evaluate conditions, milestones (a state where rules decide when to progress to the next state), timers with built in conditionals, actions supporting pluggable dialects, etc.

Specific workflow languages
On top of the PVM+, different (domain-)specific workflow languages can be implemented:
  • jPDL: the general purpose, expressive workflow language for the Java developer
  • PageFlow: workflow language for specifying the control flow in web pages
  • RuleFlow: a workflow language for specifying the order in which large rule sets should be
  • evaluated
  • WS-BPEL: an implementation of the WS-BPEL standard for web service orchestration
  • ...
These languages each define a process model and implementation for each of their nodes. These
implementations will be based in a lot of cases on (a combination of) common node implementations of the PVM(+).

Pluggability: New node implementations can be added to existing process languages, existing
process languages can be extended with new functionality (e.g. time constraints), or entirley new process languages can be plugged in into the PVM+.

Work Definitions
All communication with the external world is handled by using work items, which are an abstract representation of a unit of work that should be executed. Work item handlers are then responsible for executing these work items whenever necessary during the execution of a process instance. This approach has the following advantages:
  • A much more declarative way of programming, where you only define what should executed (using an abstract work item), not how (no code)
  • Hides implementation details
  • Different handlers can be used in different contexts:
    • A workflow can be reused without modifications in different runtime execution contexts (e.g. different companies or different hospitals in the context of clinical workflow) by creating custom handlers for each of these settings
    • A workflow can behave differently depending on its stage in the life cycle. For example, for testing, handlers that do not actually do anything but simply test the execution of the workflow could be registered. For simulation, some visualization of the work items that should be executed, and the possibility for the person doing the simulation to complete/abort these work items is possible.
  • Work item definitions and handler implementations can be reused across nodes, across
    processes, and even across process models.
The different work items that are available in a specific workflow languages should be defined (by defining a unique id for that type of work item, and parameters for that work item). Different sets of work definitions can be defined:
  • Generic work definitions (and their handler implementation) can be defined for common
    task that might be useful in different workflow languages, e.g. related to communication
    (sending a mail, SMS, etc.), invoking a web service, logging a message, etc.
  • People can define their own domain-specific work items (and their handler implementation), which can then be used for modeling processes in that domain. For example, a clinical workflow language could define work items like “nursing order”, “medication order”, “contact general practitioner”, etc.
Extensions
When a unified approach to processes and rules is used, as part of the Drools Platform, extensions on top of these concepts and APIs can easily be reused for all rules and process
languages:
  • Eclipse-based IDE supports developing applications on top of the Drools Platform
    supporting the use of rules and processes. This IDE includes
    • a graphical workflow editor
    • unified error handling
    • integrated debugging
    • unified simulation
    • pluggability of process languages, custom property panels, etc.
    • ...
  • B(R)MS: Business (Rules) Management System, a web-based application that serves as the repository for all business knowledge. Supports unified packaging, versioning, management, quality assurance, etc.
  • Security management: who is allowed to perform which operations on the Drools Platform.
  • (Human) task list management component that can be shared across rules and process
    languages, for integrating human tasks.
  • Reasoning on business logic, which is a combination of all rules and processes of a business.



Pluggable work items (currently in svn trunk)

I'll be at Javapolis this year presenting a BOF on the concepts of Declarative Programming with Rules, Processes and CEP which will cover most of this blog and more. The BOF is on Monday 10th from 21:00 to 22:00. So please do come along, if you want to talk about this in more detail.

15 comments:

  1. Dear Marc, this is very interesting initiative. As I can see RuleFlow defines order of RuleSets to be executed. One thing that it is not clear (for now) is how rules are integrated in the process ? And how did you mapped process events to the rule constructs (if any) ?

    ReplyDelete
  2. It's not that the business rule industry can't do stateful. It's that users and developers don't know how to use it correctly. think of it this way. How many people use stateful EJB and how many use stateless EJB?

    Most developers just don't understand how to build scalable stateful applications. I'm of the opinion that doing stateful applications right, takes lots of experience and patience.

    I've blogged about this before, but I think it merits repeating. The kinds of things you want to use in a stateful manner are facts that are relatively static or have a well define life cycle. The dataset should also be manageable and small (less than 100K rows). For example, say I am building a trading system with routing rules. The kind of data I want to keep in the engine is security data. by that I mean information about the security, like issuer, legal entity, long rating, the exchange, subdivisions and other static information.

    I've used these patterns in the past. I think if you interview 30 "rule developers" less than half of them will know of this rule pattern and even fewer have used the technique.

    also, if you look at the research into reactive databases, there's lots of interesting literature about how to manage state effectively.

    one easy way to achieve a reactive rule system would be to combine coherence with a rule engine. When the facts passes a set elapsed time, it can put the facts into coherence. when the engine needs it, it can ask coherence for it. This way, it becomes more transparent and the rule engine doesn't have to handle persistence.

    That was the main reason I chose to use HashMaps for jamocha when I started. Since there's already a high performance and scalable data grid (ala coherence), makes sense to use it.

    Daniel has a new entry about his team playing around with Ehcache, so I think they are also experimenting with similar ideas. Ehcache is nice, but it's no where near coherence.

    ReplyDelete
  3. "As you might expect, I disagree and posted a response here!
    JT
    James Taylor"

    One of the beauties of a unified modelling environment is you get to choose how you want to model things. Just because you can model and deploy everything on a single engine session, doesn't mean you have to - you get to model your environment how you like from a collection of fully stateless services to a single fully stateful session and all the variations in-between - all with the unified tooling, apis, deployment and management - not forgetting the advantages of unified auditing frameworks, when Mr Sarbanes Oxley comes calling. I'm not pushing one approach to solving enterprise problems over another, but I am pushing the idea to give the user a choice to make those decisions themselves. You want a rule centric view of your current model, you got it, you want a process centric view of your model, you got it, you want something in-between, you got it - it's the users choice they know their requirements best. It's my fault in being ambiguous with my frustrations, after all blogs are just informative rants :) My frustration wasn't with SOA and decision services which is why I say "Not to mock this model, as it is actually quite useful" but more that the industry stops here, when more can be achieved. I don't want to go to FileNet for my workflow and Ilog for my rules, meaning I have to learn and manage two ways to deploy the stuff, two different apis and two server side management systems and have my interoperability points limited to the integration that those two companies thought to provide - and suffer from impedance miss matches else where. And when you start to take a more unified view of things it starts to allow for some more interesting behaviour, that would also be valid in SOA environments too. Rules and processes are inextricably linked; you simply cannot have processes without rules of some level, while rules can do without processes there are many types of problems that are hard to solve cleanly without them (which is why many rule companies are introducing simple ruleflow capabilities).

    So I agree with you on the importance of SOA and decision services, sorry I phrased my frustrations badly in the blog. What I'm proposing does not limit any of the bulleted points you make on decisioning, in fact it enables them. But I don't agree with your view that unifying rules and processes is like unifying the UIs and Databases, the two are very complimentary with lots of overlap - but luckily I'm willing to put my money where my mouth is, so watch this space :) The important thing is we need to push our understanding and vision in these areas, R&D has moved far too slowly in the rules industry and we have to push the boundaries and see what falls out.

    Mark
    http://blog.athico.com The Drools Blog

    ReplyDelete
  4. Yaakov Kohen has just blogged this blog with some intereting comments on the lines of giving people enough rope to hang themselves with :)
    http://javarules.blogspot.com/2007/11/blogs-on-top-of-blogs.html

    ReplyDelete
  5. I've gotta agree with Mark and disagree with JT, assuming that we're trying to capture interesting business logic (non-trivial decisions or exception rich processes), which we do as this is the bit the business really cares about.

    Integrating rules and process is a good thing and I hope this goes somewhere, as we've been too hung up on technology platforms and not focused on what sort of things business solutions need to do. The sort of business logic we see in the wild doesn't make this distinction between rules and process; complex decisions are often decision making processes that involve trial-and-error, while any worthwhile process is exception rich, requiring embedded decisions. Often the two are interwoven. The separation between rules and process is just an artifact of where the technologies came from, and not where we want them to be. Having separate rules and process engines creates a huge amount of overhead that we can better do without.

    There's a lot of good technology we could be looking at to help push this forward, from PRS to modern beasts which vendors like OSLO are offering.

    ReplyDelete
  6. http://processdevelopments.blogspot.com/2007/11/unifying-processes-and-rules.html

    ReplyDelete
  7. ah... so you have to put the html tags in yourself :-)

    My blog response

    ReplyDelete
  8. marcs vision lets hope that in the future the life as software architect for business applications becomes easier :-)
    I'm developing an ERP solution (driven by workflow, business processes and business rules), so its the daily frustration how to make all things run and how to integrate the business guys into the development lifecycle.
    its always dificult to tell a business guy when a part of his work is a drools-ruleflow-process and when its a jbpm-process.
    at the moment we're using drools-ruleflow only for complex rules where the order of steps is important or with complex conditions when a step can be executed. but if its long-running we have to use jBPM because the flow is persisted. or if there are different actors we also use jBPM because drools has no swimlanes.
    as you see: some decisions are only technical - if the rules-engine and the process-engine were better integrated I can imagine easier ways ;-)
    to hide this from the business guys and make life easier, I'm trying to unify it the following way:
    using a customized MagicDraw UML activity diagram together with domain specific UML profiles for business processes and rules.
    now the business people can model all the processes using a single environment.
    but this is only the first step - our project is model-driven and uses the MagicDraw - EMF - integration and also the openArchitectureWare (oAW) tools from eclipse modeling project:
    domain specific oAW-XPAND-templates generate ruleflows and jPDL and much infrastructure - code.
    at the moment its not so easy to bring it all together because same data can exist inside jBPMs context and also as facts inside the drools engine. but using oAW-XPAND-templates we can hide this technical barrier from the business application developer.
    if something of Marcs vision becomes true it will be easy to change the behavior of the templates to support PVM+ or something else.
    of course it would also be great to have drools and jBPM integrated inside eclipse IDE with ONE editor for ruleflows and business processes and to have ONE web-based management tool like BRMS and ONE Monitoring/Auditing-Tool for rules and processes together and and and... ;-)
    but I also have to mention that the combination/integration of rules and processes is not the only scenario where we use drools inside our application: drools also works perfectly as rules-only-engine to make complex validation inside the GUI of an eclipse RCP client.
    thanks to drools and jBPM (and using eclipse as a platform) its possible to develop flexible business applications. we'll publish our way to integrate drools and jBPM in some weeks as open source and are looking forward what will happen with Marcs vision.
    ekkehard

    ReplyDelete
  9. As i said over on JT's blog,

    1) What does the business get or see from this approach? Can business people manage their rules and decisions directly or do they have to depend on a modeling expert?

    2) I would simply state that everything is inextricably linked, one could say that rules and data are are more tightly linked than rules and process, given the similarity of data models and terms & facts models. I know you don’t want separate process and rule platforms, do you want to get rid of separate data platforms as well? If it is the nature of the marketplace that today there are separate process products and separate rules products, that will change when vendors realize that it makes sales sense to combine them as a product, and I think recent purchases/mergers are showing this to be happening already. Still, some customers may prefer ‘best-of-breed’ solutions and want to integrate the whole themselves. There have always been pros and cons to both approaches beyond this domain, and I can’t see anything yet that will change the minds of those who favour either approach.

    Anyway, my 2 pennies/pence worth.

    ReplyDelete
  10. Although the idea of a unified approach sounds desirable, I don't think it is feasible. At best, I think the only thing you can do is unify the runtime semantics. Trying to cover all modeling scenarios, feels like an unwieldy thing to do. The industry can definitely improve it and make it better.

    ReplyDelete
  11. Unification doesn't mean one modelling paradigm - you can't model everything with just a process paradigm, or a rules paradigm.

    When the execution environment provides seamless communication between your rules and process it makes it easier to exploit the overlaps.

    There is also a lot of core areas in the code, api and infrastructure that you can leverage.

    Rules themselves should not be procedural, but life is never that simple, so to model many problems you start to pollute those rules model with procedural execution semantics. Unification is about extracting that information to be properly modelled in a procedural graph language. At the same time the rules metaphor can make our procedural metaphor much richer.

    Typically when a process has to make a decision it's an evaluation of the context at that point in time - while we will still allow this we also allow the conditionals on splits (decisions), milestones, timers to be control by a rule LHS "when" statement.

    Let's take another use case, campaign management. The emailer task in the process definition can have its data driven by a lhs "when " rule statement. We can then combine the rules and process definitions to provide strategies for this to ensure maximum delivery - i.e. only send 100 in the last 2 hours to this domain, then switch to another domain.

    Kris, the brains behind this, has spent 6 years on his Phd researching clinical diagnosis, which is both rules and process intensive, and this vision of finding out the overlaps of rules and processes and make sure they are easily exploitable in a single platform is the results of this - along with the model of user pluggable "work items" to enable a stronger declarative programming paradigm.

    ReplyDelete
  12. Mark,

    Sounds like you should check out NASA's RCS system and BDI agents. There's already been a lot of work done in this area.

    r.

    PEG

    ReplyDelete
  13. this is just my bias opinion. usually in large firms, those pieces are owned by different groups, so it's politically undesirable to unify those. Trying to get those groups to work together might be like sky diving without a parachute.

    Small startup is the only place where I feel that has a good chance of working.

    The other major challenge I see the process may be long running and could take days to complete. In those cases, many products persist the state to a database or file system and then reload it. One could consider that a stateful application, but it's more like a continuation. A business rule engine shouldn't be handling persistence in my mind. It should be handled at a higher level like a rule service. To the consumer, it looks like a stateful session, but under the hood it's really stateless.

    Running a rule engine in a stateful mode should only be done for real-time, event processing or reactive systems. That's my bias 2 bits.

    ReplyDelete
  14. Greetings, Programs:

    Lots of comments - this makes 15 so far - and most seem to have lost the idea of AI in the beginning. (WHAT did he say??? Forget it! He's some kind of lunatic, radical, old time AI guy!) OK, all jokes aside, let's go back to what we were trying to do originally; we were trying to emulate the human mind. The two main branches of AI back then were RBS (RuleBased Systems) and ANN (Artificial Neural Nets) and development in both fields seemed to go to different corners to die in oblivion.

    Somewhere along the way during the Great Winter of AI in the late 80's and early 90's, the business guys came to the rescue by using rules in the business world. In about 2000 they (meaning ILOG and ND) developed the spreadsheet approach to doing rules - and the business guys just ate it up. They started queueing up to spend millions of dollars, pounds, yen and Euro to get the "jump" on the competition. It became all the latest craze.

    This is nothing more than giving the business guys access to Java code (or C++ or whatever) to make changes so that they don't have to go through IT to make the changes for them. And we, the geeks in AI, have the audacity to call this structured programming approach a BRMS, Business Rules Management System. Y U C K ! ! !

    Any time that we have to do something really cool and really "intelligent" that would give that business user a REAL jump on the competition, they get sold a bill of goods by some sales guys who can not understand the difference between declarative programming and procedural programming. (And I have knows some rulebase guys who could not understand the difference!) A spreadsheet approach is something that is understood by sales, understood by the business guys and is understood by the pseudo-geeks at the vendor pretending to be rulebased professionals.

    Why do we do it? Because doing something declarative would mean totally restructuring and rethinking the entire business process. It would mean having to understand using a non-procedural approach by using something like Intelligent Agents (remember FIPA?) and tying to get ANN into the rulebase. All of this is nothing more than trying to simulate an Analog Computer that also has storage of rules available.

    So why do we do it? Because we're all whores and we have all sold out to the easiest (not necessarily the best) solution. All of us can understand the necessity for process and flow of documents and things of that nature because we have grown up that way.

    But - what if we stopped and began to actually think about what we need to do and how to go about it the most intelligent way? Would we do it differently? Maybe not. But we never go down that road because it takes more time and is more expensive and the bloody salesmen can't explain it in their silly Power Point presentations.

    So, BREAK OUT of the mold. Try to think. If you can understand the Rete Algorithm (and about 80% of the Senior Rulebase Consultants can not do that) then you can help design a system that will actually "THINK" rather than just implementing rules though up by some underpaid employee who didn't understand the business in the first place.

    OK, Rant Over - go back to sleep.

    SDG
    Yaakov

    ReplyDelete