Drools Clips progress

Made some progress over the weekend with Drools Clips, which will provide a Clips like language for Drools. Deftemplates are now working and I did some work on PackageBuilder so that it's now able to handle multiple namespaces and have a RuleBase attached to provide a more "shell" like environment suitable for Clips. Michael Neale also got a basic command line shell working. So what does it support?

• deftemplate
• defrule
• deffuction
• and/or/not/exists/test Conditional Elements
• Literal, Variable, Return Value and Predicate field constraints

You can look at the ClipsShellTest and LhsClipsParserTest get an idea of the full support. It's still early stages and it's very rough in places, especially on error handling and feedback as well as no view commands to display data. For a little fun here is a screenshot of the shell in action:

(click to enlarge)

The screen shot is a contrived example but it does show a shell environment cleanly mixing deftemplates and pojos - note that Drools 5.0 does not require shadow facts, due to the new asymmetrical Rete algorithm. It also shows deffunction in use. This will be part of Milestone1, that I'm hoping to tag tomorrow.

Drools Clips

I've done some more work on Drools Clips in the last few weeks and it's starting to take shape now and the basic shell is working and we should have something useful as part of the next Drools milestone release in a few weeks. You can see the unit tests here:
http://anonsvn.labs.jboss.com/labs/jbossrules/trunk/drools-clips/src/test/java/org/drools/clips/ShellTest.java

Below you can see two of the unit tests demonstrating deffunctions and rules:

public void testRuleCallDeftemplate() {
    String function = "(deffunction max (?a ?b) (if (> ?a ?b) then (return ?a) else (return ?b) ) )";
    this.shell.eval( function );
    
    this.shell.eval( "(import org.drools.*)" );
    this.shell.eval( "(defrule testRule (Person (age ?age) ) => (printout t hello)
                      (printout t \" \" (max 3 ?age) ) )" );
    this.shell.eval( "(assert (Person (name mark) (age 32) ) )" );
    this.shell.eval( "(run)" );
    assertEquals( "hello 32",
                  new String( this.baos.toByteArray() ) );        
}

public void testTwoSimpleRulesWithModify() {
    this.shell.eval( "(import org.drools.*)" );
    this.shell.eval( "(defrule testRule ?p <- (Person (name ?name&mark) ) =>
                      (printout t hello) (printout t \" \" ?name) (modify ?p (name bob) ) )" );
    this.shell.eval( "(defrule testRule (Person (name ?name&bob) ) => (printout t
                       hello) (printout t \" \" ?name))" );
    this.shell.eval( "(assert (Person (name mark) ) )" );
    this.shell.eval( "(run)" );
    assertEquals( "hello markhello bob",
                  new String( this.baos.toByteArray() ) );
}

We have support for a large set of the Clips LHS rule syntax and functions are easy to add. As Drools DRL also has support for the Jess 7.0 style infix notation we will add support for that too. The goal is to support the full LHS syntax for Jess and Clips. For now we will be missing out logical assertions, Jess slot specific and Clips COOL. Our logical assertion approaches are slightly different, so it's easier to leave that out, and we currently have no way to support slot specific which is also the behaviour of Clips COOL. As we have full support for Java pojos and their approach to OO we have less need for COOL at this stage. It would be nice to add slot specific support soon and I hope to do it some time this year.

We do not yet support deftemplates but the rules will reason directly over pojo classes, as our DRL already does. I will add support for those soon.

One of the cool things about this is that it doesn't just allow you to execute Jess and Clips rules but that it also provides you with a migration path. Using DrlDumper you can now load Jess/Clips rules via drools-clips and have the DrlDumper dump them to the more modern Drools DRL syntax.

I've actually changed my design approach for this. Originally I was building my own Lisp execution engine and then realised that I was creating much of the infrastructure that we already had put into MVEL. As Lisp really is just an abstract syntax tree you can actually take any Lisp statement and dump it to MVEL for execution. So I gave up on my own Lisp execution engine and instead did a Lisp to MVEL converter. This is only a stop gap though as Mike Brock has promised to eventually do a direct S-Expression lexer and parser for MVEL. It's trivial to add new built-in functions to drools-clips by simply creating an MVEL language dumper. You can look at the existing functions like the 'if' function in the link below to see how simple this is:
http://anonsvn.labs.jboss.com/labs/jbossrules/trunk/drools-clips/src/main/java/org/drools/clips/functions/IfFunction.java

update - Drools adds Clips parser

Progress is going well with the Clips parser I now have the full LHS working, exception functions. That includes the 'and', 'or', 'not and exists conditional elements, with full nesting, including 'and' and 'or' inside the 'not' and 'exists'. Patterns work with literals, bound variables, predicates and return values. I'm now working on functions at which point we should be able to execute Clips rules inside the JBoss Rules engine. Probably the hardest part with functions is finding a sane way to deal with primitives in functions, especially built in Math functions. After that we'll look at mapping our 'accumulate', 'collect', 'forall' and 'from' implementations.

Drools adds Clips parser

We have often stated that the Drools rule engine is fully language independant, but to date it's only had two parsers - XML and DRL. For a bit of fun this weekend, hey I'm a willd type of guy, I went ahead and started on an experimental Clips grammar with ANTLR, although its still a long way from being finished. I hope It should eventually provide a migration path for Clips users as well as demonstrate how people can build their own grammars for the Drools rule engine using ANTLR, you don't have to use ANTLR, but its our parser of choice. Clients who use other vendor products and want to migrate, but worried about the investment in the authored rules, can now write parsers for those products targetting JBoss Rules; obviously there are other things to consider like feature parity and execution models. We also support "dumpers", via the visitor design patter, making round tripping possible - i.e. currently you can load an xml and dump drl, and vice versa, we will try and make the same possible with Clips.

So far it's parsing multiple rules with multiple patterns, it supports literal, bound variable, predicate and return value field constraints as well as & and | field constraint connectives. I don't yet have it working with functions, so I've just put text into the predicate and return value for now, for the purposes of testing. The next stage is to get nested Conditional Elements working including 'and', 'or', 'not' and 'exists'. The really adventerous can help make 'accumulate', 'collect' and 'from' available :)

Clips/Lisp is a fairly simple grammar so this is a great learning project for anyone that wants to learn ANTLR and how to write custom parsers for the Drools Rule Engine; so if you want to help out why not pop onto codehause IRC #drools, or subscribe to our developer mailing list, and we'll help you get started.

The Clips ANTLR grammar is here http://anonsvn.labs.jboss.com/labs/jbossrules/trunk/drools-compiler/src/main/resources/org/drools/clp/CLP.g

And for those interested here is the current unit test. This demonstrates the intermediate AST we use to build a language agnostic view of a grammar. This AST is "dumb" and we call it Descr, short for Description, this is because everything at this stage is held in a String format; little or not validation has been done, it's just a pure string based tree representing the rules, this is then pass to the PackageBuider which validates the descr tree and builds the resulting rule AST.

  
  public void testRule() throws Exception {
       RuleDescr rule = parse(
       "(defrule xxx ?b <- (person (name \"yyy\"&?bf|~\"zzz\"|~=(ppp)&:(ooo)) ) 
                     ?c <- (hobby (type ?bf2&~iii) (rating fivestar) )").rule();
      
       assertEquals( "xxx", rule.getName() );
      
       AndDescr lhs = rule.getLhs();
       List lhsList = lhs.getDescrs();
       assertEquals(2, lhsList.size());
      
       // Parse the first column
       ColumnDescr col = ( ColumnDescr ) lhsList.get( 0 );
       assertEquals("?b", col.getIdentifier() );
       assertEquals("person", col.getObjectType() );
      
       List colList = col.getDescrs();
       assertEquals(2, colList.size());
       FieldConstraintDescr fieldConstraintDescr = ( FieldConstraintDescr ) colList.get( 0 );
       List restrictionList = fieldConstraintDescr.getRestrictions();
      
       assertEquals("name", fieldConstraintDescr.getFieldName() );       
       // @todo the 7th one has no constraint, as its a predicate, have to figure out how to handle this
       assertEquals(8, restrictionList.size());
              
      
       LiteralRestrictionDescr litDescr = ( LiteralRestrictionDescr ) restrictionList.get( 0 );
       assertEquals("==", litDescr.getEvaluator() );
       assertEquals("yyy", litDescr.getText() );

       RestrictionConnectiveDescr connDescr = ( RestrictionConnectiveDescr ) restrictionList.get( 1 );
       assertEquals(RestrictionConnectiveDescr.AND, connDescr.getConnective() );
      
       VariableRestrictionDescr varDescr = ( VariableRestrictionDescr ) restrictionList.get( 2 );
       assertEquals("==", varDescr.getEvaluator() );
       assertEquals("?bf", varDescr.getIdentifier() );       
      
       connDescr = ( RestrictionConnectiveDescr ) restrictionList.get( 3 );
       assertEquals(RestrictionConnectiveDescr.OR, connDescr.getConnective() );
      
       litDescr = ( LiteralRestrictionDescr ) restrictionList.get( 4 );
       assertEquals("!=", litDescr.getEvaluator() );
       assertEquals("zzz", litDescr.getText() );
      
       connDescr = ( RestrictionConnectiveDescr ) restrictionList.get( 5 );
       assertEquals(RestrictionConnectiveDescr.OR, connDescr.getConnective() );
      
       ReturnValueRestrictionDescr retDescr = ( ReturnValueRestrictionDescr ) restrictionList.get( 6 );
       assertEquals("!=", retDescr.getEvaluator() );
       assertEquals("ppp", retDescr.getText() );
      
       PredicateDescr predicateDescr = ( PredicateDescr ) colList.get( 1 );
       assertEquals( "ooo", predicateDescr.getText() );
      
      
       // Parse the second column
       col = ( ColumnDescr ) lhsList.get( 1 );
       assertEquals("?c", col.getIdentifier() );
       assertEquals("hobby", col.getObjectType() );  

       colList = col.getDescrs();
       assertEquals(2, colList.size());
       fieldConstraintDescr = ( FieldConstraintDescr ) colList.get( 0 );
       restrictionList = fieldConstraintDescr.getRestrictions();
      
       assertEquals("type", fieldConstraintDescr.getFieldName() );       
      
       varDescr = ( VariableRestrictionDescr ) restrictionList.get( 0 );
       assertEquals("==", varDescr.getEvaluator() );
       assertEquals("?bf2", varDescr.getIdentifier() ); 
      
       connDescr = ( RestrictionConnectiveDescr ) restrictionList.get( 1 );
       assertEquals(RestrictionConnectiveDescr.AND, connDescr.getConnective() );
      
       litDescr = ( LiteralRestrictionDescr ) restrictionList.get( 2 );
       assertEquals("!=", litDescr.getEvaluator() );
       assertEquals("iii", litDescr.getText() );
      
       fieldConstraintDescr = ( FieldConstraintDescr ) colList.get( 1 );
       restrictionList = fieldConstraintDescr.getRestrictions();       
      
       assertEquals("rating", fieldConstraintDescr.getFieldName() );
      
       litDescr = ( LiteralRestrictionDescr ) restrictionList.get( 0 );
       assertEquals("==", litDescr.getEvaluator() );
       assertEquals("fivestar", litDescr.getText() );       
   }