Monday, June 15, 2009

The Benefits of (Automated) Testing

A quick post to point to this article I was reading today. Nothing ground-breaking in there, but a fair statement of testing goals at the several levels of a project.

My personal opinion is that does not matter if you use TDD principles or not: software needs testing! And the lower levels of testing must be automated. It is as simple as that.

This is a common mantra in Open Source projects like Drools, specially because you have many many people coming and going, contributing a feature here or a patch there. Although, that is not always the case in closed source projects, specially inside companies (and I worked in several of these projects in the past).

IMO, the most overlooked benefit of testing is the long term stability of the project. Just to mention one example, it is not necessary to achieve the 90% code coverage in unit tests right from the start. As a developer, write tests that cover the boundaries and common use cases of the code you are creating and let it rest in your test suite. Whenever a bug is found, make sure to add more unit tests that cover exactly the bug and the fix you implemented. This way, you will not overburden the project development time line writing "too many tests", but will have your project mature and stabilize nicely and incrementaly.

Drools uses unit and integration testing extensively. One of the latest builds of version 5 on our CI server shows that 2814 unit and integration tests were executed. Code coverage tools usually don't deal well with code and bytecode generation, so it is hard to state exactly how much code we cover, but the important thing is that we are confident enough in them to make changes to the code and having tests to fail if we do something wrong. Something extremely important when you have a software that evolves constantly and heads to directions that are results of all the reasearching we do, but that we could not imagine we would go if someone asked us 2 or 3 years ago.

2 comments:

  1. Edson et al:

    My dream is that one day all rulebased programmers will realize that the ultimate goal is 100% automated testing. While, at first glance, that sounds like I agree with everyone else, I do not agree that 99.999 per cent automated testing is enough, which obviously is not in agreement with the rest of the programming world.

    Test-Driven Development (TDD) is still pretty much an unobtainable goal in the rulebase world. It isn't enough to test the code and the integrations and the GUI (which is exactly what you guys, the vendors, have to do - OK, should be doing - when building any application) but the real testing of a rulebase application come when testing the rulebase itself, that of verification testing. How do you test ALL of the rules?

    In the years past I have worked with almost all of the so-called "giants" in this small industry of ours and many have capitulated to the idea that, "we just can't test everything." This idea is usually based on time constraints, people constraints, hardware constraints, etc, etc. But I would like to propose that not only CAN we do 100% verification testing but we MUST do 100% verification testing in this day of automated hacking that will find any small flaw and exploit it to the usually illlegal goal of those who would exploit those tiny flaws in logic.

    Welcome to the world of Beta-Error testing. So many times I have drawn my four little boxes on the board for the IT department only to receive blank stares that usually indicated that they either (1) hand never heard of such a thing or (2) there was no way that they wanted to be involved in such a thing. The chart is really very simple: (TL) we expected X and received X, (TR) we expected X and received Y - an Alpha error, (BR) we expected NOT X and got NOT X and, finally, the infamous Beta-error, (BL) we expected NOT X and received X; all this is such that X is success, Y is failure.

    The dreaded Beta Error is something that can be tested for only when directed by experienced persons in the field. The thought goes something like this: We have seen a success when it really should not have tested correctly because we did not think to look for such a thing. For example, someone qualified for a really low rate on a mortgage when they should not have qualified at all. Or someone was not detected as an intruder when, in fact, that person did not belong in that area.

    Usually this happens because of some unknown fact when the program was tested in the early days and that part of the program was "certified" as being Green. Later, however, something crept in unawares and that code should have detected something that it did not know about, something that the original programmers did not know about as being an error. Beta Error!

    Anyway, just a comment about something that comes about because of poor planning on the part of the "so-called" architects of the system who did not KNOW about the pitfalls of something happening that could be know by those who wrote the program, only by those who have been in the business a long time and can recall the pitfalls of simply slinging code and NOT having a War Room Development Process, of depending on the Testing to catch all of the problems that could happen.

    SDG
    jco

    ReplyDelete
  2. James,

    I completely agree with you, that on an ideal world, 100% scenario/code test coverage and, even better, formal proof of code/rules correctness would be the definitive goal and the solution for all the bugs users face.

    Although, in real world, setting feasible goals on a cost/benefit basis is the best development teams can aim for, it seems. And even that is proving, sadly, sometimes to be hard to achieve in some companies.

    Business Rules technologies, being relatively new to the field of software development have even less tooling for that purpose and you are right we need to invest in that area. We, at Drools, added such tooling for version 5 and are working to make it the best of the breed for the next release cycle. Keep an eye out and feedback always welcome.

    Cheers,
    Edson

    ReplyDelete