Monday, June 15, 2009

The Benefits of (Automated) Testing

A quick post to point to this article I was reading today. Nothing ground-breaking in there, but a fair statement of testing goals at the several levels of a project.

My personal opinion is that does not matter if you use TDD principles or not: software needs testing! And the lower levels of testing must be automated. It is as simple as that.

This is a common mantra in Open Source projects like Drools, specially because you have many many people coming and going, contributing a feature here or a patch there. Although, that is not always the case in closed source projects, specially inside companies (and I worked in several of these projects in the past).

IMO, the most overlooked benefit of testing is the long term stability of the project. Just to mention one example, it is not necessary to achieve the 90% code coverage in unit tests right from the start. As a developer, write tests that cover the boundaries and common use cases of the code you are creating and let it rest in your test suite. Whenever a bug is found, make sure to add more unit tests that cover exactly the bug and the fix you implemented. This way, you will not overburden the project development time line writing "too many tests", but will have your project mature and stabilize nicely and incrementaly.

Drools uses unit and integration testing extensively. One of the latest builds of version 5 on our CI server shows that 2814 unit and integration tests were executed. Code coverage tools usually don't deal well with code and bytecode generation, so it is hard to state exactly how much code we cover, but the important thing is that we are confident enough in them to make changes to the code and having tests to fail if we do something wrong. Something extremely important when you have a software that evolves constantly and heads to directions that are results of all the reasearching we do, but that we could not imagine we would go if someone asked us 2 or 3 years ago.


  1. Edson et al:

    My dream is that one day all rulebased programmers will realize that the ultimate goal is 100% automated testing. While, at first glance, that sounds like I agree with everyone else, I do not agree that 99.999 per cent automated testing is enough, which obviously is not in agreement with the rest of the programming world.

    Test-Driven Development (TDD) is still pretty much an unobtainable goal in the rulebase world. It isn't enough to test the code and the integrations and the GUI (which is exactly what you guys, the vendors, have to do - OK, should be doing - when building any application) but the real testing of a rulebase application come when testing the rulebase itself, that of verification testing. How do you test ALL of the rules?

    In the years past I have worked with almost all of the so-called "giants" in this small industry of ours and many have capitulated to the idea that, "we just can't test everything." This idea is usually based on time constraints, people constraints, hardware constraints, etc, etc. But I would like to propose that not only CAN we do 100% verification testing but we MUST do 100% verification testing in this day of automated hacking that will find any small flaw and exploit it to the usually illlegal goal of those who would exploit those tiny flaws in logic.

    Welcome to the world of Beta-Error testing. So many times I have drawn my four little boxes on the board for the IT department only to receive blank stares that usually indicated that they either (1) hand never heard of such a thing or (2) there was no way that they wanted to be involved in such a thing. The chart is really very simple: (TL) we expected X and received X, (TR) we expected X and received Y - an Alpha error, (BR) we expected NOT X and got NOT X and, finally, the infamous Beta-error, (BL) we expected NOT X and received X; all this is such that X is success, Y is failure.

    The dreaded Beta Error is something that can be tested for only when directed by experienced persons in the field. The thought goes something like this: We have seen a success when it really should not have tested correctly because we did not think to look for such a thing. For example, someone qualified for a really low rate on a mortgage when they should not have qualified at all. Or someone was not detected as an intruder when, in fact, that person did not belong in that area.

    Usually this happens because of some unknown fact when the program was tested in the early days and that part of the program was "certified" as being Green. Later, however, something crept in unawares and that code should have detected something that it did not know about, something that the original programmers did not know about as being an error. Beta Error!

    Anyway, just a comment about something that comes about because of poor planning on the part of the "so-called" architects of the system who did not KNOW about the pitfalls of something happening that could be know by those who wrote the program, only by those who have been in the business a long time and can recall the pitfalls of simply slinging code and NOT having a War Room Development Process, of depending on the Testing to catch all of the problems that could happen.


  2. James,

    I completely agree with you, that on an ideal world, 100% scenario/code test coverage and, even better, formal proof of code/rules correctness would be the definitive goal and the solution for all the bugs users face.

    Although, in real world, setting feasible goals on a cost/benefit basis is the best development teams can aim for, it seems. And even that is proving, sadly, sometimes to be hard to achieve in some companies.

    Business Rules technologies, being relatively new to the field of software development have even less tooling for that purpose and you are right we need to invest in that area. We, at Drools, added such tooling for version 5 and are working to make it the best of the breed for the next release cycle. Keep an eye out and feedback always welcome.


  3. After reading this web site I am very satisfied simply because this site is providing comprehensive knowledge for you to audience.
    Thank you to the perform as well as discuss anything incredibly important in my opinion. We loose time waiting for your next article writing in addition to I beg one to get back to pay a visit to our website in

    Selenium training in Chennai
    Selenium training in Bangalore

    1. IEEE Project Domain management in software engineering is distinct from traditional project deveopment in that software projects have a unique lifecycle process that requires multiple rounds of testing, updating, and faculty feedback. A IEEE Domain project Final Year Projects for CSE system development life cycle is essentially a phased project model that defines the organizational constraints of a large-scale systems project. The methods used in a IEEE DOmain Project systems development life cycle strategy Project Centers in India provide clearly defined phases of work to plan, design, test, deploy, and maintain information systems.

      This is enough for me. I want to write software that anyone can use, and virtually everyone who has an internet connected device with a screen can use apps written in JavaScript. JavaScript Training in Chennai JavaScript was used for little more than mouse hover animations and little calculations to make static websites feel more interactive. Let’s assume 90% of all websites using JavaScript use it in a trivial way. That still leaves 150 million substantial JavaScript Training in Chennai JavaScript applications.

  4. This comment has been removed by the author.

  5. It's a nice article, Which you have shared here about the Benefits of (Automated) Testing. Your article is very useful for those who are looking to electrical tag testing at an affordable price. I would like to thanks for sharing this post here.

  6. The article is so informative. This is more helpful
    software testing training courses
    selenium course Thanks for sharing

  7. It’s great to come across a blog every once in a while that isn’t the same out of date rehashed material. Fantastic read.
    software testing training chennai | software testing course chennai

  8. I have been following your post for a long time. I always found it very interesting and valuable. keep posting it is really helpful.
    Cloud Migration services

    Aws Cloud Migration services

    Azure Cloud Migration services

  9. We are a part of the success story for many of our customer's successful cloud Migrations.
    Vmware Cloud Migration services

    Database Migration services

  10. Thank you for the informative post about Security challenges in AWS , Found it useful . cloud migration services have now become secured and with no-risk

    Lia Infraservices

  11. I am really impressed with the way of writing of this blog. The author has shared the info in a crisp and short way.
    Cloud Migration services

    Best Cloud Migration Tool

  12. Nice Sharing..! I have been following you for a couple of months now but this is my first time commenting on a blog post. Thank you for sharing your knowledge and experience with us. Keep up the good work. Already bookmarked for future reference.

    software testing company


  13. This is an awesome post. Really very informative and creative contents.
    WordPress website development company in Chennai


  14. This great article has really peaked my interest. I am going to book mark your blog and keep checking for new information about once per week.
    Selenium Training in chennai | Selenium Training in anna nagar | Selenium Training in omr | Selenium Training in porur | Selenium Training in tambaram | Selenium Training in velachery

  15. Thanks for sharing this article here about the automated testing. Your article is very informative and I will share it with my other friends as the information is really very useful. Keep sharing your excellent work. If anyone looking to Buy software application automation tool, selenoid is good for you.

  16. I must thank you for posting this blog because the topic is very much in demand today and everyone wants to read about it.Test And Tag Melbourne

  17. Thanks for sharing this article here about the guidance of automated testing. After reading your article I got very much information about the tips of testing and it resolved many of my doubts. testing and tagging sydney

  18. Very informative article, which you have shared here about the automated-testing. After reading your article I got very much information as we provide electrical test and tag at an affordable price

  19. After reading your article I was amazed. I know that you explain it very well. And I hope that other readers will also experience how I feel after reading your article Test And Tag Adelaide

  20. Good article!!! Thank you so much for sharing this pretty post, it was so good to read and useful to improve my knowledge as updated one, keep blogging…
    DevOps Training in Chennai

    DevOps Course in Chennai

  21. Good article!!! Thank you so much for sharing this pretty post, it was so good to read and useful to improve my knowledge as updated one, keep blogging…
    DevOps Training in Chennai

    DevOps Course in Chennai