Monday, June 15, 2009

The Benefits of (Automated) Testing

A quick post to point to this article I was reading today. Nothing ground-breaking in there, but a fair statement of testing goals at the several levels of a project.

My personal opinion is that does not matter if you use TDD principles or not: software needs testing! And the lower levels of testing must be automated. It is as simple as that.

This is a common mantra in Open Source projects like Drools, specially because you have many many people coming and going, contributing a feature here or a patch there. Although, that is not always the case in closed source projects, specially inside companies (and I worked in several of these projects in the past).

IMO, the most overlooked benefit of testing is the long term stability of the project. Just to mention one example, it is not necessary to achieve the 90% code coverage in unit tests right from the start. As a developer, write tests that cover the boundaries and common use cases of the code you are creating and let it rest in your test suite. Whenever a bug is found, make sure to add more unit tests that cover exactly the bug and the fix you implemented. This way, you will not overburden the project development time line writing "too many tests", but will have your project mature and stabilize nicely and incrementaly.

Drools uses unit and integration testing extensively. One of the latest builds of version 5 on our CI server shows that 2814 unit and integration tests were executed. Code coverage tools usually don't deal well with code and bytecode generation, so it is hard to state exactly how much code we cover, but the important thing is that we are confident enough in them to make changes to the code and having tests to fail if we do something wrong. Something extremely important when you have a software that evolves constantly and heads to directions that are results of all the reasearching we do, but that we could not imagine we would go if someone asked us 2 or 3 years ago.
Share/Bookmark

21 comments:

  1. Edson et al:

    My dream is that one day all rulebased programmers will realize that the ultimate goal is 100% automated testing. While, at first glance, that sounds like I agree with everyone else, I do not agree that 99.999 per cent automated testing is enough, which obviously is not in agreement with the rest of the programming world.

    Test-Driven Development (TDD) is still pretty much an unobtainable goal in the rulebase world. It isn't enough to test the code and the integrations and the GUI (which is exactly what you guys, the vendors, have to do - OK, should be doing - when building any application) but the real testing of a rulebase application come when testing the rulebase itself, that of verification testing. How do you test ALL of the rules?

    In the years past I have worked with almost all of the so-called "giants" in this small industry of ours and many have capitulated to the idea that, "we just can't test everything." This idea is usually based on time constraints, people constraints, hardware constraints, etc, etc. But I would like to propose that not only CAN we do 100% verification testing but we MUST do 100% verification testing in this day of automated hacking that will find any small flaw and exploit it to the usually illlegal goal of those who would exploit those tiny flaws in logic.

    Welcome to the world of Beta-Error testing. So many times I have drawn my four little boxes on the board for the IT department only to receive blank stares that usually indicated that they either (1) hand never heard of such a thing or (2) there was no way that they wanted to be involved in such a thing. The chart is really very simple: (TL) we expected X and received X, (TR) we expected X and received Y - an Alpha error, (BR) we expected NOT X and got NOT X and, finally, the infamous Beta-error, (BL) we expected NOT X and received X; all this is such that X is success, Y is failure.

    The dreaded Beta Error is something that can be tested for only when directed by experienced persons in the field. The thought goes something like this: We have seen a success when it really should not have tested correctly because we did not think to look for such a thing. For example, someone qualified for a really low rate on a mortgage when they should not have qualified at all. Or someone was not detected as an intruder when, in fact, that person did not belong in that area.

    Usually this happens because of some unknown fact when the program was tested in the early days and that part of the program was "certified" as being Green. Later, however, something crept in unawares and that code should have detected something that it did not know about, something that the original programmers did not know about as being an error. Beta Error!

    Anyway, just a comment about something that comes about because of poor planning on the part of the "so-called" architects of the system who did not KNOW about the pitfalls of something happening that could be know by those who wrote the program, only by those who have been in the business a long time and can recall the pitfalls of simply slinging code and NOT having a War Room Development Process, of depending on the Testing to catch all of the problems that could happen.

    SDG
    jco

    ReplyDelete
  2. James,

    I completely agree with you, that on an ideal world, 100% scenario/code test coverage and, even better, formal proof of code/rules correctness would be the definitive goal and the solution for all the bugs users face.

    Although, in real world, setting feasible goals on a cost/benefit basis is the best development teams can aim for, it seems. And even that is proving, sadly, sometimes to be hard to achieve in some companies.

    Business Rules technologies, being relatively new to the field of software development have even less tooling for that purpose and you are right we need to invest in that area. We, at Drools, added such tooling for version 5 and are working to make it the best of the breed for the next release cycle. Keep an eye out and feedback always welcome.

    Cheers,
    Edson

    ReplyDelete
  3. After reading this web site I am very satisfied simply because this site is providing comprehensive knowledge for you to audience.
    Thank you to the perform as well as discuss anything incredibly important in my opinion. We loose time waiting for your next article writing in addition to I beg one to get back to pay a visit to our website in



    Selenium training in Chennai
    Selenium training in Bangalore

    ReplyDelete
  4. This comment has been removed by the author.

    ReplyDelete
  5. It's a nice article, Which you have shared here about the Benefits of (Automated) Testing. Your article is very useful for those who are looking to electrical tag testing at an affordable price. I would like to thanks for sharing this post here.

    ReplyDelete
  6. The article is so informative. This is more helpful
    software testing training courses
    selenium course Thanks for sharing

    ReplyDelete
  7. It’s great to come across a blog every once in a while that isn’t the same out of date rehashed material. Fantastic read.
    software testing training chennai | software testing course chennai

    ReplyDelete
  8. I have been following your post for a long time. I always found it very interesting and valuable. keep posting it is really helpful.
    Cloud Migration services

    Aws Cloud Migration services

    Azure Cloud Migration services

    ReplyDelete
  9. We are a part of the success story for many of our customer's successful cloud Migrations.
    Vmware Cloud Migration services

    Database Migration services

    ReplyDelete
  10. Thank you for the informative post about Security challenges in AWS , Found it useful . cloud migration services have now become secured and with no-risk

    Lia Infraservices

    ReplyDelete
  11. I am really impressed with the way of writing of this blog. The author has shared the info in a crisp and short way.
    Cloud Migration services


    Best Cloud Migration Tool

    ReplyDelete
  12. Nice Sharing..! I have been following you for a couple of months now but this is my first time commenting on a blog post. Thank you for sharing your knowledge and experience with us. Keep up the good work. Already bookmarked for future reference.

    software testing company

    ReplyDelete

  13. This is an awesome post. Really very informative and creative contents.
    WordPress website development company in Chennai

    ReplyDelete