Monday, June 8, 2009

Testing Tribulations

Testing is an essential component of the system development lifecycle. However, typically too much emphasis is placed on it to make up for earlier inefficiencies. I deliberated on this topic two weeks ago in the “Building Quality” post where I recommended that quality has to be built into a product or service by performing each step in the lifecycle correctly with frequent checks and balances. That being said, testing is an important part of the overall lifecycle and must be given the importance it deserves.

My goal in this post is to highlight some of the incorrect approaches to testing that I have observed during my travels. I do not propose to go into the details of proper testing methodologies here, as that is beyond the scope of a blog post and moreover is readily available on the internet.

As mentioned, one typical incorrect approach is to rely on testing to rectify poor requirements and design magically. Another common mistake is to produce inadequate documentation and information during the earlier stages resulting in a challenge for testing staff to fully understand the scope of the testing required. This, then, results in poorly prepared test plans and test cases which result in incomplete and inadequate testing that ultimately allows defects to appear in production.

Even if an organization chooses to not incorporate frequent checks during its entire lifecycle, at the very least, care must be taken to ensure that the testing team is given the information that is required for them to complete their task successfully. This consists of involving representatives of the testing team in the requirements discussions at the requirements stage. Test personnel should also review requirements documents and provide feedback from a testing perspective. Design and development should also involve the test team in the creation of the functional specifications and obtain review and feedback from them. Testers should be involved in the display of mockups and prototypes as well, in order to adequately prepare for the testing to come. In this way, when the time for testing actually arrives, the test team will have a deep understanding of the product to be tested and have well thought out test plans and test cases produced. This, then, will result in thorough and detailed testing performed that will effectively detect defects in the product.

The second misstep commonly made is the inadequate allocation of resources to the test team. I have personally worked at a company where they had 3 testers when they really needed 20. As might be imagined, not only was the testing inadequate, it was an extremely stressful and frustrating time for the testers most of whom left the company. Management often does not have an accurate idea of how time consuming writing and executing a test case can be. Management also often insists on operating with the best case scenario forecasted as if simply wishing for the best will result in the best actually happening. Adequate resources must be planned for right at the beginning, during the project charter and made available at the correct times for testing to be performed appropriately.

The third mistake frequently made is that testing is often the first candidate for outsourcing. A very important rule of thumb for outsourcing that I advocate is that an organization should never outsource a functionality that is not under a high level of control and maturity at its home location. If you don’t have it under control at your own place, how on earth are you going to make a success of it when it is on the other side of the planet? Things will only go from bad to worse. However, management taken in by superficial analysis of reduced costs make the transition of testing to outsourced locations far from home which then results in more problems.

The fourth mistake typically made is the excessive trust on automation and other “tools” to bring about magical savings in cost and time. I have covered this issue in my previous posting titled “Automation Angst” which explains that automation can easily result in increased time and cost if improperly implemented. Care must be taken to setup automation for only those situations that will provide a positive return on investment.

With these mistakes being made, testing usually (and naturally) performs inadequately after which it comes under fire. I have always been fascinated by the blame always falling on testers while incompetent business analysts and programmers that are also responsible (if not more so as they are the ones who introduced the defects in the product in the first place) get away scot free. Perhaps readers might comment on this observation? This pressure results in test personnel leaving the organization which only worsens the situation as new personnel have to then be hired and brought up to speed.

Truly, testing must be set realistic expectations and supported in a mature and educated way by management as opposed to the typical “test us out of trouble” mentality. It is in the organization’s best interest to make the right choices as the competitive nature of today’s world will not allow for this kind of immaturity.


  1. Wonderful article Vivek,

    Testers are blamed because testers in some cases treated as bug creater and not the Bug reporter.
    There has to be a mechanism in the process it self which will measure the gaps and root cause for a defect rather than measuring the fault.

  2. Improper or Inadequate reviews in initial phases (Requirements & Design)of project life cycle has been the problem we also faced.

    Also the Code Review techniques are so weak & also sometimes ignored which adds to this plight of the testers.

    But after when we have introduced a practice of DPC(Defect Prevention Coordinator) at project level & a DAPG(Defect Analysis & Prevention Group) at the organization level, we were able to arrive the root cause more effectively, control defects earlier than we did.

  3. Vivek,

    I specialise in creating complex mathematical systems and have found that automation tools do hugely increase the quality of the codebase, but don't necessarily decrease the time or resources taken to test.

    Automation tools are just one of the ways to test, and as you said you should not place excessive trust in them.

    It should also be remembered that as well as replay tools such as QTP, automation tools cover the xUnit family as well. This is an automation tool that is really used by the development team rather than the test team, but used correctly it ensures that a large percentage of bugs are found by the developers themselves and never get a far as the test team.

    In a previous project we had a rule that if a bug was found then a Unit test needed to be written, this will then catch the bug early in the testing cycle if it re appears.

    Tarik: I covered the problem of testers being blamed in a post of mine. The basic premise of the article was a mind shift to celebrate bugs rather than pass blame, and how this can improve quality. The post can be found here: Software Has Bugs – Celebrate Finding Them