Monday, September 27, 2010

Testers Should Drive Software Testability

Software testability is a key aspect to allow the detection of difficult to uncover defects in software. Software testability supports the testing process and facilitates the creation of better quality software.

Software testability can be described as the probability that a piece of software will fail on its next execution during testing if the software includes a fault.

Testing and testability are complimentary: testing can reveal faults (testability cannot) but testability can suggest locations where faults can hide from testing (something testing cannot do alone).

Software testability must be designed into the software as it is developed. Therefore, it is an attribute of the software that requires close development cooperation with test.

Designing for testability requires that software is designed with a greater ability to fail when faults do exist.

James Bach proposes a set of Heuristics of software testability:

  • Controllability
    The better we can control it, the more the testing can be automated and optimized. 

  • Visibility
    What you see is what can be tested.

  • Availability
    To test it, we have to get at it.

  • Simplicity
    The simpler it is, the less there is to test.

  • Stability
    The fewer the changes, the fewer the disruptions to testing.

  • Information
    The more information we have, the smarter we will test.
A given piece of software will or will not hide a given fault from testing. The software testability of software will determine whether that fault is easily detectable by testing or not.

The more controllable the software, the more we can automate. The more we can automate, the less likely that human error will allow a defect to escape into the customer's hands.

Software testability is extremely valuable where functionality cannot be tested using a black box methodology. Software testability is tightly aligned with white box testing. Software testability must be designed into the software, so tester knowledge of any incorporated testability is required.

By including and improving the testability of algorithms in a software product, it will allow the testing organization to add automated and optimized white box tests to the automated test suite which will immediately fail and highlight a newly introduced bug.

For example, a particular algorithm when triggered in debug mode could print to STDOUT a specific message communicating that it is executing. If that STDOUT message is not detected by automation, the test will fail, thereby highlighting that the algorithm is no longer executing.

In black box and customer focused software testing, only those aspects of the software that are observable can be tested by the testing organization. The inclusion of debug information on specific software features will increase the ability and ease with which to test those software features which are not observable via black box techniques.

The simple inclusion of debug information will facilitate automated tests being developed to test whether an algorithm is executing or not. These automated tests will execute each and every build so if a regression is introduced it is detected immediately, a bug is filed, and development is made aware of the bug sooner rather than later.

The automation of these types of tests will also free up manual testing time so that a tester's time is spent on testing that cannot be carried out in an automated fashion, thus increasing test coverage of the software product.

By setting the tests up in automation, it also excludes the possibility of human error in not detecting the existence of a bug.

Software is said to have high testability if it tends to expose faults during exploratory black box testing, producing failures for most of the inputs that execute a fault. Defects can be identified and fixed quickly.

Software has low testability if it tends to project faults from detection during exploratory black box testing producing correct output for most inputs that execute a fault. These defects are extremely worrisome as customers using the software believe that everything is going as expected until down the line, suddenly the defect is exposed. This category of error can adversely affect the customer's business and thereby undermines the software's credibility and trustworthiness.

The goal of increasing the testability of software is not just to detect defects but more importantly, to detect defects as soon as they are introduced. Thus, reducing the cost and time to fix the bug and producing higher quality software each build of the release.


Wednesday, September 22, 2010

Brain Rules

The best book I've read this year is "Brain Rules: 12 Principles for Surviving and Thriving at Work, Home, and School" by John Medina.

It is very enjoyable to read, not one of those books where you bribe yourself with a cup of tea and a biscuit if you just finish the chapter you're currently reading. It's easy to get engrossed in the text and not realize that hours have gone by.

The 12 principles are:

  1. Exercise
  2. Survival
  3. Wiring
  4. Attention
  5. Short Term Memory
  6. Long Term Memory
  7. Sleep
  8. Stress
  9. Sensory Integration
  10. Vision
  11. Gender
  12. Exploration

I'm not giving away. A lot of material is available for free at However, you'll get a lot more out of the material if you read the book too.

So the question remains: What does this have to do with testing?

Well, testing is an intellectually demanding task where concentration, focus, being alert is essential if you are to be effective. Based on what I learnt in this book I made some changes.

For example, I am a lark! I am at my most productive in the morning. Why fight that? So I get into work before 8am every morning to absolute joyous silence (apparently larks are rare in engineering and testing – who knew?) and I get a good 3 or 4 hours of productive work done. If I test during this period, I am far more likely to uncover high priority bugs. To allow for this interrupted time, I've rescheduled all my manager-type meetings to the afternoon.

Another example is how I've integrated little walks into my day, even if it's just going all the way to the canteen (a good 10 minute return journey) to get coffee instead of making it on my floor. These little walks recharge my brain and boost my brain power. I come back to my desk and easily refocus and apply myself to my work.

Etc, etc, etc.

Read the book – it really will open your eyes to the physiology and evolutional biology behind how we think and work.