Monday, April 11, 2011

How to do end to end exhaustive testing?

Testing a software application (except maybe a very simple program a few lines long) may well be an impossible task due to large number of:

1. All possible inputs
2. All possible input validations
3. All possible logic paths within the application
4. All possible outputs
5. All possible sequences of operations
6. All possible sequences of workflows
7. All possible speeds of execution
And the above for just with a single user
8. All combinations of types of users
9. All possible number of users
10. All possible lengths of time each user may operate the application
And so on (we have not even touched the types of test environments on which the tests could be run).

However, it is possible to exhaustively execute your test suite using the following tips:

1. Your test suite should have test cases covering each documented requirement. Here my assumption is that each requirement is documented clearly.
2. The test cases should be specific, concise and efficient. Each test case should have clear and unambiguous steps and expected results.
3. The configuration data, input test data and output test data should be clearly specified.
4. You should have a clean and stable test environment in which to execute your test suite.
5. In a perfectly working application, it should be possible to execute each test case in the suite.
6.. Each confirmed bug (found during testing or found by the client) should result in another test case written or an existing test case updated.
7. Important: You should not assume the correctness and completeness of your test suite by yourself. Review of such test suite by peers, business people, managers, clients and users may provide you valuable inputs to correct your test suite.
8. Discipline in maintaining your test suite and executing it would go a long way in preventing bugs leaked to the clients/ users of your application.