Smart automated testing scripts

Here is a question to ask yourself. Is your automated test script just a translation of your manual test script? Or does it do something clever?

Let me explain with the help of a very common example.

Say that you are automating the test of the login function of your application. Your application has two textboxes, ID and Password on the login page. You design your auto test script to read the ID and Password from a data file and go about its business (e.g. enter these values in the fields, click Sign On, verify that you get the homepage or the error page etc.).

But, if you think about it, a checkbox for Staying Signed On is quite common on the login pages of web applications. If you do add this, as an optional parameter to your auto test script, you may get two benefits. One, you may be able to use your existing script to test the log in functionality of other applications. Two, if this checkbox is added to the login page of your application, your script requires no change.

Let us continue with this example. Your auto script needs input test data. Say, you provide five sets of this:

a. Valid ID and valid Password

b. Valid ID but no Password

c. No ID but valid Password

d. Invalid ID but valid Password

e. Valid ID but invalid Password

But, if you think about it, just providing a valid ID and its valid Password should be enough for your script to run with each test data combinations. For set b, it does not use the Password. For set c, it does not use the ID. For set d, your auto script may well be able to generate an invalid ID by using a GUID or timestamp and valid Password. Likewise for set e.

Keep going with our example. When testing the login functionality, you are likely to have many pairs of ID and Password data. Several tools provide features to allow iteration of your auto test script with each test data pair (some even allow one to select the pairs sequentially or randomly). If your automated testing tool does not provide this functionality, you may want to have your auto test script iterate through each test data pair.

Think about it. Smarten your automated test scripts :)

Tips to handle the challenges you may face as a software testing professional

Here are the possible solutions to common problems that you may face during your work as a software tester. These tips are based on my discussions with hundreds of software testers over the last several years.

Problem	Tips to handle
Hectic schedule due to unforeseen delays, parallel projects or resource constraints	Plan your testing tasks assigning them priority according to their value or urgency.Execute these tasks from highest priority downwards.Many a time, it is not possible to get more time and it makes sense to stay calm, accept the need of the extra effort needed and put that effort in.
Inadequate knowledge (technical or functional)	In the long-term, one has to build the required knowledge in order to perform well. Until that time, one should use all the available resources to get rid of this handicap. Useful sources of help are colleagues possessing the required knowledge, project documentation and the wealth of information available on the web (e.g. product literature, FAQs, forums, blogs and so on).
Poor or incomplete requirements	The tests one executes should cover each requirement. Discussions with business analysts/ system analysts/ developers/ support personnel/ other testers, prior project requirements, design documents, comparison with prior application and similar applications are just some of the sources one can use to get a solid handle on the entire requirement set.
A problem with other teams e.g. development	Such a problem is highly specific to the particular situation. It is often useful to first understand where the other person is coming from, solicit help from one's seniors or managers and try to resolve the situation prudently before it becomes big.
Adhoc test process or re-work	Any process has certain inputs, certain steps and certain outputs. In order to minimize own re-work, review each input (e.g. functional requirements, test case format, available test data) before accepting it, perform each task carefully (informing your manager/ client and the people downstream about the progress) and help each output get accepted by the stakeholders (e.g. development team, management and release team). Other than this, it is often useful to "be in the know" regarding the developing situations in the project in order to prepare for them ahead of time and avoid work which would need to be repeated.

Test Automation Metrics

Based on my understanding, the following are the automated software testing metrics that I think are useful. My list is a work in progress so use your own judgment before using these metrics to analyze your progress.

Automation Development

1. Number (or %) of test cases feasible to automate out of all selected test cases - You can even replace test cases by steps or expected results for a more granular analysis.
2. Number (or %) of test cases automated out of all test cases feasible to automate - As above, you can replace test cases by steps or expected results.
3. Average effort spent to automate one test case - You can create a trend of this average effort over the duration of the automation exercise.
4. % Defects discovered in unit testing/ reviews/ integration of all discovered defects in the automated test scripts

Automation Execution

1.  Number (or %) of automated test scripts executed out of all automated test scripts
2. Number (or %) of automated test scripts that passed of all executed scripts
3. Average time to execute an automated test script - Alternately, you can map test cases to automated test scripts and use the Average time to execute one test case.
4. Average time to analyze automated testing results per script
5. Defects discovered by automated test execution - As common, you can divide this by severity/ priority/ component and so on.