Well, you know how it is. You have been testing an application for some time now. One fine day, your manager walks over to you. He tells you that the management (or your client) is interested to get the functional tests automated. Since you are very familiar with the application, you should suggest the most suitable automated software testing tool for the purpose. You may know about your application’s technology and about a number of automated testing tools. However, you may not be clear about how to do justice to the evaluation exercise without burning time over this extra work. View the video, How to Select Automated Testing Tools. Then read on.

How to select the best automated software testing tool quickly

Evaluation of automated software testing tools can be a challenging and effort-prone task. Over the years, I have successfully analyzed, compared and evaluated a number of automated functional testing tools for applications written in various technologies. Below, I will describe the process for evaluation that I think is balanced in terms of effort and results.
1. Define your requirements

The primary requirement is that the automated testing tool should work (in other words, be compatible) with the application under test. There may be other requirements such as:
a. Ease of use
b. Available documentation
c. High quality customer support
d. Cost effectiveness (i.e. affordable price of license)

See the more comprehensive list of requirements towards the end of this post. Take your pick and/ or add requirements that are specific to your project, client or organization.

The requirements are rarely equally important. This means that some requirements are more important than others. After you choose your requirements, assign a weightage to each requirement using a scale, say 1 to 10 (1 being the worst and 10 being the best).

2. List the tools

There are many lists of automated functional testing tools on the web. One example is here. It is important to cast your net wide. Otherwise, you may miss a testing tool that beautifully suits your purposes. Later, someone (your manager, your colleague or your client) may ask why you did not consider another “obvious” tool.

3. Create a scorecard

Creating a scorecard need not be complex. Just list each of the requirements on the X-axis and the tools from your list on the Y-axis. See the sample. Add more columns and rows as required.









Look up the documentation on each tool's website. You could also browse the popular software testing/ QA forums e.g. www.sqaforums.com to get a feel of the kind of problems test automators generally face with a particular tool. Fill in the score (along with optional notes) in each cell. Use a consistent scale, say 1 to 5 (1 being the lowest and 5 being the highest), in each cell. You should have the raw data available at this time.

4. Analyze the scorecard

Next, you create the total scores for each tool. In order to get the total score, take the following steps for each row.
a. Multiply each score with the respective requirement weightage.
b. Add up all products in step a.

See the example scorecard with dummy scores. Note that the score for Tool1 is 4*8 + 5*5 + 2*10 = 77.







5. Try the short-listed tools

After you analyze your scorecard, you should see the tools that satisfy your requirements more than the others do. Short-list the top tools. You should select at least the top two tools (in order to give you some choice) e.g. Tool2 and Tool1 in the above table and at most the top three tools (in order to spare yourself too much effort).

It is the norm for tool vendors to offer trial or evaluation copies of the automated testing tools. Download and install the evaluation copy of each of the short-listed tools. Use each tool with your application. Create and run a few tests. Explore each of your requirements (given in your scorecard). Soon, you should have a good handle on:
a. the suitability of each short-listed tool with your application
b. your comfort level while using the tool

6. Present your results

You should create an automation demonstration with each of the short-listed tools. At least, you should create the demo for the top tool so far. Try to cover concepts related to as many requirements (from your scorecard) as you can. For example, if keyword driven testing were a requirement, it would be useful to have a keyword driven test within your demo.

With your scorecard data and your demo, you should be confident in presenting the results of your evaluation to any stakeholders.

Please fee free to use any information in this post for your purpose. Kindly let me know your thoughts on this evaluation process.

Possible requirements from the automated software test tool

• System requirements for tool installation
• System requirements for test creation
• Platforms supported for test creation (including platforms supported by using add-ins)
• Popularity (wide-spread use)
• Ease of use
• Object recognition
• Object management
• Integrated development environment
• Test debugging
• Integration with version control software
• Data driven testing
• Keyword driven testing
• System requirements for test execution
• Platforms supported for test execution
• Error recovery
• Reporting
• Documentation and training
• Customer support
• Load testing capability
• Test management capability
• Cost effectiveness (i.e. affordable price of tool and add-ins licenses)