Software Testing Space

Database Normalization: What to test for Second Normal Form?

2011-11-06T20:00:00.000+05:30

In the last post, Database Normalization: What to test for First Normal Form?, I explained database normalization briefly. You also saw the tests that should be executed to check the first normal form (1NF). In this post, let us understand the second normal form (2NF) and the tests that need to be executed for checking it.

A table that is in 2NF is already in 1NF. Additionally, each column that is not part of any candidate key depends on the entire candidate key (and not a part of it). Now, let us see the examples of some tables that are not in 2NF and how to convert them to 2NF.

Database Normalization: What to test for First Normal Form?

2011-10-29T20:35:00.000+05:30

Designing a relational database to minimize data redundancy (and therefore maximize data integrity) is called normalization. The concept of data normalization was introduced by Edgar Codd, right in the years after he invented the concept of the relational model of storing data. There are various degrees of normalization 1NF (First Normal Form), 2NF, 3NF and so on. Each degree of normalization is stricter than the previous one e.g. if a table is in 3NF then it is automatically in 1NF and 2NF. In this article, I will explain the First Normal Form and what to test for it. Articles on testing the other normal forms will follow.

Understanding sections of Performance Test Scripts

2011-10-28T22:28:00.000+05:30

Performance test scripts model the virtual user's expected interaction with the system. A performance test script is usually created within the performance testing tool. The default performance test script generated by the tool needs to be re-factored, parametrized, co-related and unit tested before it can be used in a performance test. Each performance test script contains various sections. It is important to know about these in order to create robust scripts that work correctly.

Test your knowledge of SQL queries

2011-10-22T20:42:00.000+05:30

Structured Query Language (SQL) is the programming language used to find out and modify the data stored in a relational database. Knowledge of designing SQL queries is important in software testing. Take the test below and find out your level of knowledge and the areas in which you should improve. Each question has exactly one correct answer. There is no need to consult any reference for answering these questions.

1. SELECT statement can fetch data from _______ table(s) of the database.
a. exactly one
b. at least two
c. one or two
d. any

2. What values can be included in the SELECT expression (the list following the SELECT keyword)?
a. Any columns
b. All columns
c. Computed values e.g. Price * 10
d. All of the above

3. Which function gives the total number of rows in a table?
a. SUM
b. COUNT
c. ROWCOUNT
d. This has to be done indirectly by executing a SQL query (e.g. SELECT * FROM Authors) and noticing the number of rows.

4. Which of the following SQL queries is correct?
a. SELECT * FROM Books WHERE Price BETWEEN 10 AND 25
b. SELECT * FROM Books WHERE Price BETWEEN 10, 25
c. SELECT * FROM Books WHERE Price BETWEEN (10, 25)
d. SELECT * FROM Books WHERE Price >10 AND Price < 25

5. Which JOIN clause returns only the matching values from two tables?
a. CROSS
b. INNER
c. LEFT OUTER
d. RIGHT OUTER

6. Which statement is correct for the GROUP BY clause?
a. GROUP BY allows grouping by only one column
b. GROUP BY needs to precede the WHERE clause
c. An aggregate function needs to be specified based on the column specified in GROUP BY
d. HAVING clause can be used in place of GROUP BY clause

7. What is true about Normalization?
a. It avoids data duplicities within and across tables.
b. It is easier to extend the database structure of a normalized database.
c. A normalized database structure is better than a de-normalized one when the SQL queries against it cannot be predicted in advance.
d. All of the above.

8. Which of these SQL queries is correct?
a. SELECT * FROM Employees ORDER BY LastName + FirstName
b. SELECT * FROM Employees ORDER BY LastName ORDER BY FirstName
c. SELECT FirstName, LastName FROM Employees ORDER BY LastName, FirstName DESCENDING
d. SELECT FirstName FROM Employees ORDER BY LastName, FirstName

9. Which of these statements is incorrect for the UNION operator?
a. Both SELECT statements have the same number of columns.
b. The UNION operator returns values that are duplicated in the two resultsets.
c. The column names returned by the UNION operator are taken from the first SELECT statement.
d. Either of the two SELECT statements can have WHERE, GROUP BY, HAVING and ORDER BY clauses.

10. Which of these is valid for a correlated sub query?
a. It is specified in the WHERE clause of the outer query.
b. It is specified in the FROM clause of the outer query.
c. It uses value in the outer query in its WHERE clause.
d. It is mentioned in the outer query's SELECT clause.

Click the Read More link for the correct answers.

Risk Management in Software Testing

2011-10-09T22:12:00.000+05:30

Risk management is a critical activity in software test planning and tracking. It includes the identification, prioritization/analysis and treatment of risks faced by the business. Risk management is performed at various levels, project level, program level, organization level, industry level and even national or international level. In this article, risk management is understood to be done at a project level within the context of software testing. Risks arise from a variety of perspectives like project failure, safety, security, legal liabilities and non-compliances with regulations. An important thing to understand is that risks are potential problems, not yet occurred. A problem that has already occurred is an issue and is treated differently in software test planning. Risk management in software testing consists of the following activities:

What is SQL Injection?

2011-10-04T17:44:00.000+05:30

If you have read my earlier article, Code injection attacks, you would have some idea about SQL injection attack. This post explains SQL injection in detail so that you may understand it well.

What is the SQL injection vulnerability? Vulnerability is a weakness in the application software under test that can be attacked to cause the application (or even the underlying operating system) to behave in an undesirable manner. The SQL injection (SQLi in short) vulnerability lives in the middle-layer or the database layer of the application. It exists when the application executes a dynamic SQL query against the database without validating, escaping or rejecting the unexpected inputs given by the attacker. These inputs become a part of the dynamic SQL query and are executed against the database.

What is the SQL injection attack? It occurs when some text or even another SQL query is inserted into the application's SQL query. Attacks can be successful or unsuccessful depending on the application and the underlying database. A successful SQL injection attack may show confidential data to the attacker, allow the attacker to impersonate another user, increase the attacker's privileges to higher levels, insert/ modify/ delete data in the database tables or even perform administrative operations on the database like shutting down the database instance.

With this background, let us see examples showing SQL injection.

10 ways to ensure that your team members excel and grow

2011-10-02T18:53:00.000+05:30

This post is about the softer skills of software test management. It is about how to have your people excel in their jobs. It is something about which I feel strongly. The end result of a project is not the only important thing. Even more important is the career benefit to your team member. Is it possible to run a project such that throughout the project, your team member matures his skills, his attitude and his professionalism? Executing projects consistently like this will ensure that your team member grows professionally. Better performance on subsequent projects will be a given. So, how does a test manager go about having their team members excel and grow? It's not easy. It's also not very difficult. Here is how.

Checklist for Accessibility Testing

2011-09-27T19:50:00.000+05:30

Accessibility is the attribute of a software application that makes it possible for people with disabilities use it. Accessibility is very important because a large number of potential users have limited abilities or disabilities. Examples of disabilities include visual impairments (from colorblindness to partial sight to complete blindness), deafness (partial or complete hearing loss), mobility impairment (inability of use hands or other parts of the body) or neurological (ADD, epilepsy etc.). Examples of limited ability people include people with limited education, old people with medical conditions and children. Thankfully, a number of assistive devices (e.g. screen readers, speech recognition software and Braille terminals) are supported in a number of operating systems. As testers, it becomes our responsibility to ensure that people of limited abilities or with disabilities are able to use the assistive devices with the application that we are testing. So what things we must test to ensure that the software application is accessible?

A/B Testing of web applications

2011-09-25T20:57:00.000+05:30

Many websites use a type of software testing called A/B testing or split testing. The objective of this testing is to determine and positively influence the user experience on the website. It involves testing distinct design layout options of the website. A/B testing is also performed on non-web elements of the website such as emails. Many large companies use A/B testing. So, what is A/B testing?

Code injection attacks

2011-09-24T20:47:00.002+05:30

If you are going to do software security testing of applications, you must be aware of possible code injection attacks. This is especially required when testing web applications because they face a hostile environment, the internet. Code injection means adding extra code to an executing application with the purpose of modifying the application behavior. This extra code can be in the form of HTML, Javascript or SQL or even unhandled type of text (e.g. special characters and long strings). Here are the types of code injection attacks.

Key Computer Security terms

2011-09-22T14:22:00.000+05:30

Since I am planning to write articles on computer security testing in the coming time, I thought about mentioning some important computer security terms. It would be good if you know and understand these terms which are commonly used in computer security.

More guidelines on how to write test cases that will be automated

2011-09-11T20:21:00.000+05:30

Everyone knows that a strong house can be built on a strong foundation only, never upon a weak one. This post is in continuation to the earlier post, How to write automatable test cases? Test cases here mean "manual" test cases, the kind that a tester can execute against the application under test by hand. Each of the following guidelines is also applicable to create valid test cases that would be executed, so there is no special effort here to prepare such test cases for automation.

1. The test case must be correct. It is obvious that the test case must be correct with respect to the workflow and expected application behavior. This guideline is especially important for automation because though it is possible for a manual tester to ignore obvious incorrectness in a test case, the automated test script would not be able to do the same. False negatives would be reported every time the automated test script is executed.

2. All the business scenarios must be covered by the test data. This refers to the completeness of the test case. The test case must contain test data for all applicable business scenarios that users would face.

3. The test case should have sufficient detail so it is possible for another person to execute it without asking questions or getting clarifications. Pre-conditions, test steps, test data, expected results and post-conditions are important components of a test case. The test case should be written with the target consumer in mind. If the automation engineer has good knowledge of the application under test, the test case components may be summarized. If not, they should be detailed out.

4. [Important] The test case must be capable of finding bugs in the current release of the application. If a test case has not caught a bug in the last few releases of the application, the likelihood of it doing so now is limited. Extra effort is required to automate test cases. So, why not automate the test cases with a high likelihood of catching bus?

5. The test case should have been accepted by the team. The test case that would be automated should not be in a raw state e.g. just whipped up by someone. It should have been reviewed/ discussed, stored in the correct file and accepted by the team as a valid test case.

6. The test case should be under version control. Placing the test case in the version control repository shows the changes made to it subsequently. Changes to the test case should be propagated to the automated test script, whether the latter is under construction or already built. Therefore, there must be a process to update the automated test script whenever the test case is revised and accepted.

Correct, complete and up-to-date test cases are important assets for any testing team. Due attention is paid to such test cases. Similar attention should also be accorded to the automated test script of such a test case. After all, its the same test case, just written in a different format. Therefore, the automated test script should be reviewed/ discussed, accepted and placed under the version control repository. The results of each execution of the automated test script should be given similar attention.

Image Courtesy: renjith krishnan / FreeDigitalPhotos.net

Do managers have bigger brains?

2011-09-10T19:57:00.000+05:30

Within the last couple of days, I have been intrigued by a news item that said that Managers have bigger brains. Then, I used Google to find out the related text. The purpose of this post is not to analyze whether or not "managing" results in a bigger brain, or even the implications of this discovery. The focus of this post is to analyze how simply using natural language to describe the result of a study can distort what is being communicated. Then, move on to the application of this analysis to requirements analysis in software testing.

Managers have bigger brains: Since this study was conducted University of New South Wales researchers, I searched the UNSW website and found this article here. My comments were:

1. I found the following text at this link "UNSW researchers have, for the first time, identified a clear link between managerial experience throughout a person’s working life and the integrity and larger size of an individual’s hippocampus – the area of the brain responsible for learning and memory – at the age of 80." So, they are not talking about the entire human brain but only one of its parts.
2. The article talks about finding a relationship between the size of the hippocampus and the number of employees managed. It does not state the exact relationship.
3. Per the article, the researchers used MRI in a sample of 75 to 92 years old. Around the middle, the article moves on to the relationship between exercise and learning and other topics presented in the symposium where this study was presented.

This news item also appeared on other websites such as MSN India here.
4. I found the text "Staffers agree or not, managers do have bigger brains, says a new study." The prior article had no mention of the staff agreement to the manager having a bigger brain. So, did the research take the subjects' staff's agreement into account?
5. This news item says "Researchers, led by the University of New South Wales, have, in fact, for the first time, identified a clear link...". The previous article just mentions "UNSW researchers". So, were there teams from elsewhere involved in the research?

What can we take away from the analysis?
a. It is possible for people to over-generalize or over-specialize a description. So, we should probe further to find out the caveats and exceptional conditions. For example, a requirement may say "The user account shall be locked after 3 unsuccessful attempts". On probing further, we may find that this is true for all users, except the administrator user. The system may be rendered in-operational if the administrator gets locked out :)
b. It is possible for people to just provide references to other information, without naming it explicitly. Instead of relying on our assumptions, we should ask for the reference directly. For example, a requirement may say "Other requirements implemented in the previous release shall continue to be supported". We should ask for which release - the last major release, the last release (major or minor), the first release or any other release? If there is a conflict between the requirements of this release and the "previous release", is it okay to have the requirement of this release take priority?
c. We should question the source of the requirement. Has it come directly from the customer or system analyst or just a suggestion from somebody? In other words, is it a genuine requirement? For example, someone may suggest making reports completely configurable. It may be against the client's objective that any user comes up with any format of a report, leading to confusion. The suggestion should be declined. Of course, suggestions made by the management need to be declined tactfully, if infeasible to implement.

How I train others in software testing?

2011-09-03T14:35:00.000+05:30

Ever since I started my career in software testing, I have been asked to give training sessions on testing topics. The topics on which I have trained people include basics of software testing, writing correct and re-usable test cases, rapid test execution, bug tracking systems and defect management to test automation approaches, performance testing, system security testing and test methodologies.

I do have a couple of advantages. First, two years experience in giving software training during my initial career. Second, I have always been keenly interested in how people learn (probably because of 6 teachers in my family, including my mother). I am life-long student of the psychology of learning.

Let me explain the approach I use during training. This is both for your benefit and my benefit. Your may benefit from these tips by increased recall of the participants after the session which leads to more application of the training material in projects. I will benefit by looking up my tips to ensure that I continue to follow them and build them further.

1. Know your topic
This is most important. You should know your topic really well. Not just a little more than the participants, but many times more. Why? In order for learning to take place, the participant has to believe in the superior knowledge of the trainer. For example, let us say that you are training on creating automated test scripts. You have done this before using a functional test automation tool. Someone in your class asks about parameterization and you are stuck. What will happen? What you say from that point onwards would not be credible. Therefore, give training only on topics that you know inside out.

2. Plan your training session
You should know the objectives that the training session should achieve. This information is available from the sponsor of the training. Also, if you know the participants, you can find out their current knowledge level and their expectations from the session. If you don't know the participants in advance, you should spend some time at the beginning of the training session to find this information. For example, if you are training on system security, the participants need to know the basic security concepts like information integrity, information confidentiality and information availability. The training session should cover the material required to increase the participants knowledge from the current state to the desired state.

3. Create your training material
Once you are clear on the objectives of the training, the next step is to design the sub-topics and training material. For example, if you would be training on writing test cases, you should cover the inputs to the test cases (requirements documentation, design documentation, prior test cases, test case formats etc.) and tips on how to write test cases (with all scenarios, pre-requisites, test steps and expected results). The training material can include practical examples of good test cases.
One caveat: The training material should not be too lengthy or cluttered. The main focus should be on the participants understanding the topic well. They can always look up the references if they need more details later.

4. Starting the training session
For learning to take place, the participants should be engaged in the training session. A useful way to do this is to explain the objectives of the training session in terms of what they already know and how the training will help them work in a better way. For example, if training on bug tracking systems, the objective is to use the bug tracking system more efficiently, you could mention that you will share tips on using the bug fields correctly and setting up email notifications for more efficiency.

5. Treat the topic logically
If you plan your training session well, you would have identified the logical sequence of the sub-topics. Answer questions from participants as required but stay on course. For example, if training on performance testing, the logical sequence for you may be creating the automated test scripts, parameterization and correlation, modeling the test, test execution and analyzing the results. If you have moved to test modeling and you get a question on parameterization, answer it quickly and then say you are moving back to test modeling. Also, explain any new concept or term with more than one example as soon as you introduce it.

6. Make the session interactive
People like sessions during which they can ask a question any time. If the question is related to the training session, you should answer it. In fact, you should always pause for questions on the conclusion of a sub-topic. If a question is not directly related, you could park it. For example, if you are training on functional test automation approach and someone asks you which tool is best for automated testing, you could say that it is not directly related and you would take it offline later. You always need to be respectful of the participants' time that they have chosen to spend listening to you.
One trick I use to keep my session interactive is to pretend that a term or word has slipped my mind. This forces people to think and they come back with suggestions energetically.

7. Summarize the training session
After completing the session and answering questions, be sure to summarize the main points in the training session. The material covered must be linked to the objectives of the training session. For example, if you trained others on object repositories, you could say that by now the participants should have awareness of the types of object repositories and they should be able to make an informed decision to use the best type in their project.

8. Finishing up
Spend a little more time sharing the training material, providing further references and providing a contact to answer questions later on. Also, reiterate the expectation from participants now they attended the training session.

Well, these were but some of the guidelines I use when training others. Your training session should be more productive if you use the above. What do you think is most important in making a training session successful?

Practice database testing

2011-08-21T22:39:00.000+05:30

I received several emails from the readers of the popular post, How to do database testing?, asking for advice on how to practice database testing. This post contains one example to practice the same.

Create a practice database with the following three tables, Products, Orders and OrderDetails. It does not matter which database management software you use. It is simple to anticipate the relationships here. Products table is related to OrderDetails table, via the ProductId field. Orders table is related to OrderDetails table via the OrderId field.

Now, write and execute queries to check the following:

1. Is there any product which exists in the OrderDetails table but is missing in the Products table?

2. Does any OrderDetails row have an OrderId that is missing in the Orders table?

3. Is any ProductName, Serialnumber or ProductDescription duplicated in the Products table?

4. Does any product have a negative UnitsInStock?

5. Does any order have a ShipDate which is earlier than the OrderDate?

6. Does any order have a future OrderDate?

7. Does any OrderDetails row have a Quantity value less than 1? Or a Quantity value which is fractional?

8. Does any OrderDetails row have a zero or negative SalePrice or UnitPrice value?

9. Does any OrderDetails row have a UnitPrice for a product, different from that product's UnitPrice in the Products table?

10. Can you enter Price values (UnitPrice in Products table and SalePrice and UnitPrice in OrderDetails table) in decimal (up to 2 digits after the decimal point)?

Super simple ways to impress your manager and team

2011-08-20T19:52:00.000+05:30

Would it not be great if you could impress your manager and team by putting in just a little more effort? Here are some ideas for building a great perception of you. Note that 3 out of the 8 ideas relate to communication, very important to impress others.

1. Maintain stable work timings

It is important for others to know when to reach you. You can make it easier for them if you are always in office, say before 9 a.m.. Maintaining fixed in-timings shows commitment to work. When you are on vacation, make sure that you set up an Out of Office message notification with information of another person to contact, if urgent.

2. Ask lots of questions during meetings

If you are attending a meeting, you might as well ask questions. Use the 5 W's (What, Why, Where, Who, When) and How to frame questions to get more information for you and others. Asking questions during meetings shows that you are engaged with the topic and eager to understand more.

3. Volunteer for high-visibility assignments

Volunteers are asked for in meetings or written communications. Be eager to take up a task that only one person can do, but which will benefit the whole team. Also, look for small tasks that nobody else is doing. And complete them.

4. Communicate regularly

Whatever you have completed and are doing at present, be sure to communicate it regularly. If you maintain data about your daily tasks, it will be possible for you to communicate daily. Then summarize weekly, monthly and even annually, and communicate those summaries also.

5. Communicate using the correct vocabulary

Different companies and teams have different preferred vocabulary. If you use the specific terms preferred by your manager and team, your communication will be easier to understand. Also, these words show that you are engaged with your team and company.

6. Keep building material for later communication

Have you seen well-structured emails from others with a number of original thoughts? And wondered how much time and effort they spent on creating that email? You, too, can do the same. For important emails, instead of writing whatever comes to your mind and shooting off the email, collect your thoughts in the draft first. Don't send it. Keep updating the draft when you get any good ideas. Soon enough, you will have a worthy email that will impress others.

7. Ask for inputs

When you create a project deliverable (test plans, test cases, automated test scripts etc.), always ask for review comments from your manager and team. You may not get review comments every time. But, whenever you do, it is a good input to refine your deliverable. Also, this shows your commitment to quality.

8. Say "Hi"

Finally, take a couple of seconds to wish your colleagues whenever you meet them. This helps others think of you as a courteous and approachable person.

Please let me know if you liked these ideas and the benefit you got from them.

Image Courtesy: renjith krishnan / FreeDigitalPhotos.net

Why bad software testing happens?

2011-08-12T23:54:00.000+05:30

It is possible for software testing to go wrong in a project. Look out for these red flags in your project and make corrections immediately to avoid bad testing.

1. Customer inputs ignored

2. Poor knowledge of the application under test

3. Application technology ignored

4. Incorrect test strategy

5. Old test process followed mechanically

6. Unorganized testing

7. Poor observation and analyses

8. Insufficient communication

9. Lack of perseverance

10. No self-critique/ correction in the team

Checklist to review automated test scripts

2011-07-18T20:43:00.000+05:30

Reviews are a great way to find common problems with a number of test automation artifacts, such as automated test scripts. By an automated test script, I mean script code implementing a test case's test steps and expected results. The automated test script may be written in any programming or scripting language. It may be generated by a functional test automation tool or written by hand.

Here is a checklist that you can use to review your or others automated test scripts. How? Simply customize the questions according to your specific needs.

Inputs

1. Does the script implement each step of the test case?

2. Does the script implement each step in the correct order?

3. Does the script implement each validation given in the test case?

4. Does the script have each validation after the correct step?

Execution

5. Does the script logic call the correct functions? Are these functions called with the correct arguments?

6. Does the script take application input values from the correct input test data file(s)?

7. Does the script execute user operations against the correct objects?

8. Does the script give the application sufficient time to transition to the correct test state e.g. the correct screen?

9. Does the script validate application output values against the correct output test data file(s)?

10. Does the script validate the state of the correct objects?

11. Does the script load only the necessary objects in memory?

12. Is the script free from unnecessary delays?

13. Does the script handle application errors as designed?

Miscellaneous

14. Does the script have a unique identifier?

15. Is the script free of any magic numbers or magic strings?

16. Is the script commented at each appropriate place in it? Is each comment correct?

17. Is the script free from unnecessary or extra code?

Let me know if you find this checklist useful in your reviews.

Save time by automating common testing tasks

2011-05-21T21:24:00.000+05:30

When the term automation is mentioned, it is common for people to think of automated tests. But, time and effort can also be saved by automating other tasks such as generating test data and reporting test results automatically. A few factors should be considered for effectively deciding whether to automate a task or not. Last year, we had fruitful discussions on this topic within the STS group.

Let us see some candidate tasks for automation. Testers spend a lot of time on these tasks. If such tasks are fully or partially automated, the saved time can be spent on testing the system.

1. Test identification based on risk or other factors

List all the available test cases along with related features, components and priorities. The automation can take the inputs e.g. impacted features and generate the list of test cases that should be executed on the new application build.

2. Test estimation

Automation can be used to help estimate the test effort and duration. This can be done based on the preferred estimation approach. Whether by querying the historical data for actual efforts and durations, or by applying a custom formula to estimate test effort and test duration.

3. Generation of test data

Build a library of business rules for your test data. Build the initial seed data manually. The automation can use the seed data and generate test data based on the chosen business rules. Since test data generation is usually consumes much time, the automated generation can be performed ahead of time. Then, the pre-generated test data can be used directly during test execution.

4. Generation of bug reports

This automation can be built into the test automation framework. Whenever an automated test script confirms an error, it logs into the bug tracking system and reports a bug with required information. Such as bug title, steps to reproduce, test data used, environment used and so on.

5. Generation of test reports

This automation can execute queries against the test management system (and bug tracking system, if different) and generate test reports. Even distribute them by email or publishing to a website.

6. Release notes preparation

Release notes contain both static and dynamic data. This automation can execute queries on the test management system to retrieve the dynamic data such as features passed, bug fixes passed and known bugs.

Before prioritizing the automation development, always analyze the following factors.

a. Degree of automation achievable

b. Skills required to automate

c. Effort required to automate

d. Number of proposed users

e. Effort required to train users

f. Effort required to execute automation

g. Effort required to maintain automation

h. [Important] Manual effort saved

In the future, I see a number of such tasks automated with the help of vendor tools or bespoke in-house automation.

Let me know if you liked this post. I would love to know your thoughts on this topic.

Image Courtesy: Ambro / FreeDigitalPhotos.net

Software testing myths

2011-05-15T17:19:00.000+05:30

Myths abound in the software development industry. Software testing is no exception. People tend to hear such statements and pass it on to others minus the full context. Not only that, they even tend to use such myths when making own decisions, especially when in hurry.

1. A ratio of 1 tester to 5 developers is enough to get good test coverage.

This myth may be true in long-term new development projects with multiple stages of testing (e.g. SIT, alpha, beta and UAT). This ratio may fall short in:

a. Maintenance projects with highly inter-dependent components, requiring testing of all impacted components

b. Projects requiring huge effort to create an independent test environment

c. Projects requiring extensive regression tests (due to, say, business impact, contracts or regulations)

2. Anyone can test software.

Anyone can test software if they know what to do. A professional tester is well-versed in the project requirements, software testing concepts, test design and test execution techniques and is also an assertive communicator.

3. Developers can test software better than testers. After all, they are the ones who developed it.

While developers know the code they develop intimately, they may not be the best people to test it (or test code written by other developers), especially at the integration level or system level. Testers execute their tests in a clean test environment with a fresh mindset, doing both positive and negative testing. Developer testing complements (and does not substitute) testers' testing.

4. Test cases are only derived from requirements.

Requirements are only one input to write test cases. A professional tester uses many other inputs like design documents, standards, checklists, prior test cases, past bugs reported by the customers and prior versions of the application/ similar applications.

5. Testers need only execute a set of test cases.

A static test case suite is unlikely to lead to the discovery of new bugs in the application. Before execution, the test suite should be updated by removing the useless or redundant test cases, completing or correcting the existing test cases and adding new test cases to increase the likelihood of discovering fresh and interesting bugs in the application.

6. Tests already executed need not be repeated in the release.

It depends. If the new builds do not impact the application areas covered by the tests already executed. Much more likely, some tests need to be re-executed to re-gain confidence in those areas.

7. Testers test every permutation and combination of inputs a user can provide to the application.

This is not possible except in very trivial cases. Even testing all possible inputs of a form with a few text, date and numeric input controls within the given time may well be impossible. What testers actually do is to design the minimum number of input sets that tests each input control. Specific inputs to each control are limited by using testing concepts like equivalence partitioning.

8. Testers can test equally well (if not better) when the time is short.

Nobody can perform a thorough job if rushed. Testers are no exception. If the available time is reduced too much, a tester prioritizes the tests to execute and keeps executing the higher priority tests until time runs out.

Image: Idea go / FreeDigitalPhotos.net

On simplifying testing

2011-05-02T23:36:00.000+05:30

Lately (actually since a while), I have been looking for ways to increase my productivity. One of the ways I recently came across is simplification. Why?

1. Better focus, especially on creative or complex tasks
2. Less stress, due to elimination of so many unimportant tasks
3. More time on hand

So, how to simplify our software testing tasks? Here are some examples. Design your own according to your unique situation.
1. Analyze your work and establish what is important in your role. Put it down in a sentence. For example, Maintaining your test suite, executing it and reporting defects may be critical to your role. Formatting your status report or pointless gossiping with your colleagues is definitely not.
2. Allow yourself time to plan. Plan simple i.e. think about the easiest set of actions that will complete your task.
3. Guard your time. If someone interrupts you, dismiss the interruption asap. If someone approaches you for help, consider helping once you are done with your tasks or say no.
4. Own your work. If some task had a problem and requires re-work, don't blame other people. They would respond with their reasoning and you would end up spending time on this exchange. Then attend to the problem. Instead, just fix the problem at once and move on.
5. Use software apps to your advantage. See examples.
a. Block your mail calendar for important tasks in advance and in order of importance. This would allow you to see your upcoming tasks for the next day, week or month and you can better prepare for them.
b. To be responsive, check your email a limited number of times in the day. If you leave it on, you would get distracted every time you get a new email. And you will have the urge to stop your current task and read/ take action on the email you got.

This was a short and simple post and it made me feel better writing it.

Interviews of Thought Leaders - Freddy Gustavsson

2011-04-28T23:34:00.000+05:30

Having started his career as a software developer, my dear friend and a key member of the Software Testing Space group, Freddy Gustavsson made the leap into testing in 2001 after realizing that a carefully designed test approach will greatly increase the chance of success in any software project. He is interested in all parts of the test process, and has experience from several international projects. He works as a consultant for System Verification in Gothenburg, Sweden, where he specializes in test strategy, test design, test automation, test process improvement and test education. Freddy is an ISEB/ISTQB certified test analyst who also teaches courses on software testing and serves as a member of the internal Senior Advisory Board.

Here is the interview. Enjoy and learn.

[Inder] Did you start your career with software testing or have you held other software roles as well?

[Freddy] I actually made somewhat of a detour to reach my current position in software testing. At first I had no intention at all to work in the IT industry. Instead, I wanted to pursue a career as a teacher of languages at college level. So I studied German linguistics for two years at the university and prepared for some advanced English studies. However, my passion for web development grew stronger, and I decided to make a shift. In 1999 I started working as a web developer for one of Germany's e-business providers. There I learned tons of good stuff about development, projects, team work and testing. In 2001 I joined the QA team and became a tester. Later, I've finished a 3 year university education and a B.Sc. in Software Engineering because I wanted to understand my profession truly. I've also taken complementary courses in project management, usability, accessibility etc. Additionally, I spent one year teaching programming courses at university level, which was useful.

[Inder] Not all people appreciate software testing as a highly intellectual work; they feel that it is quite easy to test. Can you talk about few technical challenges that you have faced in your career?

[Freddy] It's a known fact that some employers will view testing as an entry point to development. It's as if they told their candidates: "We're sorry, but you're not qualified to play with the developers yet. But let's move you into testing. If you work hard and prove yourself worthy, then some day you might actually be allowed to work in development." Even worse is the situation where employees, who were rejected by other disciplines, are brought into testing, because managers feel that they "do least harm" there. This approach is just so wrong.

Testing is by no means a trivial task. Anyone who has worked as a professional tester knows this. However, not all people coming from other disciplines will understand testing well enough to realize its challenges. They might think of testing as "happy testing" or "randomly clicking around", not realizing that testing requires a controlled process. The presence of structure is vital to our job. At the core of the tester's job is the comparison of actual and expected results. However, this is surrounded by numerous items such as plans, strategies, procedures, methods and tools. Designing and implementing all of this in an effective and efficient way requires excellent business, technical, administrative and social skills.

Common technical challenges include quickly learning new applications and tools, understanding the environment in which each tool operates. Sometimes you also need to learn new protocols, scripting languages, or get an understanding of complex system architectures. This is part of the tester's job. Being curious and eager to learn new things is definitely helpful.

[Inder] Looking back at your software testing career, what have been some of the highlights?

[Freddy] The first highlight would be the recognition of testing as a profession. The second would be the insight that some of my carefully crafted test cases were actually detecting failures. The developers would lovingly refer to me as The Merciless Tester. Their indignation over each found problem would not last long. Instead they asked me to do more testing on their code. Although I have occasionally, and unintentionally, upset a few people through my work, in most cases the work with other disciplines has been both interesting and rewarding. The third highlight would be the move into consulting and a focus on test strategy, which might be the best job ever. Number four is the mentoring and educative role where I get to spread the word and (hopefully) make other people interested in learning more about our craft. Highlight number five would doubtless be this interview. ;-)

[Inder] Do you find testing enjoyable? How can software testing professionals derive more satisfaction from their craft?

[Freddy] Yes. Just like for other professionals there are a number of things that motivate me to do a great job. The most important is probably the feeling of making a valuable contribution to the client. Sometimes also to the community. For instance, a few years ago I was on the system test team for the national Swedish command and control system. The system was built for operation in emergency situations where ambulance, police or fire brigades might be needed. Any failures in the software might possibly have catastrophic (life or death) consequences. In that case every defect found and removed would benefit the whole community. As a tester you knew your work made a difference.

Having a good, respectful working relationship with coworkers and managers also makes it to the top of my list. Another important thing is the possibility to control your job, to be able to suggest ideas and improvements.

[Inder] What are some good resources for people wanting to enter software testing or establish them in this career? What activities can one do in their spare time to enhance their testing skills?

[Freddy] I want to refer to a good seminar on this topic, which I attended at the EuroSTAR 2010 conference in Copenhagen. Markus Gärtner from Germany gave a presentation on alternative paths for self-education in software testing. A number of options were presented:

Using social media (e.g. Twitter, LinkedIn, web sites, forums)
Learning to program (e.g. scripting languages, design patterns)
Reading books on testing
Joining online testing courses
Participating in testing challenges
Participating in organized problem solving activities like testing dojos, weekend testing or the Miagi-Do school of testing.

[Inder] What are your future career plans?

[Freddy] I look forward to an exciting future in which software testing will undoubtedly play an important role. Personally, while keeping an eye at the entire field of testing, I plan to specialize further into the areas that interest me most: test strategies, test process improvement and test design methodologies. I also plan to extend my teaching assignments in the future, since lecturing is a great way to spread the knowledge about testing while learning a lot myself. As they say: you learn as long as you teach.

How (not) to drag your team's productivity down?

2011-04-23T23:33:00.001+05:30

If you are fortunate enough to lead or manage a team of software test engineers, consider it a valuable responsibility. Check yourself if you find yourself adopting any of the following tactics. If you do, you can be assured that your team would be working far below their potential. And that would not the only problem you face. I am going to refer to the hypothetical Lead or Manager as LM for my examples below.

1. Poor communication

The LM knows about the incoming project or test run. She has been involved in planning, meetings with other stakeholders and knows a lot about it. But, the quantity of the information passed to the team members is so low or the quality so poor, that they get stuck at every step. They make assumptions and their work de-rails. Unless, of course, they check with her at each step. Takes up their time away from project work.

A variant of poor communication is an overdose of information. The LM floods the team members with documents not long before the start of the project. Worse, the info flood takes place just as the project starts. Now, a team member has to make one of two poor choices - stick to the schedule unprepared OR prepare and lag behind schedule.

2. Stalling

The LM is too process-oriented. She just loves checklists. A team member can consider himself successful only if she completes each of the 1,000 tasks in the checklist successfully. Did I mention that this is in addition to his testing task?

3. Unnatural competition

The LM promotes competition among the team members. Who writes test cases fastest? Who creates bug reports that developers always agree too? Who provides the LM the test results in the exact format that she likes? All this means that not only each team member has to do his work; he also needs to watch what others are doing. Nobody wants to be the last one, so the team members take short-cuts. Project work suffers.

4. Back-stabbing

The LM talks sweetly. After receiving so much appreciation (verbal and written - but always one-to-one) from her, a team member can never suspect the true feelings the LM has bottled up inside. These feelings are released in the reports the LM gives to her manager, other managers, customers and HR. The team member does come to know about the real feedback provided by the LM, but only when the damage is already done.

5. Feedback given unequally

Her praise is concise and in private. Reprimands are public affairs listing the mistakes committed in the past, the current lapse and hopelessness for the future. Needless to say, such reprimands make the team member quite uncomfortable and unsure of how to see his co-workers in the eye again.

A variant is when the LM provides no feedback. The team member has no idea whether he is performing well or poorly. The team member does not know what is round the corner. He does not know what he would be doing tomorrow.

6. False promises

The LM says anything to get the work done. It works, but only the first few times. In time, the team member is disillusioned.

7. Taking undue credit

The team members take the difficult project as a challenge. They work professionally and very hard. They surmount the obstacles. Even though tired, they don't cut corners. The LM takes all the credit for her "leadership" in time of need.

8. Scape-goating

The LM schemes. She already knows the personalities of her team members. She creates a Plan B (and Plan C) if things go south. When they do, she lets any one problem develop, gathers substantial "data" incriminating a chosen team member and dumps the entire situation on his head.

9. Unwillingness to change

The LM is aware of the negative effects some of her actions have on the team. But, it has worked for her in the past and she sees no incentive to change. So, if he is able to work with her, the team member makes the necessary mental adjustments or start looking elsewhere, in time.

Final words - As human beings, we tend to make the best possible choice. As I mentioned above, if you lead or manage a team, you have a big responsibility. Keep in mind that your actions affect not only yourself but also your team members. Take the correct action, even if causes pain in the short-term and is difficult. It will benefit both you and your team.

What is software regression?

2011-04-14T23:43:00.000+05:30

Things are not as good as they used to be.

Before one can do an informed regression testing, it is important to understand software regression, which can happen after an event that changes the system. Software regression is deterioration in the software. Such decay can be functional, meaning one or more functions working earlier no longer do so. Or, it can be non-functional, for example, the software becomes slower/ outputs less or becomes (more) vulnerable to security threats.

The events that change the system include the change of the executable code of the software (e.g. when a new software release is deployed), change of the underlying components (third-party components or operating system components e.g. when an update/ upgrade/ patch is applied), change of the configuration of the software (e.g. with hardware/ network changes) or change in user data.

Some examples of regressing changes to the software include:

a. Bug fixes incorrectly done - The developer attempts to fix a bug without truly understanding it. The result may now be that the software works in a particular scenario but fails in other scenarios not tested by the developer.

b. Bug fixes incompletely done - The developer fixes a bug with a temporary solution. Or the developer fixes a bug in only one part of the code, missing to fix it in all the impacted parts of the code.

c. Incorrect version control - The latest changes to the code were not made on the latest version of the code, overwriting some prior changes.

Examples of regression caused by component changes include:

a. The new component version functions differently, requiring the software to be updated to work with the new version.

b. The new version supports a different data format, requiring the software to be updated to handle the new format.

Changing the system configuration and user data is in the hands of the users. Examples of regression caused by such changes are:

a. New configuration was never tested by the developer and is not supported. It results in intermittent errors, system slowdown up to irreversible user data loss.

b. The software was never tested with the current user data volume, resulting in very slow performance.

As you can see, software regression can be caused by a variety of reasons. Regression testing should discover such regression (functional or non-functional) after examining the particular changes to the system.

How to do exhaustive testing?

2011-04-11T21:27:00.000+05:30

Testing a software application (except maybe a very simple program a few lines long) may well be an impossible task due to large number of:

1. All possible inputs

2. All possible input validations

3. All possible logic paths within the application

4. All possible outputs

5. All possible sequences of operations

6. All possible sequences of workflows

7. All possible speeds of execution

And the above for just with a single user

8. All combinations of types of users

9. All possible number of users

10. All possible lengths of time each user may operate the application

And so on (we have not even touched the types of test environments on which the tests could be run).

However, it is possible to exhaustively execute your test suite using the following tips:

1. Your test suite should have test cases covering each documented requirement. Here my assumption is that each requirement is documented clearly.

2. The test cases should be specific, concise and efficient. Each test case should have clear and unambiguous steps and expected results.

3. The configuration data, input test data and output test data should be clearly specified.

4. You should have a clean and stable test environment in which to execute your test suite.

5. In a perfectly working application, it should be possible to execute each test case in the suite.

6.. Each confirmed bug (found during testing or found by the client) should result in another test case written or an existing test case updated.

7. Important: You should not assume the correctness and completeness of your test suite by yourself. Review of such test suite by peers, business people, managers, clients and users may provide you valuable inputs to correct your test suite.

8. Discipline in maintaining your test suite and executing it would go a long way in preventing bugs leaked to the clients/ users of your application.

Conventional wisdom applied to software testing

2011-04-09T20:01:00.000+05:30

Here are the conventional English proverbs as I see applied to the field of software testing:

1. A journey of a 1000 miles begins with a single step.

A test run of a 1000 test cases begins with a sanity test.

2. Genius is 10% inspiration and 90% perspiration.

Software testing is 10% test preparation and 90% test execution.

3. As you make your bed, so you must lie in it.

As you write your test cases (good or poor; complete or incomplete), so you must execute them.

4. A bird in hand is worth two in the bush.

A confirmed bug is worth two suspects.

5. A chain is no stronger than its weakest link.

The quality of an application is no better than that of its most buggy component.

6. Discretion is the better part of valor.

Review is the better part of executing a task.

7. Empty vessels make the loudest noise.

People who least understand software testing speak of it in the most black-and-white terms.

8. An ounce of prevention is worth a pound of cure.

One bug fixed in the beginning is worth 16 updates.

9. A penny saved is a penny gained.

One test case re-used is a test case written.

10. Where there's a will, there's a way.

Where there's an application, there are bugs yet to be found.

Hope you enjoyed these :). If you like any other English saying, let me know. I will translate it to software testing.

Checklist to test software release notes

2011-04-03T12:16:00.000+05:30

It is a norm to distribute release notes along with a software release. Software release notes are similar to the product literature that you get when you buy a physical product. You may vary the contents depending on whether the software release is a major release or a minor release, an upgrade to the application or just an update, customized for a specific customer or available generally. Find answers to the following questions to test the release notes that you receive before software testing.

Main information

Does it mention the application name and the correct version number?
Does it correctly list the features released and/ or bugs fixed in the software?
Does it mention the correct software vendor name and date of the release?
Does it mention the release numbers it requires as prerequisite or the release numbers it supersedes?
Does it mention each of the available distribution modes (e.g. website download, email and disc)?
Does it mention the correct distributed media (e.g. install package, executable programs or scripts) in which the software release exists?

Release feature information

Does it explain each new feature (summary, benefits and details of how it works)?
Does it explain bug fixed in this release?

Supporting information

Does it list the minimum and recommended hardware, software and network requirements to deploy and use the release?
Does it provide steps to backup the existing configuration and user data in the system?
Does it provide detailed steps to install the release?
Does it provide steps to restore the configuration and user data of the previous release?
Does it provide the contact channels in case the software does not install or installs with problems?

Others

Is the release notes available in a file format that can be viewed by the customer?
Is the file format consistent with prior releases?

How to identify high performers in your team?

2011-03-13T23:59:00.000+05:30

Having worked with several high performing software testing professionals, I find that it boils down to two things. A high skill level and a positive attitude.

And both are required. A high expertise but no willingness to apply it in the project would result in meager performance at best. And tons of positive attitude but a low skill level would also not translate to solid work. Having said that, attitude is the more important of the two. Why? Skill level can be increased by putting in time and effort but it is much more difficult to change a poor attitude.

I have found consistent high performers to exhibit the following four characteristics:

1. Demonstrates a higher technical knowledge

This person knows more. And it is evident in a number of ways, the things this person says or does, the questions he asks or the responses he gives.

One day we were having a team meeting. I brought up the topic of pair-wise testing to discuss whether it would be useful in the project. To my pleasant surprise, a high performer of our team already knew about this test design technique. He ably explained this technique to the team and we ended up identifying test items that would benefit from pair-wise testing.

2. Communicates more

High performers communicate more than other team members. You can see them asking more (insightful) questions when new work is being explained. They also keep others abreast of their current status and any current/ foreseen problems.

I fondly recall one high performing team member. Whenever I assigned him some work, I would definitely get a response email. The email would mention any unresolved questions and his schedule for this work. Then the work would be completed per schedule and I would get another email summarizing what had been done, the approach used and next action items.

3. Delivers more or higher quality work

By definition, high performers deliver better performance. You would find them doing one or more below consistently:

a. Finishing tasks ahead of schedule

b. Finishing more tasks than initially planned for them

c. Finishing tasks with quality better than anticipated

4. Loves challenges

All high performers love a challenge. In fact, lack of challenges may be boring or even de-motivating for them. So, it is important to raise the bar for them from time to time.

One new addition to our team faced a series of challenges. The industry domain was totally new to this person. A couple of months later, he was using the same vocabulary as the rest of us. He came from a manual testing background. He had to work on the automated testing project. Again a few weeks later, he was very hands-on with the automation tool, scripting business workflows with ease, correlating automated test scripts and even able to perform a reasonable analysis of the test results. Every time he faced a new challenge, he came on top of it.

Now you know about the characteristics of high performing software test engineers. These qualities are not different from what one may expect from high performers in any other field. Think about each of your team members in light of the above four qualities. You would know who are the high performers are. You can also develop the above qualities to increase your own performance. Good luck.

How much time should a tester spend testing every day?

2011-03-06T00:36:00.001+05:30

Not so long ago, I went to attend a walk-in interview. While we were all waiting to be called, I struck a conversation with a test engineer who had also come to be interviewed. One of the things we talked about was the daily routine of this person. Here is a rough daily schedule:

9:00 a.m. Arrive at office and settle down

9:15 a.m. to 9:45 a.m. Check emails

10:00 a.m. to 11:00 a.m. Team meeting

11:00 a.m. to 1:00 p.m. Test the application (2 hours)

1:00 p.m. to 2:00 p.m. Lunch break

2:00 pm. to 3:30 p.m. Test the application (1.5 hours)

3:30 p.m. to 4:00 p.m. Log bug reports (0.5 hours)

4:00 p.m. to 5:00 p.m. Enter test results in the test management system and enter timesheet

5:00 p.m. to 5:30 p.m. Reply to questions on prior bug reports

5:30 p.m. to 6:00 p.m. Write status report for the day

6:00 p.m. Leave office for home

It shocked me. This person was spending 50% or less of his daily time on testing. This does not even take any ad-hoc meetings, training, test environment down-time or usual interruptions into account. If what this person said is true and it is a good representation of the daily schedules kept by testers in companies, I don't think it is okay. Why?

1. Testing software is the core function of a software test engineer. Most of the time should be spent on this important task.

2. Non-testing tasks e.g. team meetings, email exchanges and compiling test statuses are important but not at the cost of testing time.

3. Time should not be spent on redundant activities. A 10-minute stand-up summarizing what everyone did yesterday is fine but listening to the details of what each of the 10 team members did yesterday is not the best use of time.

4. There is duplicity of effort. If data is accurately entered in the test management/ project management system, it should be sufficient to produce any desired test status report.

However, a test engineer may not be able to guard his time because many of these non-testing tasks are driven by their managers. If a manager insists on long meetings and detailed status reports everyday, which task do you think would suffer? Yes, it is testing that would be impacted. It may not be as deep, wide or inspired as it could have been.

If you are a test manager, you may want to think about how to minimize the burden of non-testing tasks of your testers. Keep them accountable to find problems with the software while allowing them to focus on their core function. Most of the time.

How to be more successful in software testing interviews?

2011-02-27T01:15:00.000+05:30

Hope that you liked my previous post, How to write a resume that is selected? If you are really interested in a position and they have short-listed your resume, the next step is to get selected in the interview :). Here is how to increase your chances of success in interviews.

A couple of thumb rules first:

1. For each interview you attend,

Make the effort to understand (I mean internalize, not just read) the job description.
Gather more information about the job, the project and the organization.
Anticipate questions and prepare excellent answers ahead of the interview.

2. Interview more :)

Pre-interview

You should reach the interview venue well in time, go over your notes (see the 1st thumb rule) and relax before the interview starts.

Interview

Now is the time to use your preparation, thinking skills and communication skills. Every question you are asked is an opportunity for you to demonstrate your suitability for the position. When you answer a question, do not blurt out the first thing that comes to your mind. Understand the question and frame a very good reply in your mind before speaking. But don't take too long to respond; a pause of few seconds is okay. Provide specific details from your own experience. Aim to surprise the interviewer positively. Here are some examples:

Question. What tests can you design for a common object, say an office chair?

An excellent response would highlight the test design approach as well as really good test ideas. For example, one could say that I would analyze the documented requirements, designs, contemporary objects, standards and use my own ingenuity to design the tests. Some test ideas would be:

1. Does the chair accommodate the weight of a large person for extended hours and continuous movement? Does the chair allow 360 degree movement, height/ back/ neck/ armrest adjustments? Is the chair comfortable? Ergonomic?

2. Is the chair good to see and feel? Does the color match the office decor? Does it require maintenance/ cleaning infrequently?

3. On being used daily, how long can it stay in perfect condition? What are the steps to install and adjust the chair? and so on.

Question. Can you describe a bug that you discovered?

A good response would indicate the process used to find the bug and some details of the bug itself. The chosen bug should be a high severity bug (e.g. one impacting financial calculations, one affecting system availability to a large number of users), not caught by the pre-existing test suite and potentially disastrous if released to the customers.

Post-interview

This is the time when you have to judge the success your interview.

If you do not receive outright acceptance, it is possible that there is an objection. You need tact to get this from your interviewer. Possible ways to know the objection are:

1. Ask for the next steps and timelines - If there is an objection, you may get a vague response e.g. "We will get back to you".

2. Summarize each part of the job description and highlight how suitable you are with respect to each part. Note the reaction of the interviewer to each part keenly.

3. If required, be direct and ask for the objection.

It may not be possible to remove the objection due to say salary/ position fitment, regulatory requirement (e.g. security clearance), customer requirement or specific instructions from top management. However if you truly feel that the objection can be removed, talk about it professionally. Hollow promises will not move a professional interviewer. Give examples from your experience when you faced a similar challenge and how you solved it. State your plan how you would remove the objection quickly.

If there is no objection and you are accepted, good luck with the new offer and congratulations.

How to write your test cases faster?

2011-02-13T00:44:00.000+05:30

In one of my prior projects, I was writing test cases with the deadline only a couple of days away. It made me anxious at the time and I put in longer hours to finish the task. But, I decided to later search ways to complete writing my test cases faster. Here are my thoughts. I hope that they benefit you too. Especially useful if you work on projects with very tight deadlines.

1. "A picture is worth a thousand words."

Tables are a great way to consolidate test cases. Especially, when you have to test a number of combinations. Compare the ordinary version (TC1, TC2, TC3 and so on) with the tabular version (only TC1) for the login test case example.

When you use the table, your focus will shift from writing the sentences to designing the test combinations.

2. Refer other documents in your test cases.

If another document (e.g. a user manual) already lists the clear steps to perform an operation the application, you can refer the appropriate section within your test case. No need of pasting the section within your test case. This not only reduces your effort. It also makes your test case more maintainable. If the source document changes, there is no need to update your test case since it just contain a reference. However, the source document should be readily available.

You can also use this approach across test cases.

3. De-clutter your test cases.

Write only what is essential. Not one word more. Separate test data from test steps and expected results. This will allow you focus better on one item at one time. It will also make your test cases cleaner.

Nice side-effects of using the above approaches include:

a. Your test cases will be shorter and hence easier to read, understand and execute.

b. Your test cases will have less duplicity making them more maintainable.

c. It will be easier for you to review them to eliminate mistakes.

In short, not only will you take lesser time to write your test cases now, but your test cases will end up with higher quality too.

How to write a high quality bug report every time?

2011-02-06T00:44:00.000+05:30

Bug reports are one of the most important deliverables in software testing. We know that a truly great bug report is the result of thorough isolation of the bug and its accurate documentation, among other things. But, it is very common to have a tight schedule. So, the software tester may not be able to spend too much time on perfecting the bug report. If they do, they run the risk of not being able to complete the planned tests in time.

In this common project situation, if you want to write a high quality bug report every time, just focus on getting the basics right. Use the following checklist to ensure the quality of your bug report. Note that few data items in your bug report may be static in a single test run. It is useful to create the bug report as a local document so that you can reuse as much of the existing data as you can.

1. Does the title of your bug report accurately summarize the issue?

2. Is the application name (and component name) mentioned?

3. Is the application version and build mentioned? Note: possible static data

4. Does it have the requirement (required behavior)?

5. Does it have the steps to reproduce the issue? Note: If you have a written test case that failed, you can paste that test case up to the step which failed.

6. Does it mention the observed behavior? Note: You can use screenshots or video clips here.

7. Does it mention all the data required to reproduce the issue?

8. Is the severity correct?

9. Are the name/ address (e.g. URL) and configuration (hardware and software configuration) of the server mentioned? Note: possible static data

10. Is the client configuration (e.g. O/S, browser) mentioned? Note: possible static data

11. Is it assigned to the correct team or person?

12. Is it copied to the correct people?

Out of the above data items, I consider title, steps to reproduce, observed behavior and required behavior as most important.

You can take the above checklist and customize it to your project needs. Execute the checklist to review your bug report before submitting it. This way, you would double-check that your data is correct and that no important data is missing. Ensure that you write a high quality bug report every time.

Is your automated test script smart?

2011-01-31T23:36:00.000+05:30

Here is a question to ask yourself. Is your automated test script just a translation of your manual test script? Or does it do something clever?

Let me explain with the help of a very common example.

Say that you are automating the test of the login function of your application. Your application has two textboxes, ID and Password on the login page. You design your auto test script to read the ID and Password from a data file and go about its business (e.g. enter these values in the fields, click Sign On, verify that you get the homepage or the error page etc.).

But, if you think about it, a checkbox for Staying Signed On is quite common on the login pages of web applications. If you do add this, as an optional parameter to your auto test script, you may get two benefits. One, you may be able to use your existing script to test the log in functionality of other applications. Two, if this checkbox is added to the login page of your application, your script requires no change.

Let us continue with this example. Your auto script needs input test data. Say, you provide five sets of this:

a. Valid ID and valid Password

b. Valid ID but no Password

c. No ID but valid Password

d. Invalid ID but valid Password

e. Valid ID but invalid Password

But, if you think about it, just providing a valid ID and its valid Password should be enough for your script to run with each test data combinations. For set b, it does not use the Password. For set c, it does not use the ID. For set d, your auto script may well be able to generate an invalid ID by using a GUID or timestamp and valid Password. Likewise for set e.

Keep going with our example. When testing the login functionality, you are likely to have many pairs of ID and Password data. Several tools provide features to allow iteration of your auto test script with each test data pair (some even allow one to select the pairs sequentially or randomly). If your automated testing tool does not provide this functionality, you may want to have your auto test script iterate through each test data pair.

Think about it. Smarten your automated test scripts :)

Practical tips to handle the problems you may face working as a software testing professional

2011-01-29T23:39:00.000+05:30

Here are the possible solutions to common problems that you may face during your work as a software tester. These tips are based on my discussions with hundreds of software testers over the last several years.

Problem	Tips to handle
Hectic schedule due to unforeseen delays, parallel projects or resource constraints	Plan your testing tasks assigning them priority according to their value or urgency. Execute these tasks from highest priority downwards. Many a time, it is not possible to get more time and it makes sense to stay calm, accept the need of the extra effort needed and put that effort in.
Inadequate knowledge (technical or functional)	In the long-term, one has to build the required knowledge in order to perform well. Until that time, one should use all the available resources to get rid of this handicap. Useful sources of help are colleagues possessing the required knowledge, project documentation and the wealth of information available on the web (e.g. product literature, FAQs, forums, blogs and so on).
Poor or incomplete requirements	The tests one executes should cover each requirement. Discussions with business analysts/ system analysts/ developers/ support personnel/ other testers, prior project requirements, design documents, comparison with prior application and similar applications are just some of the sources one can use to get a solid handle on the entire requirement set.
A problem with other teams e.g. development	Such a problem is highly specific to the particular situation. It is often useful to first understand where the other person is coming from, solicit help from one's seniors or managers and try to resolve the situation prudently before it becomes big.
Adhoc test process or re-work	Any process has certain inputs, certain steps and certain outputs. In order to minimize own re-work, review each input (e.g. functional requirements, test case format, available test data) before accepting it, perform each task carefully (informing your manager/ client and the people downstream about the progress) and help each output get accepted by the stakeholders (e.g. development team, management and release team). Other than this, it is often useful to "be in the know" regarding the developing situations in the project in order to prepare for them ahead of time and avoid work which would need to be repeated.

Useful Software Test Automation Metrics

2011-01-20T00:09:00.000+05:30

Based on my understanding, the following are the automated software testing metrics that I think are useful. My list is a work in progress so use your own judgment before using these metrics to analyze your progress.

Automation Development

Number (or %) of test cases feasible to automate out of all selected test cases - You can even replace test cases by steps or expected results for a more granular analysis.
Number (or %) of test cases automated out of all test cases feasible to automate - As above, you can replace test cases by steps or expected results.
Average effort spent to automate one test case - You can create a trend of this average effort over the duration of the automation exercise.
% Defects discovered in unit testing/ reviews/ integration of all discovered defects in the automated test scripts

Automation Execution

Number (or %) of automated test scripts executed out of all automated test scripts
Number (or %) of automated test scripts that passed of all executed scripts
Average time to execute an automated test script - Alternately, you can map test cases to automated test scripts and use the Average time to execute one test case.
Average time to analyze automated testing results per script
Defects discovered by automated test execution - As common, you can divide this by severity/ priority/ component and so on.

Database testing is interesting

2010-12-31T22:16:00.000+05:30

A vast majority of business applications store and retrieve data. Many applications use databases for this purpose, owing to a variety of reasons such as a standard and methodical way of storing information, performance and security. In addition to testing the application software, we should also be testing the database. You can avoid a number of bugs if the database schema used by your application is created correctly. What is more, a correct database schema prevents having to implement a complex logic in your application to handle the different scenarios that can lead to data problems, and consequently application bugs affecting its users.

Now let us look at a simple database that may be used by a school to store its student information. Since the number of rows and columns is small, we can inspect the data and look for problems manually. Of course, when testing tables with a large number of rows, you need to design and execute SQL queries, but let us not worry about that right now. The intention is to look for database issues, both existing and potential.

Here are the problems with this database:

1. Missing tables
One can see that the Grades (Classes) exist just as a column in the Students table. Grades are really a different entity. This school may have a Grade 3 even if there were no students enrolled in it yet. And this is not apparent from the current data. There should be another table for Grades and each row in the Students table should point to the corresponding Grade via a foreign key.

2. Missing or incorrectly named columns
The Fees table has an ID column. But what ID is this? Is it the Fees ID or the Student ID? If it is the Fees ID, then there is no linkage between the Students and Fees table. If it is the Student ID, it should be named as such to avoid confusion for the developers.

3. ID columns
The Students ID column is has a text datatype. This means that every query reading this column will have to parse the value to rid the leading zero or zeroes. A more severe problem is that text based indexes are slower than numeric indexes leading to slower performance.

4. Dates
One cannot make out the date format from the Date of Joining column values. It could be mm/dd/yyyy or dd/mm/yyyy. This means that every query on the Date of Joining column has to know the date format before it can parse and process the value.

These are not the only problems with the data presented. There are more. Can you spot them?

I hope that this analysis would have piqued your interest in database testing and you would show this interest in your own software testing.

How to write a resume that is selected?

2010-12-12T18:50:00.000+05:30

If you are looking to change your job in software testing, the first challenge facing you is to write an effective resume. That is, a resume that would get short-listed. In this post, I will share with you tips that will increase your resume's success rate. I have divided these tips among three sections - Preparation, Execution and Post-Execution.

Preparation

Going by the number and frequency of new job postings on popular job sites, you may have too many choices. However, you should not start chasing all. There isn't enough time to do so. You need to realize that the recruitment activity is a process with the following steps. The next step is taken only if the prior step is successful.

a. Sharing details of the job and the candidate

b. Review of these details

c. Interview

d. Verification and decision-making

e. Joining

Take the time to visualize your ideal job. Use the following questions to clarify your thoughts and be as realistic as you can.

a. What role would you like to play?

b. What kind of organization (small or large, product-based or service-based or hybrid, conservative or modern and so on) would you like to work with?

c. What location(s) would you prefer?

d. What kind of salary you would like to draw?

e. What working conditions (shifts/ hours, organization culture, benefits and working culture) do you prefer?

Once you have a good handle on the kind of job you would like to have, it is time to move to Execution.

Execution

By Execution, I mean the actual task of writing the resume that would be selected. Let us say that you have come to know about a job vacancy that is similar to the job that you are looking (see the Preparation phase above). Now, you should take the following steps:

a. If the job posting is in writing, read it carefully. If you received a call, jot down notes from that call. Note down each keywords in the job posting. Keywords are the essence of the job posting. For example, if you are looking to be a Software Test Engineer, the job post may contain keywords like "Test Plan", "Bug Reports", "QC processes" and so on.

b. The next step is to gather more information about this job. There are several ways to do this.

1. Browse the organization's website. If you read the relevant sections, you will have a better idea of the work they do and the vocabulary they use. Add more keywords to your existing list.

2. If you know someone working in the organization, give him or her a call.

3. Talk to the poster. They may just highlight the main points of the profile they are looking or provide extra information. Let the poster know that you are interested in the job and would be sending them your resume.

c. By now, you should have a very good idea of the job. It is time to go through your list of keywords and ensure that the keywords applicable to you are included in your generic resume. Adding a keyword once may not be sufficient. For example, if "Automated Testing" is a keyword and you have experience in automated testing, you may include it in the following ways:

1. In the summary section, mention Performed Automated Testing in XYZ project

2. In the Tools section, include the automated testing tools e.g. QTP, WinRunner that you worked with

3. In the project responsibilities in XYZ project, mention Created automated test scripts

d. Review the major sections of your resume and ensure that they are structured. The Summary section, the Organization/ Projects section and Qualification sections should be clearly separated. Follow a uniform order within each section e.g. list the organizations you have worked with in reverse chronological order and do the same for your qualifications.

e. Make sure that your resume is current. If should include your latest organization/ projects.

f. Ensure that you have provided all relevant information within each section. For example, if the job posting mentions Bachelor's degree in Engineering as a requirement, you may want to include details like College/ University/Year of Passing/ Grade if you are comfortable and it is legally acceptable to share this information.

g. If your resume does not already mention your achievements (awards/ recommendations/ recognition) during work or college, ensure that it does so now.

h. Review the job posting and your list of keywords again and read your resume. Expand the relevant phrases and sentences. Do not highlight the keywords by making them bold (this looks desperate). Reduce or summarize the non-relevant phrases and sentences.

i. After final review of your resume, you should be ready to send it to the job poster.

Post-Execution

If you get the call to schedule an interview, congratulations. If not, wait for 2 or 3 business days and call the job poster. Maybe they received too many resumes and did not go through your resume (carefully). You should the reasons due to which you are suitable for this job. Use your list of keywords to your advantage. Follow up is very useful in securing an interview if your resume is not selected automatically. If your resume is not selected even after follow-up, it is likely due to some information not shared in the job posting e.g. the organization wanted only candidates experienced in a particular industry domain. Ensure that you ask questions to unearth such information in future.

If you have made the effort to understand the job well, presented your resume in line with the organization's expectations and/or followed-up, chances are high that your resume would get short-listed and you would get a call to schedule an interview. The next challenge is to be selected in the interview. Watch this space for tips on that topic.

Solve your own task planning and execution problem forever

2010-11-07T22:37:00.000+05:30

I like doings my testing tasks really well. In fact, I prefer not starting a task to doing it in a sloppy way. Now, this created a problem whenever I was working on a project. There were many tasks to perform every day. In the past, it used to take a lot out of me in taking the actions related to my task. First, I used to understand the task and then estimate its priority (to schedule it accordingly). Then, I planned the approach to perform my task and identified each sub-task that required to be done. I performed each sub-task and tracked my progress frequently. After completing each sub-task, I reviewed it. Then, I reviewed the entire original task first with respect to each sub-task and then with respect to the original objectives. Finally, I communicated my task completion to the relevant people.

Why did I end up expending a huge mental and physical effort in performing each task? Other than my desire to really shine at the task, I found that I was repeated a lot of planning I had done in the past on similar tasks. Attempting to excel at my task is welcome. It gives me the satisfaction that I am not doing the task mechanically. However, re-thinking everything is definitely overkill. Not required in a majority of tasks. Definitely not desirable in every iteration of the task.

I needed to solve this problem. That is, I had to balance my wish to do really well with my need for speedy execution. Here is how I created my solution to this problem. First, I identified a couple of non-trivial tasks that I had to repeat. In my case, I selected 1) Review of test cases and 2) Creation of the monthly test results. Later on, I realized that this solution can be applied to any other repeated non-trivial tasks such as 1) Design test cases, 2) Execute test cases and 3) Log bug reports. Second, I made available to myself an ample time slot when I would work on nothing else but this selected task. Third, I started working on the task. But there was an important difference to the prior executions this time. I documented my approach making notes as I went along the task. For example, starting the test cases review, I listed each item that I had to plan e.g.Whether to do a high-level or detailed review or a hybrid review, Sequence in which to review the test cases, Level of detail in my review comments. While actually performing the reviews, I listed each sub-task as I remembered it e.g. Check if each requirement in scope of the test case is covered, Check if each design is covered in the test case, Check if there is an incompleteness in steps or expected results or test data, Check for duplicity within a test case or among two or more test cases.

Finally, I ended up with a nice long list of sub-tasks (with comments) that I needed to perform to do justice to my original task. Some sub-tasks were linked to the understanding of the task, some to the planning, several to the execution and the others to the own review. Now, every time I faced this task, I did not go about it in my earlier way. I got my list out, looked at the list of sub-tasks and started executing them. Whenever I remembered a missing sub-task, I just inserted it in the appropriate place in the list. This practical list was really my own task execution procedure.

How does this help you? If you are like me and want to perform due diligence on your important tasks, your own procedure will help you do just that. You will no longer need to rely on your memory alone. The second benefit is that it will reduce the planning effort required. You will just look at each sub-task in your procedure, decide if it is relevant to the current situation and if so, perform it. The third benefit is that your procedure will keep you focused and on track. If you enhance your procedure with every iteration of your task, you get the fourth benefit, which is improving execution ability. You will get the fifth benefit when you share your procedure with your team members. Your team members will improve their execution ability by benefitting from your procedure. In turn, they are likely to give you feedback that will help you enhance your procedure even more. Your procedure will also be a good means to transfer execution knowledge to new team members.

If you are excited about creating your own procedures in order to excel at tasks, you should take a few pre-cautions:

1. Do not get over-ambitious and try to create procedures for every task that you perform. If you do, your work will slow down to a crawl and you may be overwhelmed quickly.

2. Choose to write the procedures for only regular AND non-trivial tasks. Do not spend hours writing a procedure for a task that you will perform once in six months (and the project situation may be quite different at that time). Do not write a procedure with obvious steps e.g. you know that you have to fill up each relevant field in the bug report form before you submit a bug report.

3. Once you have a procedure ready, do not delete a sub-task even if you find that it is not required in the current task iteration. This sub-task may very well turn out to be important in a subsequent iteration.

If you work with a process-oriented organization, you will find that documented procedures are available for only the major software testing tasks. In other organizations, you will find that you only have high-level industry guidelines and standards to follow. In both cases, you will find your own procedures very helpful. How else do you think high performers are able to produce substantial results in half the time that you take?

Why you shouldn't be too nice?

2010-10-24T23:52:00.000+05:30

Let us say that you are starting out in a project. You know that "people like to work with nice people". Therefore, you decide to be nice with everyone in the project team. You get these benefits:

a. The team members share their knowledge with you.

b. They help you understand the software application and the process they use.

c. They introduce you to the people related to the project.

d. They invite you to project meetings.

e. They explain and share their project work with you.

So far, so good. But what happens when you have decided to be so "nice" that you have forgotten how to say no?

1. You are given responsibilities loosely related to your role that nobody else wants to take because they are "busy". Examples could be completing some old documentation, preparing the conference room for the team meeting and arranging the team lunch.

2. When you approach someone, you are advised to fix up a meeting but you are constantly interrupted yourself. The team members feel free to walk up to you or call you and discuss whatever it is they want to discuss.

3. Your schedule is modified constantly. Here you were working on a task but suddenly something else has come up and you need to attend to the new task. While you were busy completing the new task, your old task has been cancelled or re-assigned to someone else so now you need to discard your work.

4. You always work based on someone else's estimates. Examples could be that you may estimate a task taking you 8 hours but your manager wants you to do it in less than 4 hours. Or, you may want to prepare for a task and then do it but your teammate suggests that you do it directly.

Do you think that being "too nice" will affect your performance on the job? You bet it is. Just consider the four effects above:

1. You are spending your precious time on busy work that just about anyone could perform. Worse, this work may not be required at all.

2. Your concentration is being broken repeatedly. Coupled with the loss of your productive time, you end up work superficially.

3. You cannot plan ahead of time because your schedule keeps changing. You may not get good ideas immediately when faced with an urgent task.

4. You are tired or confused because you are working according to someone else' work style or speed.

The sad thing is that you may be thinking that you are working well trying to please everyone in your team even when in reality you are putting up a mediocre performance. Worse, you are positioning yourself as a "nice" person in the eyes of your team members more and more with the passage of time.

I can say these things from personal experience. Because, I too have been the prey to my "be a nice guy" approach from time to time.

The question is what should we do? Should we suddenly stiffen up and become very demanding from others? No, we should continue to behave cordially with our team members. However, we can ask ourselves the following questions:

a. What is my role in the project?

b. What actions do I need to take to fulfill my role best?

c. What actions do I need to refuse to fulfill my role best?

d. How do I guard my own productive time best (and still be respectful of others' time)?

e. How can I create or at least influence the approaches and estimates for my own work?

The only thing left for us to do is to make the changes according to our responses.

Good job performance requires confidence in our actions. This is especially true of software testing. After all, software testing is about generating confidence.

How to define and implement Test Strategy?

2010-10-17T22:09:00.000+05:30

On this October 14, I attended a web talk by Alan Page along with several others. The topic of Alan's session was Test Strategy. I would like to list the points that I saw and heard Alan make before making my own observations:

Consider the context before creating your test strategy. It is useful to consider your own situation in terms of your team's composition, their current skills, their desired skills and other goals. For example, it may be okay communicating the test strategy verbally within a small team of say up to 20 people. However, when you have a large team, it becomes useful to document the test strategy and distribute it so that everyone is on the same page.
After considering your context, the next step in the process is your fact-finding and assessment. This helps you answer questions like how is testing at present, how would it be different in the future, would other parameters change and how could the team change to meet the future requirements.
A useful way of clarifying your thoughts is to map your facts to goals. What is your current state (fact) and what is your desired state (goal)?
The journey from your Current state to Desired state may not be a straight jump but a series of steps. However, each step should aid the transition away from the Current state and towards the Desired state.
Once the strategy is in place, just take the desired actions. Track and review the progress and adjust course if required.

It was a clear and thought-out presentation. You can view the talk here. It should take you about 30 minutes to listen to it. Now my questions and comments.

Each action (even the tiniest one) taken in an organization should contribute to the organization's objectives positively. How does the test strategist ensure that each step outlined in the test strategy maps to the organization's objectives and ultimately to its vision? A test strategist should be keenly aware of their organization's business objectives. Further, the test strategist should be aware of other factors such as the current customer experience, competition and the direction the industry is moving.
Implementing a test strategy in a sizeable team is no mean task. Other than piloting actions and showing supporting data to other team members, what are the ways to smoothen the implementation of a test strategy? It may require sessions to explain the test strategy to each team member, arranging and executing any training they may need and providing the supporting processes and tools to the team help take action to move to the Desired state. Explaining what is in it for them, recognition of good performers and championing the test strategy may also help attain buy-in from the team members.
How does the test strategist know that they have arrived and it is time for the next strategy? By ascertaining if the desired state is institutionalized (data consistently points to the desired state, team members discuss about the Desired state as the Current state and team members have become a little complacent).

Why software testing should be like good journalism?

2010-10-15T22:51:00.000+05:30

Don't you just love news? News is an important part of our lives. It keeps us informed about the happenings in the world around us. Further, news helps us update our mental model of the world so that our understanding matures from that time onwards. Now, news are gathered and communicated to us by journalists. Let us see what can we apply from the world of journalism to software testing?

Similarities between journalists and software testers
A journalist has a network of sources to inform her of interesting events.	A tester has a bunch of tests to inform her of interesting facts about the system under test.
In order to stay on top of events, the journalist stays in constant touch with her contacts.	In order to stay updated about the current quality of the system, a tester executes her tests regularly.
The journalist seeks to strengthen her network by adding new contacts or replacing her contacts with better ones.	The tester seeks to strengthen her test suite by adding new tests or enhancing her existing tests.
As soon as the journalist comes to know about an interesting event, she starts working to gather all possible information on it.	As soon as the tester comes to know about an interesting bug, she starts working to gather all information about it and isolate it.
A busy journalist has more than one story to work upon at a time. She follows all of them as they develop but focuses her attention on the ones most critical to her readers/ viewers.	A busy tester has more than one bug to work upon at a time. She follows each of her bugs but focuses her attention on the ones most critical to the system's stakeholders.
A journalist does not work alone but seeks help from others to create the best possible news story.	A tester also relies on her team to provide her insights into areas requiring more testing, re-usable tests and better testing techniques.

Now let us see what we can learn from not-so-good news stories.
A late new story: Nobody likes stale news. Stale news makes us miss opportunities and disappoints us.	Our tests should be freshly executed. Our bug reports should be promptly reported giving timely information to our stakeholders to understand, digest and decide actions based on our bug reports.
A poorly labeled news story: We decide to read/ view a news story based on its headline. It wastes time if the headline is not a true representation of the news story and we stop reading/ viewing the news mid-way.	Each of our bug reports should contain a true summary of the bug. This would enable our stakeholder effectively decide whether they want to read it it or not.
An incomplete news story: We don't feel good when we have a number of open questions after reading/ viewing a news story. It requires additional effort on our part to seek missing information from elsewhere or follow the story further.	Our bug reports should contain each required information item e.g. the steps to reproduce the problem, the actual result, the expected result, the test data used and the test environment in which the bug was discovered.
A repetitious news story: We don't want to waste our time on reading/ viewing details that we already know about.	Each of our bug reports should contain unique information and not duplicate information that is already known via other bug reports.

I am confident that the next time you read or watch news, you would think about the things that you could apply or avoid in your own software testing. Let me know if you find other lessons for us to learn from good journalism.

How fast would you progress in your software testing career?

2010-10-03T22:39:00.000+05:30

If you are a software testing professional and wonder about your career growth, take the short quiz below. Select just one answer, A or B, for each question. Add up the number of As and Bs in your answers.

1. What is the most rewarding effect of your software testing career?

A. Money

B. Job satisfaction

2. How do you plan your testing tasks?

A. You prefer your tasks to be planned by someone else and told to you.

B. You analyze your tasks, prioritize them and try to provide the most value in each task.

3. What is your driving thought when you test software?

A. Finish the tests that you have been assigned and go home

B. Test carefully to not miss any problem

4. What do you worry about the most before submitting a bug report?

A. That it is not an invalid or duplicate bug report

B. That it contains enough information to help identify a real problem

5. How do you handle free time (say, between projects or assignments) at work?

A. Socialize/ catch up on personal tasks/ take it easy

B. Analyze own performance and prepare to deliver increased contributions

6. Do you think about software testing in your personal time?

A. Never, what is the need?

B. Of course, it is an important part of my life

What kind of progress should you expect for yourself?

Excellent: 0 As and 6 Bs

Good: 1 or 2 As and 4 or 5 Bs

Average: 3 As and 3 Bs

Below average: 4 or more As, 2 or less Bs

Notes: 1. Money can only motivate you to a limited extent; the intrinsic job satisfaction is what drives you in your career constantly.

2. In order to contribute the most, you need to identify your most important tasks and perform them to the best of your abilities.

3. Each test that you perform is an opportunity for you to excel. Do not squander such opportunities.

4. Always focus on the positives. Instead of worrying too much about making mistakes, focus on your real responsibility.

5. You should always be planning and working towards progress in your career.

6. If software testing is a part of your personality, you cannot help thinking about it from time to time.

How to create new test ideas quickly?

2010-09-19T20:55:00.000+05:30

Test design is an important task for software testers. However, even "solid" requirements do not cover all aspects of a software product or application. As testers, we should be creative in designing tests that cover each and every system characteristic that is of importance to our clients.

In order to learn more about test design, I decided to run an experiment last month. I wanted to see the test cases that could be designed for testing a common object of everyday use, a tea cup. I posted a discussion on this topic in the Software Testing Space group.

The experiment was a success. The group members came up with more than 75 test ideas. I too participated actively in the discussion and came up with more than 75 other test ideas on my own. I learnt a lot from this experiment. In this post, I will share one learning with you.

To begin with, I just had some basic test ideas. Ideas like does the tea cup hold an acceptable quantity of the beverage, does it prevent the loss of heat while the tea is being consumed, is it possible to operate the tea cup without burning one's digits or mouth and whether is it possible to prepare (i.e. wash and wipe) the tea cup for the next use.

Other STS group members added more test ideas related to the physical characteristics of the tea cup (e.g. its weight, leaklessness, its looks, the properties of the material used in its construction and its inter-operability with other objects such as its plate). Now, the physical characteristics category is a gold mine of test ideas. I came up with more test ideas now on

Operation

Does not get caught with other objects e.g. other utensils or garments while handling
Does not lose material, shape or any other physical characteristic when washed
Does not have any hard to reach areas (difficult to wash zones)

Inter-operability

With a kettle or tea vending machine

Safety

Stability (with respect to its height, weight and center of gravity) while lifting a filled tea cup and placing it down
"Unbreakability" during use e.g. when dropped on the floor or when squeezed by hand
Size and shape of pieces (e.g. small and pointed versus large and smooth) that it breaks into

Looks

Has a good shape design, symmetrical or otherwise
Has visible logo
Handles extreme temperatures without physical distortion

At this stage in the discussion, I found myself thinking hard to get fresh ideas. Sure, I would come up with a few fresh test ideas. But I also found myself coming up with several test ideas same as or similar to the ones already on the table. I felt that there were a lot more tests that I had not considered yet. But how to get at them quickly?

It was at this stage that my fellow STS group member, Freddy Gustavsson added to the discussion. Freddy suggested more test ideas but the thing that struck me about his response were the categories he listed. Things like Changeability (modify the tea cup) and Replaceabiltiy (repair a broken tea cup).

I found a sizable list of system quality attributes on Wikipedia. After looking at these attributes, I could then think up with a number of fresh tests including, for instance,

Material free from harmful (reactive or radioactive) chemicals
Material does not conduct electricity
Tea cup operates well with accompanying items like cup cover and coaster

What is even more important is that you may use the list of system quality attributes to quickly design tests for any system, including a software product or application. Like I did, you could select the quality attributes applicable to your system. Add sub-attributes as desired by you. Create a hierarchy of attributes if you want. Thinking about test ideas for your system one attribute at a time gives you focus. This could enable you to come up with a number of test ideas more quickly.

A final note: Out of all the test ideas that you collect, remember to select and prioritize your tests based on the objectives and needs of your clients. The importance of a test depends on its alignment to the objectives and needs of your clients.

Why you must know your product's competitors?

2010-08-31T23:00:00.000+05:30

Why does a customer purchase (a license) of your product? More likely than not, the primary reason for doing so may boil down to

1. Getting something new e.g. increased productivity or increased efficiency or increased resources

2. Overcoming a risk e.g. miscommunication or failing to meet statutory requirements

However, your product may not be the only solution available in the market to satisfy the customer's primary requirement. If your product is not so well-known, it has to compete with leading products in the market in the particular category. If your product is the market leader, it may have to compete with products catering to specific niche markets. Even if yours is a one of a kind product, there may be a proven manual system that it has to compete with.

When the customer evaluates or first uses your product, it is no leap of imagination to think that s/he would be actively comparing it with its competitors. If you test software, you can ill-afford to ignore your product's competitors. Software testing should include not only testing with respect to your organization's or customer's requirements but also testing to check how the product functions with respect to its competitors.

Knowing the product's competitors is not the prerogative of product managers alone. As software testers, we pretend to be customers using the product. Therefore, just like customers, we should be aware of the alternative products. Only then would we come to know about how our product functions on its own and how it functions with respect to its peers.

Is it a bug?

2010-08-21T22:00:00.001+05:30

Scenario: You are a competent tester who has just joined a company and is testing on the first project there. The application you are testing is a financial web application. During test execution, you make the following observations. Rate each observation on a scale of 1 to 10, 1 being definitely not a bug and 10 being certainly a bug.

1. The logo on the home page and other pages of the application is not the client's. For example, if the client's name were 1, the logo is that of another company, say 2. The logo in the previous version of the application is your client's.

2. There is a group of links on the pages of the application. When you click any link in this group, it opens a page with the "Not Found" error.

3. You log on as a new user into the application. You make a deposit of $ 1,000 into your account. When you visit the Transactions page, it shows you your transaction of $ 1,000. However, when you visit the Balance page, it shows your available balance as $ 2,000.

Should you go ahead and report these observations as bug reports pronto?
.
.
.
.
.
.
.
.
.
.

My advice is No. All your observations mean is that you should investigate further. For all you know, there might be perfectly reasonable explanations for these observations. For example,

1. Your client is getting this application developed for a partner organization. Or, their parent organization. Hence, the different logo. The change request is already under construction and you would get it soon.

2. The group of links will be supported by an external entity. During your test, you should just have checked if each link is correct. The external entity would populate these links before the test finish date and then all the links in the group would work.

3. There is indeed a requirement which states that when a new user makes the first deposit between $ 1,000 and $ 2,000, the client organization doubles the balance by making an equal deposit into the user's account. However, you are not aware of this requirement.

The objective of this post was to help you realize how using Heuristics may lead you along a wrong path. However, I did not state this objective at the beginning of the post because I did not want to bias you.

Testing is an investigative process. We should observe the application carefully, formulate theories if what we see is not in line with our mental model, gather more information from multiple sources and analyze and test our theories before pronouncing a discrepancy or a bug in the application.

P.S. The idea to write this post came after reading, ICICI Bank ATM bug … Try it!............. by Amit Jain, a member of the Software Testing Space group.

As you wish!

2010-08-14T22:19:00.000+05:30

Once upon a time, there was a tester who had just joined a large end-user company. He had a good profile and was promptly inducted into an ongoing project to develop an application for internal use across offices of the company. Things started moving slowly for this person.

Though nobody in his team realized at the time, this tester was quite capable. He was also quite talented, hardworking and methodical in his working style. He quickly introduced himself to each of the team members (but in his hesitant way). During his initial days in the project, he made it a point to read (and re-read) all the documentation related to the project. Whenever he had questions, he made the effort to approach the relevant team member and clarify his thoughts.

Some time passed and now he was assigned some modules to test in the project. Since the tester was quite knowledgeable about the requirements by this time, he could understand the existing test cases quite well. He was also able to see the short-comings in these test cases and made refinements to his own copy of the test cases before executing them. He found defects. In fact, he found a lot of defects. This led to the problem.

The Development Lead on the project was an influential person. He was always "in the know". He was also widely recognized in the company as a competent developer and business domain expert. But somehow, he did not like so many defects being logged by our tester against "his" application and "his" team. He called the tester for a one-on-one meeting and questioned his work. Did the tester really understand the business objectives of the application? Should the tester have been spending more time validating the application? Were his defects really valid at this point in time?

Our tester did not like confrontation. So, when the Development Lead changed the following rules, our tester said "As you wish" and smiled meekly:

1. The tester would now only test the modules specifically assigned to him and not anything else.

2. If the tester thought he found a bug, he was to approach the Development Lead, take an appointment and present his bug. Only when the Dev Lead had analyzed and agreed to the bug was the tester required to report it in the bug tracking system.

3. The tester was not to approach any team members directly. If he had questions, he was to take permission from the Dev Lead and only then meet the team members.

The Development Lead did not however stop at that. During his status reporting meetings with the management, he took the time about how he thought of our tester as a somewhat loose cannon. He said that the tester needed to be monitored closely but assured the management that he had put the tester on an "improvement plan".

Life had become tougher for our tester. While testing, he could see bugs in many areas of the application but he had to keep them to himself. When he attended the meetings with the team, his mind would be full of ideas. However, many times he hesitated to share those ideas. Other times, when he would get the courage of speaking them, he was quickly interrupted by the Dev Lead, "We will talk about it later". The tester would just smile weakly and say "As you wish". Many other team members too started taking little notice of the things the tester had to say. When the tester found bugs in the modules assigned to him, he would make careful notes and take them to the Dev Lead for discussion. The Dev Lead rejected most of the bugs he thought of as "not realistic enough", "too early to fix" or "doubtful". The result was that he allowed the tester to report the most obvious bugs only. But our tester kept on plodding without regrets.

Time passed. A number of releases were made to downstream teams. Finally, the application was deployed to production. The Dev Lead was also now the in-charge of supporting the application in production. In the first few months, internal customers reported several issues in the application. These were promptly fixed. Six months after the go-live date, the Lead happened to think about the issues reported in the go-live period. He listed the issues and started analyzing them. Just out of curiosity, he picked up our testers submitted notes from the previous year to see whether he had put forward any one of the real issues. He was amazed. All save one issue reported in production against the tested modules was discussed by the tester, while the application was still in development. Now, the Lead was very ashamed of how he had treated the capable tester.

He promptly called the tester for a meeting. He hurriedly explained his pre-occupations at the time and how he now realized the good work done by the tester. The tester was now free to discuss his ideas with others and report bugs as he saw fit. Then he asked the tester if this was okay? All our tester had to say was "As you wish", again with a meek smile.

Let us treat each team member with respect. Who knows how much a team member may be able to contribute to our success?

Why you should NOT use the Equivalence Partitioning technique AS IS in black box testing?

2010-06-16T19:25:00.002+05:30

You may already know the Equivalence Partitioning technique. Put simply, it is a test design technique. It involves dividing the input test data into partitions, valid and invalid. It promises to reduce the number of input test data values that must be used by suggesting that only one test data value be used from each partition to determine the behavior of the application for each partition.

Let us consider an example of the input, day of the week. The valid partition is 0..6 (or Sunday to Saturday). marked in green. There would be two invalid partitions, <0 and >6, which are marked in red.

-------------------------------------------------------------------------------------------

... -3 -2 -1 0 1 2 3 4 5 6 7 8 9 ...

-------------------------------------------------------------------------------------------

Therefore, you decide to execute your test case with three distinct values, say -3, 3 and 9 (a value each from the first invalid partition, the valid partition and the second invalid partition). Correct?

Wrong decision. The technique assumes that the application treats any value within a particular partition identically. Well, may be NOT. Unless you are privy to the source code and can confirm that indeed each value in each partition is handled exactly the same.

Partitioning is fine. I disagree with the "Equivalence" part of "Equivalence Partitioning" term. I will mention why. Let us say that the programmer wrote separate "Case" or "Switch" statement(s) for each value from 0 to 6. If you only ever execute your test case for the input value 3, you would never test the application's code for the other values in the valid partition (0,1,2,4,5 and 6).

At this point, I can anticipate questions. How about the invalid partition? Should you execute your test case with multiple values in the invalid partition? How far back should you go? Obviously, you cannot use each value in a large partition. However, you may consider the following:

a. Use at least one or two values in the beginning of the partition e.g. -1 and -2.

b. Use at least another value deeper in the partition e.g. -100.

c. Use a value outside the range of the data type (integer in this case) e.g. -12345678

In short, when you have to test without looking at the source code, test using each input value in a small partition. And, test using spread out sample values in a large partition.

What is the future of software testing?

2010-06-07T18:54:00.000+05:30

We don't know the future. Nobody does. However, I have been involved with software testing since 1998 and I have seen changes taking place. It used to be simple earlier. We received the requirements and designed simple test cases based on these requirements. Then we received the latest build of the application, execute our test cases and report discrepancies in bug reports. We did all of this as carefully and as fast as we could. As I experienced software testing first hand, I began to learn it. There were multiple approaches available to perform testing tasks. Each approach had its own pros and cons. There were multiple types of testing that we could do. Each of my team members had a different set of skills, strengths and weaknesses. It was not simple anymore. 3 or 4 years ago, I started following the online software testing communities. I read a lot of material and comments on hundreds of topics in software testing. I saw thought leaders in software testing repeatedly pointing us test practitioners to the basics. Over time, I have begun to consider software testing as quite complex and challenging. The good news is that software testing still has a long way to go before it truly matures.

Here is the list of my predictions. These are not revolutionary changes that are going to catch you all of a sudden. In fact, you can see some of these changes today. But, if you are aware that these changes could speed up or intensify in the coming future, then you have a better chance to prepare yourself to take advantage of them.

Prediction #1. Companies will demand more value for the testing resources they put in.

The recent downturn has shaken everyone. We have been forced to become savvier with our investments. The same is true for companies. Companies will demand a better testing service from their resources (in-house testing team or a vendor providing testing services). The companies will now demand:

a. Faster turnaround time

b. Greater coverage of specified and implied requirements

c. Testing in more perspectives (functional, performance, security, usability and so on)

d. Increased collaboration with all other teams involved in sales/ product development, software development, deployment and support

e. Lower costs (of test infrastructure (test environment), test tools and testing personnel)

f. More transparency of the test process

Prediction #2. Software testing will become more complex.

Keeping in mind the increased expectations of clients, the increasing complexity of applications and the increasing knowledge of test practitioners, software testing will become more complex. In future, testers will need to find answers to the following questions among others:

a. What are the most important business objectives of the application that I am testing?

b. What technologies does my application employ? How do I test each of those?

c. What test infrastructure will I need to test my application? How can I set those up with the least cost (of setting it up, using it, maintaining it and tearing it down)?

d. What tests would provide the best value against the cost of creating them?

e. How is my application integrated with other systems? How do I test various aspects of each integration?

f. What is the best test methodology that I can use?

f. Which of my communication provides the best value to other stakeholders?

g. How do I utilize my natural strengths in testing? How do I circumvent my natural weaknesses?

Prediction # 3. Crowd-sourcing will continue to become popular.

uTest is becoming more popular by the day. Today, uTest has more than 20,000 testers and a client list that includes Google and Microsoft. The clients of crowd-sourcing companies can buy just the testing services they need when they need them and even select the individual testers for the test. No wonder, many companies consider crowd-sourced testing services as viable alternatives to large in-house testing teams or inflexible testing services vendors.

Prediction # 4. In order to get hired and stay hired, testers will need to distinguish themselves from the crowd.

Today, there are masses of software testers. Their profiles and resumes look similar. If I were going to hire someone for my team, I would not like to just go for someone with the basic knowledge and skills. I would like to get the details. And, I would probably like to interview someone with substantial achievements. Someone who has "walked the extra mile". Someone who has achieved more than their counterparts at the same level. Be it extra-ordinary knowledge, uncommon or advanced skills or a solid recognition from testing experts.

Prediction # 5. Social skills and working style will become important.

Other than software testing knowledge and skills, testers will be required to be socially adept. They will be required not only to plan and test well, but also communicate well. They will be required to establish themselves as part of a team, support the team, speak up when required and influence others when required. In future, just testing won't do for the testers. They will also be required to collaborate effectively and strive to maintain long-term relationships with their extended teams. Further, testers will be required to show align themselves to the (stable or changing) business objectives and the team.

What do you feel? Do you think that you are ready for these changes? Are you going to take advantage of them? What other changes do you see on the horizon?

The Ultimate Machine: How would you test it?

2010-06-06T21:22:00.000+05:30

As many of you know, we now have our Software Testing Space group on LinkedIn. A lot of exciting discussions are taking place there. If you have not joined it, you should consider doing so.

My friend, Freddy Gustavsson from our STS group posted an interesting challenge on testing the Ultimate Machine. Well, the Ultimate Machine is attributed to Claude Shannon (1916 to 2001). It consists of a simple box with a single switch on its top surface and a closed lid. If you flip the switch on, the lid opens, a mechanical hand arises from it and the hand flips the switch off. Thereafter, the hand goes back into the box and the lid closes. You can watch the Ultimate Machine in action in this video.

Freddy's challenge was that if he asked me to test this Ultimate Machine, which tests would I perform?

Here are the questions that I thought of. I would like to design tests based on these questions.

Positive tests

a. If the tester flips the switch on, does the hand (or an arm in certain implementations) switch it off?

b. If the tester does not touch the switch, does the machine just sit there and do nothing?

Negative tests

a. What happens when the tester flips the switch on and switches it off before the hand can reach it?

b. Is it possible to flip the switch somewhere between the on and off positions? What does the machine do in such a case?

Non functional tests

a. Can the tester operate the machine as it is or does it need to be installed? If installed, what does it take to install it?

b. Once the machine is switched on, how long does it take to switch itself off?

c. How many times can be machine be operated before it stops working (due to battery discharge, wearing off of mechanical parts and so on)? Does it slow down after a while or show operational problems (make hitherto unknown noise for instance)?

d. Does the machine operate correctly if the ambient conditions are changed?

i. The machine is vibrated e.g. during travel or a minor tremor.

ii. The machine is operated outdoors (e.g. during a dust storm or in rain/ sleet/ snow).

iii. The machine is operated in a strong magnetic field (e.g. certain places on a factory floor or in a hospital).

iv. The machine is operated in different light conditions (e.g. direct sunlight, in the office daytime with blinds open/ blinds closed, at dusk or in total darkness).

v. The machine is operated in different temperatures (e.g. at room temperature, in a freezer or near a furnace).

Usability tests

a. Is it easy for the tester to locate the switch?

b. Is the switch easy to flip (works smoothly, clicks to indicate it has flipped)?

c. Is the switch safe for the tester to operate (no rough surfaces or pointed end)?

d. Is the box pleasing to look at? How much space does it occupy on the tester's desk?

White box tests

We could open the box. Then, we can try to understand the mechanism inside and create more tests based on our understanding. For e.g. I would like to know if the time it takes to switch off is configurable and is the machine connected to any external system.

I am sure that you can think of more questions to help test the machine. Did you notice that I used the various types of testing to think about the above questions? What approach would you have used to design tests for the Ultimate Machine?

Do Developers really HATE testing?

2010-06-05T20:27:00.002+05:30

Having recently read a post on the UTest blog, I decided to think back about my developer days. What challenges did I remember from my experience as a Developer? Did I test my code or not?

1. Sometimes, it required a huge effort on my part just to know the correct set of requirements. Sometimes, the requirements would be given by a business person who would be focused on the promises already made out to the customers. The product manager/ business analyst would rarely verify the technical feasibility of implementing the requirements before committing to the customers. In such a situation, it became a matter of identifying the design alternatives available and exploring the promising ones in detail. As a developer, I used to take such situations as intellectual challenges. This resulted in me saying Yes, even when a simple No would save me a lot of work.

2. At other times, I was handed over source code that had been written, changed and enhanced (all these in different orders) by at least 3 or 4 developers before me. These developers were now either busy with other projects or no longer available in the company. I found this source code riddled with problems. There would be sections of the code that were incomplete, the sections would work with some input values but not with others, were partial duplicates of other sections or had logical problems. Getting such code to work was challenging. It involved plowing through the code (reading a section, unit testing it, resolving the problems, unit testing it again, refactoring it to make it more understandable, unit testing it and so on).

3. The other thing I recall is the sheer number of problems I had to face in order to implement a single requirement. Examples of these problems were:
a. The design was either non-existent or did not cover the particular requirement. This meant that my task doubled in scope. I first had to create a good design and then implement it.
b. The base components were not present or had defects. So, I had to decide either to re-write the base components from scratch or debug them before using them.
c. A similar requirement was already implemented in the application. The trouble was that I was either not able to understand that implementation or suspected hidden defects in that implementation.
d. The underlying development or run-time environment had issues (meaning defects or constraints) in certain conditions. This meant that I had to either somehow circumvent those limitations or break down my implementation into multiple parts in such a way that I did not run into these issues.

4. The next thing that I remember is how tight the schedules were. There were two reasons for this. The first reason was that the development effort was routinely under-estimated. Maybe my project manager (who estimated the development effort) wanted to increase my productivity constantly. Or maybe s/he estimated the effort considering the simplest development possible. The second reason was that I wanted my code to be perfect (in some cases, more than perfect). This meant that I would write the code implementing a requirement and run it repeatedly looking for problems. Once I had code running without problems, I would refactor it to make it leaner and strictly according to the coding standards (for example, naming the variables according to the specified nomenclature, adding comments every few lines and so on). If time permitted, I would even attempt to enhance the design or add extra functionality (though I stopped adding any extra functionality quite early on). My worries about how my code may not work under specific conditions led me to test it, modify it and again test it repeatedly. Given the tight schedule and my worries about the quality of my code meant that I was under stress.

It is not that as a Developer, I hated testing. Quite the reverse actually, I always wanted my code to be perfect and tested constantly. It was the sheer number of problems that I had to solve in a tight schedule that allowed some defects to fall through the cracks. And the defects discovered by the testers were only small fraction of all the defects that originally existed in the code.

Though I tested my code constantly looking for defects, there was only so much I could do given the tight schedule. In fact, I found that in internal projects (where the delivery deadlines were relaxed), I was able to submit code in which the testers could hardly find any defects.

Software testing involves a lot of work. The tight schedules do not help because it is very common for the development scope to increase during development. If the schedules are more realistic, I am confident that developers would test their code better and there would be a lot less defects in their applications.

How to compete against other software testing teams and win?

2010-06-01T20:29:00.002+05:30

There may be occasions when you or your software testing team is pitted (compared from a business point of view) against another individual or another team. Examples include:

1. You are a senior tester and the management has just hired a junior person. Now, some tests have been assigned to the junior tester. The idea is to see if there is a discernible difference in the outputs of the two of you.

2. Your management or client has decided to off-shore (OR bring on-site) some of the testing work. Some work is now being done in parallel in both locations. The idea is to see which team provides more value with respect to the cost it incurs.

3. An experienced person with the business background has been added to the team. The idea is to see if business knowledge or software testing knowledge provides greater value in testing.

Before I go on, let me clarify that healthy competition is a fast way to grow your capabilities. Otherwise, things tend to stagnate. And quite often, the competing teams end up collaborating with and supporting each other the other side of the tunnel.

But, in the meantime, you have to think about at least maintaining your position. If you do NOT take any additional action, your role may be diminished or change into something else regardless of your preference. How can you maintain (and subsequently grow) your position?

1. Study the other team.
When sports team prepare for competitions, they actively study the other team. You should do the same. You should find out the following information about the other team:
a. The number of members in the team
b. Their names and location
c. Their general background, experience, knowledge, achievements and struggles
d. Their prior exposure to your company's specific applications, tools and processes
e. Their working style/ culture
f. Any natural advantages they have over you

2. Follow their work and deliveries.
You should actively follow their work and the deliveries they produce. Use all channels (official and informal) to keep you updated about their progress.

3. Analyze the other team's strengths and weaknesses.
Your team should now be able to have a fair idea of the other team's strengths and weaknesses. Be ready to update their image in your mind as you follow their progress regularly.

4. Identify your own strengths and weaknesses.
This is a good time to be more aware about your own team. What things is your team good at? What tasks is it not so good at? In what respects does your team absolutely struggle?

5. Change your approach appropriately.
If are keenly aware of the other team and your own team, it should be easy for you to identify the ways in which you can out-perform them. Some examples are given below but you should find out your own ways to out-perform the other team. And, implement the changes quickly.
a. If the other team only executes the predefined test cases, your team may want to additionally invest time in enhancing and then executing the available test cases. Or perform exploratory testing in addition.
b. If the other team focuses on functional bugs, your team may want additionally focus on performance bugs, usability bugs and look and feel bugs.
c. Get work done faster. If their team members watch the clock (come in at time, work and leave on time), your team can shorten your breaks or concentrate more on the tasks or stay on the task longer.
d. If the other team has individual performers, your team may want to harness the power of team work.
e. If their team works in a silo, your team may have a good rapport with the other engineering teams.

6. Inform the decision makers yourself.

You can be sure that the decision makers who initiated this change would be actively following its effects. You should inform your own achievements in a positive light to them. Do NOT rely on someone else to do this for you. If so, they may tone down your achievements according to their own biases, opinions and agenda.

7. Finally, help your NEW team.
When the dust settles, your team will have a better assessment of their strengths and weaknesses. You will also be more mature. The other team will also share the same benefits. After all, everyone wants to grow.

If the decision makers take a professional decision, the work will be more aligned according to the strong areas of the either team. The teams may even be re-distributed or merged into a single unit. It is a WIN-WIN situation. Now is the time to help yourself and your new team to the best of your abilities.

Why you need to articulate more?

2010-05-25T21:27:00.000+05:30

You need to communicate your ideas, thoughts and progress whether you are a tester, a manager or the CEO of your company.

What is the benefit of articulating your thoughts? When people hear from you, they come to know where you are coming from, what your approach is and what are the actions that you are going to take? Once they understand your position, they tend to support you. Why is this? This is because your communication makes them think in a way similar to yours.

A person with the same knowledge as you but who does not articulate their thoughts receives none of these benefits. If you don't believe this, go back to your school days. Think of the popular students. Were they the ones who always stayed silent or were they the ones who had interesting things to say to the teachers and in your friend circle?

But, what happens when you articulate more? You influence more people and in a deeper way.

1. Tester: A tester who speaks of new test ideas in team meetings and writes informative bug reports and test reports is perceived well by his colleagues and manager. The team and the manager are impressed by the tester and increasing responsibilities/ authority come his way.

2. Manager: A manager who clearly conveys the objectives of the project to his team members and updates the management effectively runs a more successful project. The management is eager to assign bigger and more important projects to him.

3. CEO: A CEO who pitches the strengths of his organization well in front of the clients tends to win their confidence (and more business for the company). When the same CEO outlines the company strategy clearly to the employees, he aligns the employees to the company objectives well.

When you articulate more, you:

a. Force yourself to think about more topics of interest.

b. You study/ research these new topics. You learn.

c. You plan according to the objectives of your communication.

d. You design what you are going to say in structure and words suitable to your recipients.

e. You overcome shyness when you actually communicate.

f. You become mentally agile to answer your recipients' questions well.

g. You become faster and more effective in each step of the communication process.

So, seize every opportunity to articulate. And, articulate better than you have ever done before. Get more and more benefits.

How to be a good reviewer?

2010-05-22T20:54:00.002+05:30

We often get artifacts such as test plans, test cases, test results or bug reports to review. This is especially true for those of us in the leading or senior positions. Do you wish to be a good reviewer? If so, try NOT to fall into the following categories as others have in the past.

1. The Silent Reviewer

You submit your material for review. There is NO response from the reviewer. You decide to follow-up a bit later. Still, there is NO response. What the Silent Reviewer is not telling you is the reason they are not able to perform the review. Often times, the Silent Reviewer assumes that you know everything that you need to provide them to do the review. Diplomatically coaxing the reviewer to share the reason can be helpful. The trouble is that not many authors think of this approach or take the necessary time to ease the review along.

A dangerous version of this type of reviewer is the Silent and Angry Reviewer. They do not perform the review but tend to form a poor opinion about you.

2. The To-and-Fro Reviewer

You submit your material for review and look forward to completing other work. Little do you realize that you are going to be spending a lot of time with this particular reviewer. The To-and-Fro Reviewer takes one look at the material and shoots off a message e.g. You have not used the correct template OR You have not provided a specific piece of information. Your reply results in request for still more information or maybe, a review comment. You feel confused and exhausted after a long chain of communication with this reviewer. The trouble is the To-and-Fro Reviewer is performing the review on their whims and fancies.

3. The Opinionated Reviewer

You request for a review and what do you receive? A personal comment on your lack of knowledge, lack of competence or something basic that you have missed or done incorrectly. You feel bad after hearing from the Opinionated Reviewer because they are not focusing on the material but rather on you. You may also find the Opinionated Reviewer transform into another category after blaming you personally.

4. The Insensitive Copier

The Insensitive Copier wants to look good in the eyes of others. Even when they do a thorough and useful review, you find that they have copied their review comments to your manager, their manager, colleagues and the Head of the unit. Their perception of the quality of your work is now visible at the high levels. Worse, they may ask you to address their review comments immediately. If you do not address the review comments promptly, you risk looking lazy in the eyes of the management. So, you have to re-schedule your other (important) work and tackle the review comments pronto.

5. The Skimpy Reviewer

This reviewer provides you little value. Maybe, it took them less than two minutes to review your work. You get comments like:

a. It seems alright at first glance.

b. The formatting seems to be off in places.

c. I am sure that you have done a thorough job. Don't have any review comments as of now.

6. The "Busy" Reviewer

This reviewer performs the review, even to a high standard, but in their own time. What good are review comments on your test cases when testing is mid-way and there is little or no time to update the test cases? In extreme cases, you receive their review comments after the event is over e.g. the release has been deployed in production. There is little use for the review comments except as learning for the future.

If you do not agree with the above, try to remember the time when you were at the receiving end of one of the above types of reviewers? How did you feel after the review? Drained, exhausted, bitter?

If the review is not too important or you are too busy to do it well, you should politely decline it giving the genuine reason. However, if a review is important, you should do it well. Schedule an available time for your review. Make sure that you have all the required material, do the review and pen your review comments using your knowledge and wisdom. Go through your review comments and make sure that they are professional, factual and focus on the material (and not on the author). And please, communicate your review comments promptly giving sufficient time to the author to address them correctly.

GUI Test Automation with RoutineBot

2010-05-18T11:13:00.000+05:30

Recently, I came across a simple and effective GUI Test Automation tool. It is named RoutineBot. RoutineBot works by searching and performing actions on image patterns. Other than that, it can emulate keystrokes and mouse movements and clicks.

A few salient points about RoutineBot are:

1. It works with both Windows applications and Web applications.

2. It supports Pascal, Basic and JScript scripting languages.

3. It provides warning and error messages to help debugging, returns results of the test and closes the test on timeout.

4. It provides Environment options (e.g. snapshot interval i.e. time during which you should navigate to the image, location of the log file etc.)

5. It currently comes with a free unlimited 30-day trial version.

6. The full version is priced quite economically when compared to other tools with similar features.

Here is my experience when I decided to try out RoutineBot.

Installation

The installer is in a single file, routinebot.exe of a small size (2.2 MB). On running the installer, the installer shows the standard wizard (confirmation to install RoutineBot, License Agreement, selecting destination folder, selecting start menu folder, creating desktop icon, selected settings so far and the complete setup screen). The installation was done in a matter of seconds on my machine.

Applications tested

Notepad

Since the example mentioned on RoutineBot website used a Windows application (Calculator), I decided to use RoutineBot to automate few test steps on a Windows application. However, I used the Notepad application instead. The steps that I decided to automate were:

a. Open Notepad.

b. Type some text in it.

c. Save the text in a file.

Opening an application is very simple in RoutineBot. The Execute command did that for me. Next, I needed to enter some text. The command to do so is EnterKeys. Note that EnterKeys can be used not only to type text but also other keystrokes like Ctrl, Alt, Enter, Function Keys and so on. The next part is where I had to learn using the RoutineBot approach of using image patterns. I needed to click the File menu item. Therefore, I changed the tab to Select Test Image. Then, I clicked on the Make snapshot button. It starts a countdown of 5 seconds during which I had to move the mouse pointer near the File menu item. Next, I used the Select and crop button to select just the File menu item text. Then, I saved the selected image (of the File menu item text) as a .bmp file (I could have chosen the .jpg or .gif format instead) using the Save sample button. Next, I selected the action of MouseFocuse (the default action). Now, I switched to the Script tab and placed a MouseClick command after the MouseFocuse command. Similarly, I automated the click on the Save menu item (under the File main menu item), typed a filename and clicked on the Save button. Every time I added something to the script, I clicked the Run script button to ensure that it worked just as I wanted it to work.

My final script looked like:

Execute('C:\WINDOWS\NOTEPAD.EXE');

EnterKeys('Hello world!');

MouseFocuse('FileMenuItem.bmp',10000);

MouseClick;

MouseFocuse('SaveMenuItem.bmp',10000);

MouseClick;

EnterKeys('abc');

MouseFocuse('SaveButton.bmp',10000);

MouseClick;

Firefox browser

Next, I chose to automate the following test steps of a Web application:

a. Open Firefox browser.

b. Navigate to Google search engine.

c. Search for "RoutineBot".

d. Click on the first result (expected as the RoutineBot website's home page).

I tried the Record option but, as expected, it generated too many lines in the script. Therefore, I chose to create my own script by using the Add Action button. The working script looked like:

Execute('D:\Program Files\Mozilla Firefox\firefox.exe');

Wait(5);

EnterKeys('www.google.co.in~'); //Note that the ~ stands for the Enter key

Wait(5);

EnterKeys('RoutineBot~');

MouseFocuse('RoutineBot.bmp',10000);

MouseClick();

Licensing and Price

1. I found the price of RoutineBot reasonable. You can see the pricing information here.

2. Note the important current licensing approach here, "One registered copy of RoutineBot may either be used by a single person who uses the software personally on one or more computers, or installed on a single workstation used non-simultaneously by multiple people, but not both."

3. It currently comes with free updates (minor version changes) but paid upgrades (major version changes).

Other tips

1. RoutineBot did not display complete UI on my screen that had the resolution set to 800X600. I found that when I increased my screen resolution to 1024X768 or higher, there was no display problem.

2. If you add actions from the buttons (Add Action or Select Action) as well as write commands in the script yourself, you should look out for formatting errors. Anyway, RoutineBot points the exact location where the syntax is incorrect.

3. When you Hide RoutineBot, it not only becomes invisible but also disappears from the list of open applications (meaning, for example, that you cannot use Alt-Tab key combination to open it). On hiding, RoutineBot displays as an icon in the taskbar. You can click this icon to unhide it.

4. When you execute your script, you should ensure (programmatically or otherwise) that the RoutineBot Interpreter toolbar (with Stop, Pause and Hide buttons) does not overlap an image of interest. In other words, the image to be searched should be visible. Otherwise, RoutineBot will not be able to find the image and perform actions on it.

5. If the image pattern changes in the application, then your script may not work correctly. Therefore, for images likely to change, it is better to use smaller images (that stay constant) or to use the button's caption to click it.

Summary

This is my overall view of RoutineBot. I was able to learn building test automation scripts using RoutineBot within a few minutes. Then, I read the documentation that is available on the tool website (see links below). After reading the documentation and trying simple examples, I became confident that I had mastered the basics of the tool. Since, I the trial period of the tool is far from over, I plan to automate test cases of a web application with it and see for myself how well RoutineBot is able to automate them.

Important links

RoutineBot website
FAQ

Screenshots

Test automation samples

Recording test automation scripts

Supported commands

Ordering

Support

Disclaimer: I have reviewed RoutineBot as a potential user. I do not represent the tool vendor (AKS-Labs). If you want to evaluate RoutineBot for your needs, you should use your own judgment and contact the tool vendor directly.

What have I learnt after blogging for a year?

2010-05-17T14:16:00.000+05:30

The timing of this post is almost correct. I have been blogging for a little over a year now. Software Testing Space has now over 50 articles. Here are the top lessons that I learnt while blogging about topics in software testing. You will find these tips handy if you blog or plan to blog in the future.

1. Getting ideas for posts is critical.

In the beginning, I spent a lot of effort of listing and evaluating ideas to blog about. Then, I would select a topic that was interesting to me. That was a mistake. Your post is helpful only if the content is useful to your readers. Off late, I have found that a good source of ideas is the questions posed by readers. Of course, the response to the question should provide value and be interesting. You can also write on topics on which there is little, misleading or no information available online.

2. The title of the post is its most important part.

A reader is more likely to read the post if your title is interesting. A short title is better than a longer one. If you can frame the title as a question, even better.

3. Posts that draw on personal experience are more popular.

Posts should be interesting. A post based on personal experience captures a reader's attention more than one that just deals with a topic in a theoretical fashion. If you have no personal experience to draw upon, then you should provide the data supporting what you write. At least, you should explain the logic of your thoughts on the subject.

4. The post itself should not be too short or too long.

The post should cover the subject well. For example, if you are writing about how to create a good test strategy, you should first define what you mean by a test strategy. Then go on to point out the benefits of a good test strategy and finally outline the method by which you can create a powerful test strategy.

If your post is too long, a reader may just skim the post. However, this is not always true. I have had readers write to me and sometimes they remember even a point made in passing one of the earlier posts.

5. Grouping and linking information makes it easy to find it.

Earlier, I just managed with a chronological listing of all the posts. If a reader had to find a related post, they would need to look up the title of each prior post. Grouping your posts by category makes it easier for all. Further, linking to other resources (including previous posts) within the post is even better.

6. Add content regularly.

I have seen blogs that only have a handful (say, 2 or 3) posts or haven't been updated for a year or more. It is sad. If a reader visits your blog again, they expect to find some new useful or interesting content. These days, I aim to create at least 10 to 15 new posts every month.

7. Use the power of humor.

Writing with humor is an art. To be honest, I find writing with humor while being sensitive and respectful to all concerned rather challenging. However, I have written one humorous post and plan to write more in the future.

8. Finally, take help from others.

You should remember that there are many other experts in your field. They may be happy to share their knowledge with you. I found that interviewing other experts is a good way to way to gain knowledge and broaden your mind. I have started interviewing experts and soon more interviews should follow. I, too, write on other sites. If you would like me to write for you, you are welcome to let me know :}.

What is the best test data for YOUR TESTING?

2010-05-15T20:17:00.000+05:30

Before we start discussing this important point, let us take a moment to realize the different types of test data:

a. Application configuration data
Your application likely needs test data to function (or even launch). Examples of application configuration data include the information to connect to database(s), admin user name and password and server information (e.g. email server) in order to send email notifications.

b. Application data
Other than the above, an application may require application specific data. Examples of application specific data are the menus/ links (and their hierarchies) and detail item information (e.g. item name, item details and item price).

c. Customer or user data
The users creating your application may create their own data within the application. Examples of customer or user data are user details (name, address and other personal details), user searches and user transactions (items browsed and purchased).

The questions that you should consider when creating or sourcing the test data for your tests are:

1. Does your application handle blank test data?
In case, your application does not find a particular test data, it should not crash. It should either display an informative (just the required information and no more; otherwise it may be security risk) message or it should display no data but working functionality. For example, if the information to connect to a database is missing, the application should throw an appropriate error message. If the user has not entered any required details, it should prompt the user to enter those details first.

2. Does your application handle invalid test data?
Data input to an application may be invalid in terms of data type (e.g. character data provided instead of the required numeric data), data size (insufficient or excessive size), outside the valid data range or just incorrectly formatted (dots instead of spaces). There are several approaches to deal with invalid test data. An application may reject the invalid test data, provide valid options for the user to select or attempt a conversion to valid data. In any case, invalid data should not be accepted by your application.
Even if the application handles new invalid data correctly, you should test the existing data for invalids. See the articles on database testing for details.

3. Does your application handle valid test data across the entire range of such data?
Of course, your application should accept valid data. However, using a single test data value for a range is not sufficient. Your test data should include the boundary values and several values (you should ask yourself how many values are reasonable) in between.

4. Does your application store confidential user data?
If you use a copy of the current production data, you could be exposing confidential user information to unauthorized persons. You could consider using only the application configuration data and application data from production. Another way to prevent exposure of confidential user information is to obfuscate the customer data from production.

5. Does the application handle the data volumes present or likely in production?
If your test data includes only a handful of data values and there are thousands of data values in production, your testing will not be realistic. You should match your test data quantity with that in production. Further, for realistic tests, your test data values should not be simple duplicates but vary as in production.

6. Is new data released into production from time to time?
You should test the data being released into production independently as well as with your application. After testing, you should update your test database with the new test data released to production.

What kinds of testing should you perform?

2010-05-13T17:39:00.000+05:30

There are many kinds of testing that one hears about. You would likely know about many types of testing e.g. white box testing and black box testing, unit testing, integration testing, system testing, system integration testing and so on. The types of testing that would be performed on the application are usually identified in the test planning phase. If you wish to know which kinds of testing are suitable in your project, consider the following questions:

1. Do you have intimate knowledge of the design, logic and code of your application?

If you do, you may want to opt for white box testing. Otherwise (or additionally), you can perform black box testing. You can base your black box testing on the requirements and your knowledge of how your application should function.

2. What types of testing are common in projects?

Most applications are written not as single lengthy source code but rather as a collection of units (e.g. routines, functions or modules). In unit testing, these units are tested in isolation. In integration testing, the interactions between the units are tested.

Further, every application has some functionality. You test the functional requirements in functional testing.

Usability testing is used to determine if the application is easy to learn and use. However, usability testing is more commonly performed on applications with a large user base.

3. Does one test run/ test cycle on your application take substantial time?

If so, you may want to perform sanity testing/ smoke testing to determine if the application is stable and functioning well enough to deserve lengthy or detailed tests.

4. Will your application be used by more than one user? Will it hold or display confidential data?

If so, you should perform security testing not only on your application but also on the application infrastructure (e.g. application servers, database and network).

5. Does your application need to satisfy specific non-functional requirements?

You can group the non-functional requirements. Each group may require a special kind of testing. For example, if yours is a web application that is supposed to run on multiple browser versions, you need to do browser compatibility testing. If your application needs to respond within specific times, you need to do performance testing.

6. Will the user need to install the application before using it?

If so, you should perform installation testing (install and uninstall of the application on each supported platform).

7. Is it a new application or an upgrade?

If it is an upgrade, testing the new features of the application is not enough. You need to ensure that the previously working features are still working correctly. You need to do regression testing.

8. Does your application communicate with other applications/ systems?

System testing (testing the application as a whole) is applicable to every application. In case your application interacts with other applications or systems, you should also perform system integration testing.

9. Do you have an independent team (excluding developers and testers) who can test the application?

If so, they can perform alpha testing on (major) releases before completion.

Sample database test plan

2010-05-11T21:28:00.000+05:30

Shell, one of our readers, went through the article, How to do real database testing (10 tips to perform serious database tests)? and requested information on database test plan.

Well, here are some tips to create a good database test plan:

1. Database testing can get complex. It may be worth your while if you create a separate test plan specifically for database testing.

2. Look for database related requirements in your requirements documentation. You should specifically look for requirements related to data migration or database performance. A good source for eliciting database requirements is the database design documents.

3. You should plan for testing both the schema and the data.

4. Limit the scope of your database test. Your obvious focus should be on the important test items from a business point of view. For example, if your application is of a financial nature, data accuracy may be critical. If you application is a heavily used web application, the speed and concurrency of database transactions may be very important.

5. Your test environment should include a copy of the database. You may want to design your tests with a test database of small size. However, you should execute your tests on a test database of realistic size and complexity. Further, changes to the test database should be controlled.

6. The team members designing the database tests should be familiar with SQL and database tools specific to your database technology.

7. I find it productive to jot down the main points to cover in the test plan first. Then, I write the test plan. While writing it, if I remember any point that I would like to cover in the test plan, I just add it to my list. Once I cover all the points in the list, I review the test plan section by section. Then, I review the test plan as a whole and submit it for review to others. Others may come back with comments that I then address in the test plan.

8. It is useful to begin with the common sections of the test plan. However, the test plan should be totally customized for its readers and users. Include and exclude information as appropriate. For example, if your defect management process never changes from project to project, you may want to leave it out of the test plan. If you think that query coding standards are applicable to your project, you may want to include it in the test plan (either in the main plan or as an annexure).

Now, let us create a sample database test plan. Realize that it is only a sample. Do not use it as it is. Add or remove sections as appropriate to your project, company or client. Enter as much detail as you think valuable but no more.

For the purpose of our sample, we will choose a database supporting a POS (point of sale) application. We will call our database MyItemsPriceDatabase.

Introduction

This is the test plan for testing the MyItemsPriceDatabase. MyItemsPriceDatabase is used in our POS application to provide the current prices of the items. There are other databases used by our application e.g. inventory database but these other databases are out of scope of this test.

The purpose of this test plan is to:

1. Outline the overall test approach

2. Identify the activities required in our database test

3. Define deliverables

Scope

We have identified that the following items are critical to the success of the MyItemsPriceDatabase:

1. The accuracy of uploaded price information (for accuracy of financial calculations)

2. Its speed (in order to provide quick checkouts)

3. Small size (given the restricted local hard disk space on the POS workstation)

Due to limitation of time, we will not test the pricing reports run on the database. Further, since it is a single-user database, we will not test database security.

Test Approach

1. Price upload test

Price upload tests will focus on the accuracy with which the new prices are updated in the database. Tests will be designed to compare all prices in the incoming XML with the final prices stored in the database. Only the new prices should change in the database after the upload process. The tests will also measure the time per single price update and compare it with the last benchmark.

2. Speed test

After analyzing the data provided to us from the field, we have identified the following n queries that are used most of the time. We will run the queries individually (10 times each) and compare their mean execution times with the last benchmark. Further, we will also run all the queries concurrently (in sets of 2 and 3 (based on the maximum number of concurrent checkouts)) to find out any locking issues.

3. Size test

Using SQL queries, we will review the application queries and find out the following:

a. Items which are never used (e.g. tables, views, queries (stored procedures, in-line queries and dynamic queries))

b. Duplicate data in any table

c. Excessive field width in any table

Test Environment

The xyz tool will be used to design and execute all database tests. The tests will be executed on the local tester workstations (p no.s in all).

Test Activities and Schedule

1. Review requirements xx/xx/xxxx (start) and xx/xx/xxxx (end)

2. Develop test queries

3. Review test queries

4. Execute size test

5. Execute price upload test

6. Execute speed test

7. Report test results (daily)

8. Submit bug reports and re-test (as required)

9. Submit final test report

Responsibilities

1. Test lead: Responsible for creating this test plan, work assignment and review, review of test queries, review and compile test results and review bug reports

2. Tester: Responsible for reviewing requirements, developing and testing test queries, execute tests, prepare individual test results, submit bug reports and re-test

Deliverables

The testers will produce the following deliverables:

1. Test queries

2. Test results (describing the tests run, run time and pass/ fail for each test)

3. Bug reports

Risks

The risks to the successful implementation to this test plan and their mitigation is as under:

Approval

Name Role Signature Date

1. ____________________________________________________________

2. ____________________________________________________________

3. ____________________________________________________________

What is the difference between severity and priority?

2010-05-07T14:29:00.000+05:30

After having seen this question floated in so many forums, I decided to write about it. First, the basics. These terms are used with respect to bugs. Severity and Priority are two attributes of a bug report. If you have seen Effective Bug Reports, you can see simple definitions of these terms.

Here are the main differences between severity and priority:

Severity	Priority
In simple words, severity depends on the harshness of the bug.	In simple words, priority depends on the urgency with which the bug needs to be fixed.
It is an internal characteristic of the particular bug. Examples of High severity bugs include the application fails to start, the application crashes or causes data loss to the user.	It is an external (that is based on someone's judgment) characteristic of the bug. Examples of high priority bugs are the application does not allow any user to log in, a particular functionality is not working or the client logo is incorrect. As you can see in the above example, a high priority bug can have a high severity, a medium severity or a low severity.
Its value is based more on the needs of the end-users.	Its value is based more on the needs of the business.
Its value takes only the particular bug into account. For example, the bug may be in an obscure area of the application but still have a high severity.	Its value depends on a number of factors (e.g. the likelihood of the bug occurring, the severity of the bug and the priorities of other open bugs).
Its value is (usually) set by the bug reporter.	Its value is initially set up by the bug reporter. However, the value can be changed by someone else (e.g. the management or developer) based on their discretion.
Its value is objective and therefore less likely to change.	Its value is subjective (based on judgment). The value can change over a period of time depending on the change in the project situation.
A high severity bug may be marked for a fix immediately or later.	A high priority bug is marked for a fix immediately.
The team usually needs only a handful of values (e.g. Showstopper, High, Medium and Low) to specify severity.	In practice, new values may be designed (typically by the management) on a fairly constant basis. This may happen if there are too many high priority defects. Instead of a single High value, new values may be designed such as Fix by the end of the day, Fix in next build and Fix in the next release.

I hope that you are now clear about the difference between severity and priority and can explain the difference to anyone with ease.

Top reasons given by testers when bugs are reported by clients

2010-05-06T14:34:00.000+05:30

Well, we have seen the Top replies given by programmers when their programs do not work correctly. Here is a list of reasons given by testers when they do not report or even find (important) bugs.

I found this bug BUT...

1. It was not approved for submission (by the test lead/ test manager/ fellow testers/ programmer).

2. The bug report was rejected. (never mind the reason of rejection!)

3. This bug is reported but as part of another bug report which is still open.

4. Did not report it because it is intermittent in nature.

5. Reported it verbally due to lack of time.

6. So many bugs are still open. It would not have made sense to report yet another bug.

I did not find this bug BECAUSE...

7. I was not informed that this functionality is complete (and to be tested).

8. This bug is only visible with negative testing and all our test cases are positive test cases.

9. There is no existing test case to find this bug.

10. The test case to find this bug is not in our test plan.

11. This bug can be found only by the client's test cases which we do not have.

12. This functionality was blocked during my test.

13. I have tested this module briefly (I was just assigned this module OR this module was re-assigned to another tester quite early).

14. I have been busy re-testing the numerous bug fixes.

15. They stopped the testing before I had the time to test this.

16. It worked fine the last time I tested it. They must have changed the application after that.

17. It worked fine with the test data that I used.

18. This bug is related to the latest changes in the requirements, about which I was not informed.

19. This bug is specific to the client's environment.

20. If you examine it carefully, it is not really a bug.

Don't worry; we have all used excuses one time or the other. By the way, did you note the similarity to the top replies given by programmers when their applications do not work correctly?

When you lead a team of testers, you should watch out for these remarks and make your team culture and test process robust enough to prevent these problems from occurring.

What is regression testing and the best way to do it?

2010-05-05T21:12:00.000+05:30

First, the basics. The term "regression" is used to describe the decay, weakening or degeneration of software. Software is modified to add enhancements to it, fix known defects, make performance improvements, make the code more maintainable or make the code compliant to patterns. But, software can regress when it is modified. This can happen in a number of ways:

a. The faulty design is re-used to create an enhancement

b. Bug fixes get rolled back

c. Prior bug fixes do not work anymore due to changes in the environment (e.g. a new browser version)

d. A bug fix in one part of the software creates a new bug in the same area or a related area

Regression testing determines if extra defects have been created in the modified software. As a tester, you are faced with a number of choices while performing regression testing:

1. At what levels should you perform your regression tests?

2. Which test cases do you execute (the existing test cases or new test cases or a combination)?

3. How frequently should you run the regression test?

4. Which part of the regression test should be automated and which part should be manually executed?

5. How do you ensure that your regression tests are effective?

6. How do you ensure that your regression tests are optimized?

You can come up with the best way to do regression tests if you select the best options to the above questions. Therefore, let us discuss these questions in detail.

1. At what levels should you perform your regression tests?

As you know, software testing happens at a number of levels (e.g. unit testing, integration testing, system testing and so on). Since the focus of testing is different at different levels, you are going to have a better likelihood of finding regression defects if you perform regression testing at different levels. For example, you may find defects in unit testing which would not have been captured by your existing system tests. You may find defects in integration testing which would not have been captured by any unit tests.

2. Which test cases do you execute (the existing test cases or new test cases or a combination)?

First of all, you should be aware of the impact of the changes to the software (since the last regression test). Then select a suitable set of test cases that sufficiently exercise the impacted areas of the software.

You may want to use the existing test cases if they are sufficient in their coverage. Further, they should have been updated along with the prior changes to the software e.g. the test cases are updated based on the defects discovered in the past.

If you know that the existing test cases have skimpy coverage or out of date, you may want to create new test cases for your regression test.

You should also ensure that there are no duplicate tests in your chosen regression test suite.

3. How frequently should you run the regression test?

This depends on the cost you incur in running your regression test and the value you receive out of it. Even if you don't run a regression test every day (note that some teams do), you should at least run one complete regression test before the software is released to production or to your client.

4. Which part of the regression test should be automated and which part should be manually executed?

Tests which are stable, repeated frequently, simple and require no intervention by the tester are good candidates for automation.

5. How do you ensure that your regression tests are effective?

Your regression tests should be able to discover defects. Upstream regression tests should discover a greater number or the more severe defects or at least discover defects more easily than downstream tests.

6. How do you ensure that your regression tests are optimized?

If you are aware of the scope and timing of the build process, you can align your regression tests with it. This will lead to an optimal number of regression test runs.

Further, you should examine your regression test cases to eliminate duplicate test cases, merge test cases wherever possible and automate tests (based on the criteria above) to minimize the time/ effort it takes to run your regression test.

How to design test cases without any requirements specification (or any additional documentation for that matter)?

2010-05-03T19:52:00.000+05:30

It makes for a great discussion. You have been handed an application with no requirements specification. You are supposed to create test cases. Can you do it? Sure, you can. Just look at the following test ideas.

Does the application launch?

Does the application have a help/ demo file? You can find abundant information in the help/ demo file to help you design your test cases.

Does the application accept user input?

Does the state of the application change on accepting each user input?

Does each control in the application work? Examples of controls include menu items, toolbar buttons, links, text boxes and buttons.

Does each controls have a consistent look and feel (e.g. style, size, font and alignment)?

Is each label or text in the application spelt correctly?

Is it possible to copy/ paste data to/ from the application?

Does the application show all the displayed data (with or without scrolling)?

Is it possible to perform the tasks promised by the name of the name of the application?

Does the application close?

(If it is a Windows application) Does the application follow the common standards for Windows applications?

(If it is Web application) Does the application follow the common requirements for Web applications?

I am sure that you can come up with more test ideas. And this is an extreme example. In the real world, even if you know nothing about the application for which you are going to design tests, you may have one or more of the following resources to help you beside the requirements specification:
1. A knowledge transfer (either in person or via a document) regarding the application
2. High level business requirements
3. Design documents
4. Business analyst or product manager
5. Project manager or developers
6. Prior versions of the application
7. Older requirements specifications
8. Past bug reports or customer complaints
9. Installation guide and release notes
10. Your domain/ industry knowledge
11. Laws/ statutory requirements that must be satisfied by the application

Popular open source test management tools

2010-05-03T17:49:00.000+05:30

As the name indicates, test management tools are used to create and manage tests. A test management tool commonly contains features to create:

Requirements specification

Test cases (based on the requirements)

Test plan (usually some test cases grouped according to their priority)

Reports and metrics based on the test plan execution

Built-in defect management sytem or integration with defect management systems

If you are looking for free/ open source test management tools, here are two candidates that you should seriously consider:

1. Testopia
It is the test case management extension for the popular defect management tool, Bugzilla.

2. Testlink
It allows creation of test cases and test plans, tracking test results and generating reports. It integrates with other defect management systems like Bugzilla and Mantis.

Funny things that testers hear!!!

2010-05-01T19:36:00.001+05:30

One comes to learn about the things that testers in various companies hear from their project managers. Enjoy.

1. You need two days to write test cases!? You already have the requirements specification. Just copy and paste from it.

2. You don't have the latest requirements!? I am positive that we communicated the latest changes to all developers.

3. Remember that we are using the Agile methodology. Time is critical. Do not waste it by creating any test cases or bug reports. Just test and discuss the bugs directly with the concerned developer.

4. Note that the build you are testing can change any time. Even if you notice that the build has changed, just continue your testing as if nothing has happened.

5. There is no need to test the xxxx module. I have already had a [senior] developer test it and he said it is working fine.

6. You are new to the project. We will have another [senior] tester repeat your tests and compare both your results.

7. I want to be aware of each bug as it is found. In addition to logging the bug report, call me/ mail me as soon as you find a bug.

8. You have found a bug? Are you sure that you have tested the latest build [implying a very recent untested build]?

9. What bug did you say you have found? Please confirm each bug with the developer before raising/ logging it.

10. Why doesn't your bug report have a test case ID? Note that no bug will be accepted without an existing test case ID.

11. Your bug is a duplicate of bug ID xxxx. Note that NO [implying not even the first] duplicate bugs will be entertained by the developers.

12. Finally, It is 8 p.m. The developers have worked very hard to create this release. Now, they have all gone home. You can do all your testing. Just remember that the release has to be delivered to the client first thing tomorrow morning.

Okay, I admit that these things are not at all funny for the recipient at the time they are said. But, they are hilarious once you think or talk about them later :)

Interviews of Thought Leaders - Alan Page (final part)

2010-04-29T20:31:00.000+05:30

The first part of the interview with Alan Page was read and appreciated by many. All, many thanks for your responses and questions. For me, some of the things mentioned by Alan are sure to change my approach to testing in subtle but fundamental ways. Well, here is the second and final part of Alan's interview:

7. What skills and qualities would you look for if you interview a software test engineer for the purpose of hiring?

[Alan] I interview a lot of candidates on my new team, so I can pull from recent experience. As hinted above, I look for a passion for learning – I try to find out how and when testers learn more about what they do – I want to understand what they know about testing and how they form their opinions about testing. Problem solving relates pretty well to this, and also to test investigation and debugging. I don’t ask the silly “how many piano tuners are in the world” type of questions – instead, I usually just ask test candidates to troubleshoot some random problem – e.g. an application won’t start, what do you do? This gives me some insight into methodology, problem analysis and creativity. I also look for test design (or test aptitude). For example, I may ask how to test a random number generator or a string search function. Both of these examples are simple on the surface, but have a lot of deeper problems under the covers. Good test candidates can see the whole problem quickly and get creative on testing the challenging parts (I left the details vague on purpose - don’t want to ruin the “fun” for potential candidates).

8. What advice would you give to the software test leads/ test managers working in various organizations? What should they do if they want to progress in their testing career?

[Alan] A big part of your role as a test lead or manager is to make sure that your team is performing and growing. The work you assign for your team needs to be challenging, yet achievable. As a lead or manager, you need to base your success on your team’s success – and take the heat if/when things don’t go well. I sometimes see test managers who have good ideas, but think that mandates are the only way to make change. Managers need to learn other methods of organizational change and use a variety of approaches when they want change to happen. I also encourage test managers to stay involved in the testing process. Some managers think that test management is just about management, and not about testing – these managers almost always fail.

9. Many software testing professionals find effort estimation a bit challenging. What advice can you give to arrive at a fairly accurate estimate, especially in automation projects?

[Alan] First off, as you may know, I’m not really a fan of GUI automation – but it can work if planned correctly. If you can involve development in the estimate – e.g. if programmers and testers can come up with the estimate together, you may get closer. The problem is that there usually too many “unknown unknowns” in GUI automation efforts – if you make programmers aware of this, they are more prone to add the appropriate testability features (e.g. an object model) to the UI so it can be tested in a less fragile manner.

Beyond that, normal estimation practices apply – break tasks down into the smallest unit possible – preferably a day or less. This makes it much easier to know when you’re behind (or ahead of) schedule before it’s too late, but more importantly, makes you take the time to fully understand the task in front of you.

10. We know that you handle sizable responsibilities. In addition, you are a frequent conference speaker and author. Could you give us some tips on how do you manage your time and energy so successfully?

[Alan] I don’t work 80 hour weeks – at least I don’t think I do. I have a job, I have stuff I do that’s related to my job (blogging, speaking, writing, etc), and I have my personal and family time. But I don’t really segregate them – they all sort of merge together into the 16 or 17 hours I’m awake every day. Sometimes I take care of personal stuff (I try to run 3-5 miles every day) or blog during the day, and I usually work from home for an hour or two at night after the kids go to sleep on whatever is important.

As far as managing my time, it’s very scrum-like. I have a “product backlog” of everything I do in my outlook tasklist. It includes my work projects and personal things (like “do my taxes”, or “set up conference with teacher”). Some items are recurring. My scrum team of 1 does 1-day sprints – each morning, I go through everything on the todo list for today and prioritize. Some items get deferred if I know they aren’t critical for that day, and sometimes I’ll add things. I don’t get through everything I plan every day, but I put the unfinished items back in the backlog and re prioritize the next day. It may sound a little silly (I don’t actually think of it as scrum-like when I’m doing it), but it’s the only way I can stay on top of everything I’m doing.

I frequently remind people (and myself) that there is always time to do whatever you want – you just need to make the right choices to make that time for yourself. You may think, for example, that you don’t have a single spare moment the entire week – but if you were so sick you couldn’t get out of bed for two days, I bet you’d be able to “catch up” within a week. As long as you’re healthy, take the time to invest in what you think is important, and your task list will work itself out. (note – I blogged about this here if you need clarification or elaboration)

Of course, knowing when to say “no” is also critical. I hate to turn down writing or speaking requests, but I still have to do it more often than I’d like.

11. Do you think that you made the right choice in selecting software testing as a career?

[Alan] Absolutely – I never thought I would be a software tester this many years, but I can’t imagine doing anything else. At one point in my career, I briefly entertained the idea of becoming a sw developer, and was even assigned some feature work and bug fixing on a few different occasions at Microsoft (I have written functionality that shipped with Windows and Windows CE). On both occasions, however, I realized that my heart was in testing (for example, I enjoyed investigating and fixing the bugs far more than writing features from specifications. I’m sure I’ll be a tester until I retire – and I hope that’s a long time from now, because I’m still having fun and enjoying the job a lot.

12. Finally, would you like to share any other advice/ wisdom with us readers?

[Alan] –Sure - The biggest advice I can share is to focus on learning and develop your own thoughts and opinions. I see too many testers chasing after silver bullets and shiny objects – blindly trusting testing advice from tool salesmen and self-proclaimed testing experts. If you haven’t learned enough and practiced enough to form your own opinions and build your own toolbox of testing approaches, you have little chance of success.

Many thanks to Alan for taking his valuable time to pen his thoughts on pertinent questions from all of us in the software testing space.

Please comment if you have any related questions. I will do the same.

Automated test data generation

2010-04-28T17:59:00.001+05:30

Among other things1, you need test data to execute tests. You are required to design the test data to drive your test cases. If the test data is not already listed with your test cases or you run tests in an exploratory manner, you need to design the test data on the fly. Generating your own test data takes away your valuable time. Generating test data by hand is especially time consuming if you have to run automated tests (e.g. functional tests or performance tests) which need a large quantity of distinct test data.

Of course, if the production data is available, you may get its copy and use it for test. However, this is not a very wise approach. Using live data in test may expose confidential customer information to unauthorized people. In fact, acts like HIPAA address the security and privacy of data.

The good thing is that you need not rely on generating test data by hand. Several tools are available that can help you create large quantities of test data based on the constraints or conditions specified by you. Automated test data generation has the following benefits:

Fast	If the number of test data items is large, automated test data generation may be substantially faster than creating the same test data by hand.
Accurate	Since test data generation is a tedious task, you are likely to make mistakes. Test data generated by a tool can be much more accurate.
Can use built-in algorithms	The test data generator may implement algorithms to generate correctly formatted special data like postal codes and email addresses. Therefore, you need not look up the algorithms for generating the special test data yourself.
Create both valid and invalid test data	Just as you can create valid test data, you can also create invalid test data by specifying incorrect constraints. Further, by specifying the quantity of the valid and invalid data, you can control the percentage distribution between valid and invalid data.
Data exportable to multiple formats	The automated test data generators usually provide options to either generate the test data directly in the desired format (e.g. Excel, CSV or SQL) or export to the desired format.
Create a professional look	Since automated test data generators use their own databases to source data, the test data added/ edited in your application looks professional. This way, you can avoid test data that looks odd (e.g. Name as aa or abab12). This is important in a way because the test data used by you is stored in your application and reported in your bug reports. If your client or management looks at your bug reports, you would feel better if the test data looks good (e.g. Name as Jackson or Burke).

Even though an automated test data generation may offer so many benefits, you should remember that it is just another software. Like other software, it may or may not have all the features you need or always generate the correct data. However, if you are willing to try automated test data generation, you could be boosting your productivity or the productivity of your team.

Here are a couple of free test data generators:
1. Data Generator
2. TestersDesk It requires you to register first. The Test Data Generation Toolkit has a number of options to generate different kinds of test data.

Note: I said "among other things" since sometimes it is necessary to populate an application with test data before the application can even function (fully).

How to create a powerful test strategy?

2010-04-27T19:24:00.000+05:30

Before I begin this post, let me talk about an email that I received from Natalya in Ukraine. She said that she found STS interesting and educational. She tries to read it every day. Natalya, let me say that I am grateful and honored to be of help. Since you are a regular reader of STS, this post is dedicated to you.

Let us begin with discussing what is a Test Strategy? As you know, Strategy means using the resources at one's disposal to achieve the desired objectives effectively. Test strategy means the plan that details how the testing objectives would be met effectively.

Like any other strategy, the test strategy may be defined at different levels. You may have a test strategy at the organization level, at a program level or at a project level. The thing is that it may not be called a test strategy at the organization/ department/ program level; it may just exist as a management policy or as a part of a governance plan. At the project level, the test strategy may just be a part of the test plan. Further, depending on the nature of the project, the test strategy defined at the project level may or may not satisfy the test strategy outlined at a higher level.

The advantages of an effective test strategy are:
1. It mitigates the risks to objective testing.
2. It helps focus on different aspects of the system under test (by using distinct test phases such as unit testing, integrating testing and so on).
3. It provides clarity on the required people, procedures, tools and infrastructure.
4. It can be used to communicate the test approach to the various stakeholders (clients, project/ test managers, developers and testers) in the project.

If you want to define a powerful test strategy for your project/ program/ organization, you should consider the following tips:
1. Before you define your test strategy, you should research the client(s) and the end user(s) with respect to the their needs and expectations from the application(s).
2. Your test strategy should aim at satisfying the test objectives alone.
3. Practically speaking, you may need to create two versions of your test strategy depending on the situation - one for communicating to all and a detailed/ tailored one for agreement with the key stakeholders.
4. You have the option of having different kinds of focus in different test phases. However, do not go over-board with defining too many test phases with overlapping effort. In other words, economize with the test phases. Define only the ones which you think are really required.
5. Your test strategy should be totally customized according to your specific situation. What has worked for other companies or what has worked for your company in the past may not apply to your current situation.
6. You should consider the required test environment while designing your test strategy. You should at least have one test environment that is the same as (or is a replica of) the production environment. Further, your test strategy should assume the most common test environment configurations for the most part.
7. Your test strategy should define the testing tools (test/ defect management systems and/ or automated test tools). It is best if you have evaluated the testing tools yourself. If not, base your decision on reliable (vendor-agnostic) and popular user reports.
8. You should look at any defined test processes (e.g. entry/ suspension/ resumption/ exit criteria for test, process to execute test cases and process to report a defect) to examine their feasibility in your situation. Identify the required test processes that will be re-used, modified or created from scratch.
9. Identify the data that will be recorded, measured, analyzed and reported to show the progress of testing.
10. Challenge all assumptions while you design your test strategy. Provide safeguards if any of the prior assumptions prove incorrect later.
11. Finally, remember that a powerful strategy is never static. Monitor the results obtained from the test strategy and do not hesitate in making the desired modifications to your test strategy.

Interviews of Thought Leaders - Alan Page

2010-04-23T18:09:00.001+05:30

As promised in my post, Interviews of Thought Leaders in Software Testing, here is the first interview. This interview is with Alan Page. Alan is the Principal SDET at Microsoft Corporation. In the past, he has also played the role of Director, Test Excellence at Microsoft. Along with being a frequent conference speaker and author, Alan is a prominent thought leader in software testing. Alan also writes on his blog at angryweasel.com.

Though I had been reading about Alan and his thoughts before, my interaction with him began in the year 2008. I respect his vast knowledge, professional responses and helpfulness. In this interview, you can read Alan's thoughtful and detailed responses to what I hope are pertinent questions from all of us in the software testing space. Who know, some of Alan's replies may subtly lead you to even greater testing successes.

1. We know that you are a very senior person in the area of software testing. What roles have you played in your software testing career so far and what other roles would you like to play in the future?

[Alan] I don’t know that I’m senior – perhaps just highly exposed. I started testing before I knew what testing was – I was hired to perform product support for a music software company and was notified the first day that I was also the (only) software tester. When I joined Microsoft a few years later I tested configurations and user interface for the Windows 95 networking components. My role on the windows 98 team was primarily API testing – with a slant towards compatibility testing. Later on, I ended up working a lot with the kernel and drivers and ended up owning those areas of Windows CE as a test lead for just over a year. I’ve always felt I was a stronger performer as an Individual Contributor than as a lead or manager, so when I stopped my stint as a lead I moved into a Test Architect role (which was really just a fancy title for a senior non-manager tester at Microsoft). As a TA, I functioned as a technical leader for the CE test team – acting as a go to person, a reviewer, and a strategist for the team. I spent about five years in Microsoft’s Engineering Excellence group – the first two as a technical instructor and pseudo-consultant, and the last two and a half as the Director of Test Excellence where I was sort of the overseer for Testers at Microsoft – in charge of technical training, long term strategy, etc. In my new role on the Office Communicator team I don’t test a specific part of the product – instead I try to identify everything I can that will help make the test team more effective and improve overall software quality.

In the future – at least for the next few years, I see more of the same – perhaps with larger teams or tackling even harder problems.

2. What are the different kinds of tests that you have done or led/ managed/ directed?

[Alan] You name it, and I’ve probably done it (although not always successfully).

3. Could you tell us about a particularly tough challenge that you faced in your testing career? How did you resolve it?

[Alan] My toughest challenge is always my next challenge – I’m always looking for new challenges. However, I’ve probably had a few particularly interesting challenges. The first that comes to mind was when I was first asked to join the Windows ME team.. The debug version had a ton of great error checking that enabled application developers to find a slew of programming errors early. The problem was that the debug version of Windows had been ignored for completely a year and wouldn’t even boot. In addition to kernel and driver testing, I was asked to “fix” debug windows (and own debugger development as well). I had no idea what I was doing, yet somehow happened to make it all work. It was one of those times when I felt I was completely over my head, but somehow managed to figure it out. I’ve always been able to learn quickly and to hold a bunch of disparate information in my head and somehow put it together in the right order.

4. What is the one achievement in your testing career that you are most proud of?

[Alan] It’s probably writing (and finishing) How We Test Software at Microsoft. Sometimes I still can’t believe I took on that project, but I’m glad I did.

There are plenty of things I wish I would have done differently in that book, but I’m still proud of getting it done and that people like what we came up with.

5. How do you think the software testing space is transforming now? What specific software testing roles/ jobs would increase in the future? What skills and knowledge do you think will be required by software testers in the coming years?

[Alan] Software is getting more interconnected and complex rapidly – the challenge of testing this interconnected and complex software is growing rapidly.

It’s really not enough to find surface level functionality bugs anymore – the important bugs are lurking within the connection points where functions, modules, and features connect. Testers are learning that they need to have a strategy for determining where to target testing, and what types of testing to perform to get the maximum value from their time.

Testers spend a lot of time looking at data – today, it’s most often bug data and pass rates, but many testers also look at coverage reports, code churn or complexity metrics. The problem is that most testers aren’t data analysts or statisticians, and aren’t really very good at figuring out what action to take from data. The result is that the data is either ignored, or misinterpreted. I think the business intelligence role will become more important in the future of test – without meaningful analysis of the massive amount of test collateral generated by software teams, test will never be able to catch up with the growing complexity of future applications.

6. What qualities and abilities do you like in the younger software testing professionals? Which qualities you do not like in the younger crowd?

[Alan] You’re making me feel old :}!. Beyond anything else, I think a passion for learning is the most important attribute of a tester. I also think it’s important to point out that “passion” has to be demonstrated rather than stated. In other words, I admire testers who seek new knowledge and actively look for ways to understand how to apply that knowledge – rather than those testers who state random bits of new information and look for ways to dismiss that knowledge without taking the time to understand it.

Well, this was the first part of our interview. Answers to the following questions are still to come:

* What skills and qualities would you look for if you interview a software test engineer for the purpose of hiring?

* What advice would you give to the software test leads/ test managers working in various organizations? What should they do if they want to progress in their testing career?

* Many software testing professionals find effort estimation a bit challenging. What advice can you give to arrive at a fairly accurate estimate, especially in automation projects?

* We know that you handle sizable responsibilities. In addition, you are a frequent conference speaker and author. Could you give us some tips on how do you manage your time and energy so successfully?

* Do you think that you made the right choice in selecting software testing as a career?

* Would you like to share any other advice/ wisdom with us readers?

Be sure to check out Alan Page's replies to these questions shortly. Is there any other question that you would like to ask Alan?

[Update on 29 Apr 2010] You can now go to the second and final part of this interview.

What features you should consider when you design your Automation Framework?

2010-04-22T20:10:00.000+05:30

Vivek Joglekar, who is a regular reader and subscriber of STS, mailed me asking me to add more stuff in framework and test process. As desired by him, here is a post on the Test Automation Framework. Look out for more posts on the same.

If you are interested in test automation, you have probably come across the term, Test Automation Framework (Automation Framework, in short). Different people have different understanding of the term, Automation Framework. However, an Automation Framework commonly consists of the implementation of certain concepts and code that gives a jump-start or provides support to one or more test automation projects.

Before you design your Automation Framework, you should identify the objectives that you want to achieve with it. The desired features of your Automation Framework may include:

General features
1. Re-use of existing code by writing code once and invoking it elsewhere
2. Consistency of test automation implementation by defining standards
3. Tool independence by implementing your framework as separate software
4. Ease of framework installation
5. Ease of framework maintenance
6. Ease of upgrading the deployed framework
7. Types of testing supported by the framework
8. Supported operating systems
9. Supported browsers

Features related to test design
10. Simple to learn and use
11. Common operations already built-in the framework
12. Help documentation with examples
13. Minimal changes to the test automation in case the application changes :)
14. Integration with version control systems and test management systems

Features related to test execution
15. Ability of the framework to run on multiple systems
16. Support for test configuration e.g. application name, server name/ URL and test cases to be executed
17. Support for scheduled execution
18. Supported invocations of the framework e.g. by GUI or by command-line
19. Support for invoking and closing/ shutting down the application(s) under test
20. Ability to connect to myriad test data sources and collect test data
21. Fast performance
22. Robustness
23. Error recovery e.g. by dismissing unknown windows, retrying a failed operation with/ without configurable delays, proceeding to the next step/ test case and even resetting or re-starting the application(s) under test
24. Integration with defect tracking systems
25. Clean up test data in the application(s) under test after completing each test

Features related to test reporting
26. Logging states and results
27. Reporting test results (in one or multiple formats e.g. Excel or HTML) with metrics such as number of test cases in the test plan, test cases executed, test cases passed, test cases failed and the time taken to execute the test cases
28. Distributing test results e.g. by publishing to web or email

When you estimate the effort it is going to take to build the desired features in your Automation Framework, you may find that the time available is not sufficient. You should then prioritize the features and build the most important ones first. Start small. As your Automation Framework is used in test automation projects, you may learn about the problems faced while using your framework. You should then prioritize the fixes and enhancements and build upon your existing Automation Framework accordingly.

Interviews of Thought Leaders in Software Testing

2010-04-21T14:20:00.000+05:30

I have been planning to interview prominent figures in the space of software testing for quite some time now. These are the people who are leaders in software testing. They have excellent command on the topics of software testing and have been helping and influencing software testers across the world.

The first interview in this series would be with Alan Page of Microsoft Corporation. If you do not know Alan already, he is the Principal SDET at Microsoft Corporation. In the past, he has also played the role of Director, Test Excellence at Microsoft. Along with being a frequent conference speaker and author, Alan is a prominent thought leader in software testing. Alan also writes on his blog here.

Watch out STS for a candid interview with Alan Page.

How to set up a new testing department?

2010-04-20T20:53:00.000+05:30

Many companies choose to set up a testing department instead of (or in addition to) allocating testers permanently to projects. The rationale to do so include:

A. Availability of a more comprehensive skill set
You may have access to a larger skill set (e.g. manual as well as automated testing, functional as well as performance testing and knowledge of a number of different business domains) if you have multiple members in your department.

B. Better management of the testers' time
Testers are commonly under-utilized when the code is being written and are very busy when the application is under test. Pooling testers together provides an opportunity to level their task assignments better.

C. More learning opportunities for testers

The testers in a testing department may get chances to work on multiple projects. This provides them opportunities to enhance their existing skills as well as develop new ones.

The membership of the test department may vary from just one member to several hundred members. A large testing department may be called a testing organization, a testing practice or a testing center of excellence in the company.

If you are a manager tasked with setting up a new testing department, here are the things that you should consider for your plan:

1. Objectives of your test department

You should be very particular about eliciting and understanding the objectives of your testing department. Everything else should map with these objectives. You should know the scope of activities that the testing department would perform. Examples of objectives could be:

a. Support pre-sales activities for new clients from a testing perspective

b. Provide testing for the flagship product(s) of the company

c. Provide testing on company projects

d. Deliver testing projects

e. Undertake research on testing topics

2. Team members required

You should estimate the team size (number of full time/ part time team members) required depending on the size of expected work as per the objectives of your department. You should also list the skills (technical, business and soft skills), experience and/ or testing certifications that should be available with the team members.

3. Tools required

In order to work effectively, your department may need tools. Such tools may include test management tools, automated testing tools (e.g. functional testing tools and performance testing tools) and project management tools. You may not want to purchase any tool licenses ahead of time. However, you may want to evaluate several tools of each type so that you may quickly make a decision to buy license(s) later.

4. Test processes required

In order to enable your team to work consistently and quickly, you would need a number of core testing processes. Examples of the testing processes include:

a. Test effort and duration estimation

b. Test methodologies for various types of testing

c. Test automation process

d. Defect management process

5. Internal processes required

In addition to the core testing processes, your department would require internal processes for supporting the team members. Examples of these processes include:

a. Planning, scheduling and tracking the testers time

b. Induction of new team members

c. Career development and appraisal process

d. Training

e. Team building

6. Infrastructure required

Depending on the objectives of your department, you may need to procure and establish the required infrastructure (hardware including special devices, software including tools and frameworks and network including devices and bandwidth). You would create, modify or optimize the required test labs from this infrastructure available to you.

Once you have planned your testing department, you should go about obtaining or creating the necessary assets. However, you should keep the following tips in mind while doing so:

I. The objectives of your department may change after it is established. This usually happens when your department shows promising results with some work. Therefore, you should analyze your department in light of the new objectives and re-plan accordingly.

II. It takes vastly different durations of time to obtain different types of assets. For example, you may purchase a tool license in a day but you may need to wait a month to recruit a new team member or get a new server and you may take 6 months to develop your own estimation process. Keep this in mind.

III. A testing department commonly has numerous assets. Therefore, you should plan long-term with the goal of utilizing each asset to a reasonable extent.

Now, you should know that setting up a new test department requires establishing its objectives, obtaining the team members, tools and infrastructure and creating or reusing core testing processes and internal processes. I hope that your test department is able to fulfill its objectives and more.

Free functional testing tools for testing Windows applications

2010-04-19T17:58:00.000+05:30

I have seen several requests (especially on LinkedIn) from software test professionals looking for free automated testing tools that work with Windows applications. Here is a list of tools that you may want to look at and evaluate for your test automation project:

1. pywinauto

2. AutoIt

3. AutoHotkey

4. TestSmith

5. QAliber

What is the best way to test screen access based on roles?

2010-04-16T18:00:00.000+05:30

Many applications implement role based security and role based access. This means that when the user accesses the application, the application determines the role of the user first. The application shows only those screens to the user which are meant for the particular role. The application may even hide or show certain parts or sections of a particular screen depending on the role of the user. For example, an application may show only the public screens to an unauthenticated user. If the user logs in as a member, the private screens of the particular member become available to the user. If the user logs in as an administrator, the private screens of the particular administrator as well as the application administration screens become available to the user.

If you have to test the screen access of your application, you should go about it systematically. Here is the approach that you can use:

1. List the various kinds of access in the application.
Examples of access include View (just see the information listed on a screen), Read (see more information of a data item), Drill down (see the sub-items of a data tem), Add (add a data item), Update (edit a data item), Delete (remove a data item) and Comment (add further information to a data item).

2. List the various roles in the application.
You should refer the requirements documentation of your application to list the roles in your application. Examples of roles are User, Super user, Customer, Staff, Manager and Administrator.

3. List each screen (and each section within each screen) in your application.
You should refer the requirements documentation as well as the design documentation of your application to list each screen in your application. If your application implements role based access on sections (parts) of any or all screens, you should group all the relevant screen sections under the screen name.

4. Map the roles to the screens listing the access in each case.
Finally, you should map roles to the screens. In order to do so, simply list all roles on one axis and all screens on the other. Since the number of roles is commonly far less than the number of screens, you may want to list the roles horizontally and the screens vertically for ease in viewing or editing your mapping. See the example above.

5. Ensure that your mapping is complete.
Ensure that you have listed each access, each role and each screen (with each section that is accessible separately) in your application. Ensure that the mapping is present between each role and each screen/ screen section.

How to be a recognized software testing authority in your company?

2010-04-15T21:56:00.000+05:30

You know how it is if you work in a company with multiple software testers. The management's perception differs for different members of the software testing team. A few people are held in high regard. They are the ones who are first introduced to new or existing clients. Their names come up first when the membership of a new project is discussed. Then there are others, who despite having good knowledge and a respectable experience are seemingly always sitting on the sidelines.

Don't you want to be the apple of your company's eyes? Use the following tips and lift your image in your company.

1. Update the list of your skills.
You learn new skills all the time while working in your project. Instead of waiting for the company performance evaluation/ appraisal exercise, make a list of the new skills that you have acquired. Submit the list of your skills to be updated in your company's records.

2. Participate in meetings actively.
You probably attend a number of project meetings/ division meetings or company meetings. Participate actively in these meetings. You may start by preparing questions for the meeting and then raising them in the meeting. If you are comfortable, you may provide informed comments to the topic of discussion in your meeting.

3. Ask your client/ manager to recommend you.
There may be occasions when your client or your manager is very happy with the work done by you. You should ask to be recommended but do it tactfully. If you receive a written recommendation for your good work, you can share it with others and also keep it for future reference.

4. If possible, try to get into a high visibility project.
We know that of all the projects being executed, a few projects are more important (e.g. in terms of size, cost, duration, returns or prestige). Naturally, the important projects (and the team members working in it) are monitored closely and more often.

5. Create generic assets in your project.
You create a number of assets (e.g. documents, templates, processes, procedures and utilities) in your project. Instead of creating an asset just for your project, you may create a generic one with a little more effort. The advantage of creating a generic asset is that it may be used in other projects (of course, with the permission of the owner of this intellectual property).

6. Learn and try to solve a few problems in other projects.
If you know about any urgent problems in another projects and are able to invest the time to help solve them, just go ahead.

7. Keep an eye on the upcoming projects.
If you come to know about the projects in the pipeline, you may be able to get the information to prepare yourself for the same. Company meetings and the office grapevine are some ways to get information on the upcoming projects. If an upcoming project and your skill set match, ensure that the relevant resource manager knows about this match.

8. If possible, get on site experience.
Working directly at the client's site is a great way to know about the client's business, their processes, their working culture and to develop personal rapport with the client's personnel. If you have a successful on site experience, do not be surprised if the client asks for you by name for the next project.

9. Develop rapport with the other managers in your company.
You probably already share a rapport with the manager you report into. Introduce yourself to the other managers in your company. Let them know about your experience, knowledge and skills. A good way to do this is to talk to them after-hours (when things are not so hectic).

10. Write in your company's newsletter.
If your company runs a newsletter, they probably keep looking for contributors for every issue. You may research and write a piece on software testing.

The above tips may require a little more effort from your side or may require you to change your approach slightly. However, they can be very effective in changing how you are perceived in your company and in elevating your image.

What are the challenges that you can face in your testing career?

2010-04-13T21:47:00.001+05:30

There are challenges that you may face as you progress in your testing career. By a challenge, I do not mean a (pressing) problem that you face temporarily in your project. Rather, a challenge comprises of a complex (and usually, a group of) long term problem(s) that we may face while working for a particular client or while working with our current organization.

In my opinion, we encounter a challenge when our thought process and the thought process of an important stakeholder (e.g. the client, your management or your team) do not match. Here are some examples:

Challenges due to your client or management
1. Your client or management thinks about testing as a cost which should be minimized. As is usual with cost approvals, you need to build a solid case for every piece of testing that you think would help.
2. Your client or management thinks about testing as a very predictable process. You are expected to prepare very detailed test plans (akin detailed project schedules) and explain every discrepancy between the planned and actual activities.
3. You receive little input on your planned tests. However, if bugs escape to the delivered application, you are grilled about your test plans.
4. There is limited or no budget for testing tools or support tools. You have to spend a lot of your valuable time to manually perform certain repetitive tasks.
5. The test environment is too restrictive (e.g. it is available only for limited times, it is also used by other teams or it is ridden with stability problems). You end up spending a lot of time waiting for the test environment to be available or need to check it constantly for problems.

Challenges due to your own abilities
1. You have been asked to use a tool that you know little about. Therefore, you face two problems. First, you need to learn the tool. Then, you need to find ways to effectively use the tool in your project.
2. You are expected to use a communication style that is different to your natural communication style. For example, if you prefer a verbal communication style, you have been asked to create detailed written reports in your project. Or, if you prefer to take time to carefully draft written reports, you are now expected to provide constant verbal updates.

Challenges due to your team
1. You have been added to an established team and you are finding it hard to integrate and collaborate with the team quickly.
2. Your team members are less skilled in certain testing activities. Therefore, you find that the bulk of the required work has fallen in your lap.
3. Your team members are casual in their approach to testing. You now need to double check the completeness and accuracy of their work.
4. Your team is located afar and you lack instant updates from them.

Challenges due to another team or individual(s)
1. Word is spread unofficially about your poor performance or the poor performance of your team. This discourages you.
2. Due to any reason, another team (e.g. the development team, the deployment team or the IT team) does not effectively collaborate with you or your team.

Realizing a challenge and coming to terms with it takes time. It requires your long-term commitment to resolve a challenge. Then, you need to put in constant effort to either ease the challenge or resolve it partially or completely. The good thing about handling challenges is that they force you to solve a complex problem and you emerge even more confident than before. If nothing else, talking about a challenge and how you resolved it makes a great story. What is a big challenge that you have faced in your testing career so far?

Why your offshore test team cannot perform (well)?

2010-04-02T18:14:00.000+05:30

Companies have been off-shoring work due to cost advantages and unavailability of local personnel as well as for mitigating risk and due to other practical compulsions. However, over the years I have heard about numerous instances when the offshore test team was unable to perform well. The issues in the performance of the offshore test team have the potential of breaking down the entire communication between the on-site team and the off-shore team. The presence and severity of these issues depend on

a. The level of commitment of your on-site management to the success of the off-shore team

b. Your level of knowledge of the people, processes and tools off-shore

c. The off-shore team's level of knowledge about the on-site people, processes, tools and working pressures

d. The intensity of communication and transparency between the on-site and the off-shore teams

e. The level of support given by the off-shore management to the off-shore team

f. Factors attributable to individuals

Depending on the above factors, your off-shore team may or may not face issues with their performance. In fact, I have experienced off-shore test teams that perform well as an integral part of the overall test organization. Regardless, you should be aware of the potential issues with the performance of the off-shore test teams. Let us see what these issues are.

Issues due to poor on-site management

1. Without realizing it, you do not give due credit to the work done by the off-shore team. Your on-site team members duplicate this approach.

2. Due to personal reasons, the on-site team members do not share information readily with the off-shore team.

3. The on-site and off-shore team members do not travel to the other location. Therefore, they only have a working relationship but no rapport with each other.

4. You micro-manage the off-shore team, wanting to know about their daily plan in detail and asking them to update you after finishing every task. This leaves them with lesser time to focus on their core tasks.

Due to your other commitments, you do not review the work done by the off-shore team at all or in a timely fashion.

Issues due to distance

5. Due to differences in time zones, your off-shore team does not participate in your important meetings. They miss out important information updates and come to know about them (much) later, may be after there is a problem. Due to lack of timely communication, they do not realize the urgency of situations developing in the project.

6. If you wish to pass on important information or discuss something urgently, you have limited options other than to write an email or wait to call the next day.

7. Using email as the primary means of communication forces you and your off-shore team to compromise on the completeness of information exchanged.

Issues due to language differences

8. The off-shore team cannot understand your verbal or written updates or instructions completely or in the spirit intended by you.

9. You have trouble understanding their communication e.g. verbal status updates or defect reports.

Issues due to company differences

10. The rigor of the processes followed by you and them is totally different. For example, you are agile but your off-shore team has been following a regimented approach in their projects so far.

Issues due to skill level differences

11. Some team members off-shore have little knowledge of testing.

12. The off-shore test team knows little about your application/ product/ service. Due to this, they focus on the wrong areas of the system under test or miss obvious problems in test.

13. They are not familiar with the tools used by the on-site organization.

Issues due to differences in working style

14. Unlike you, they do not work independently. They wait to be told and do only what you have specifically asked them to do, nothing more.

15. Whereas your working hours are flexible, their working hours are fixed due to local constraints.

Issues due to poor off-shore management

16. The off-shore team does not have a strong and communicative leader. You have a vague idea of the problems faced by the off-shore team or their plan or progress.

17. The work done by the off-shore team is not reviewed locally but submitted directly to the on-site team. This leads to problems that you need to resolve at both ends.

18. Your off-shore team is afraid of losing your work. Therefore, they seek approval from the on-site team members constantly thereby, losing the ability to work on their own with confidence and taxing the time of the on-site team members.

19. Your off-shore team members are simultaneously working on projects other than yours, without your knowledge.

20. Your off-shore team membership keeps changing. This forces everyone concerned to put in their efforts from scratch to bring the new individual up to speed.

Issues due to individuals

21. An off-shore team member is facing a personal issue affecting their productivity. You are not aware of this.

22. The off-shore team has one person who is the incorrect role model for the others. This intensifies the problems of the off-shore test team.

However, the situation is not all bad. You may experience success with your off-shore test team if you have the foresight to identify the potential issues, mitigate them and are motivated to make the relationship work over a long term. Build an off-shore team with hand-picked skilled, reliable, stable and motivated individuals. See what issues exist with your team. Make them aware of the issues and resolve each issue with the help of your off-shore test team. If your off-shore team is like most others, they would be more than happy to make the necessary changes to work up to your expectations.

What issues have you faced in working with an off-shore/ on-site team? Comment on the issue faced by you.

How to do Stress testing correctly and quickly?

2010-04-01T18:58:00.000+05:30

Prem Phulara had commented on my earlier post, What knowledge do you need to have in order to do good or best performance testing asking me to explain stress testing.

This post is dedicated to Prem. In this post, we will discuss stress testing.

Stress testing is a special type of performance testing. It is used to stress (or in other words, put a high load on) the system to determine its highest operational capacity. Stress tests establish the highest load that the system is able to withstand while still being operational (and not throwing too many errors). The additional benefit of stress testing is that it exposes the resources that run out the fastest when the load on the system approaches its maximum sustainable value. It is common to design a series of load tests with increasing loads to find out the maximum sustainable load.

Here are the tips to perform stress testing correctly but quickly:

1. Before you begin work on stress testing your system, you should be aware of the goals of your stress test. For example, your goal could be to determine the maximum throughput of your application for a given business transaction/ workflow on the given infrastructure or your goal could be to determine the maximum number of concurrent users with a given workload and the given infrastructure.

2. It is crucial that you identify the correct performance testing tool for your stress test. Otherwise, you would be forced to make compromises in your tests. See How to evaluate automated software test tools quickly?

Also see Open Source Software Test Tools.

3. You would need to model test script(s) according to the business transactions specified or implied in your goals.

4. If your scripts are correct, a large number of business transactions would execute during your tests. Therefore, ensure that you have a large set of test data that does not exhaust during the test. You may consider sourcing any existing test data from your application's database. Another way of quickly generating some test data is to use Microsoft Excel to extend a series. There are also free test data generators available on the web.

5. As with other tests, you should have an isolated (as far as possible) environment for conducting your tests. Unless you stress test a system on a stand-alone machine, your test environment may consist of the client machine(s), the load generator machine(s), the server(s) (web server(s), application server(s), database server(s) and so on) and the connectivity between the above. You would need your performance testing tool installed on at least one machine. You would set up your test, run it and analyze the test results from this machine. Depending on your choice of the performance testing tool, other load generator machine(s) may or may not require the full performance testing tool installed on them.

6. When you set up your first load test, you should model it realistically. For example, you should add the required scripts to the test, set up the specifications of the Virtual Users (e.g. initial VU, ramp-up, constant load, ramp-down and distribution of VU assigned to each script etc.) and give the run-time settings (e.g. test duration, time lag between iterations, network bandwidth distribution of the VU, their browser distribution etc.). You would really be limited by the features provided by the chosen performance testing tool.

You should specify a low load in your first load test. This is when you could check if your script(s) work correctly (e.g. the script reads the test data, it accesses the system and interacts with it, it is able to send data to the system which is accepted by the system and is able to send/ receive data to/ from other script(s) as required). If you have multiple scripts in your test, you could check if the scripts run in the correct sequence or in parallel, as specified. When your test is running, you should check the performance of each machine in your

test environment e.g. check that the server(s) and load generator(s) have become busier but not to their limit. On Microsoft Windows machines, you may use the PerfMon (Start > Run > PerfMon) tool to add your desired counters and monitor them during the test. See other tips in the post, Performance testing using HP LoadRunner software.

7. After establishing that your load test is valid, you should increase the load systematically. Do not go "big bang" and deploy the load that you think is the maximum sustainable load. For example, if you think that your system may support up to 1000 concurrent Virtual Users, start with a load test with 100 VU and keep on adding 100 VU every 5 minutes. You may find that your system supports 800 concurrent VU but crashes with a load of 900 concurrent VU. After resetting your test environment, you should then start with a load test with 800 VU and keep on adding 10 VU every 30 seconds. Then you may find that your system supports 850 VU but crashes with a load of 860 concurrent VU. Repeat the last test at least another couple of times to confirm your test result.

8. If you recall, I mentioned that the additional benefit of a stress test is to identify bottlenecks or resources that run out the fastest when the load is increased. You should analyze the test results of several of your last tests to find out about these bottlenecks. Maybe it is the network which is clogged, or your web server runs out of available memory. Confirm your finding by repeating the last few tests. Your system's capacity may be improved by increasing the resource that runs out first. Ensure that you include the details of the important bottlenecks observed during your stress test in your test report.

Effort estimation for Test Automation (10+ factors which you should consider)

2010-03-31T20:01:00.000+05:30

This post is in continuation with the earlier post, How to estimate testing efforts (6 approaches to get test effort estimate). In this post, we will discuss the factors that you should analyze in order to arrive at a realistic effort estimate for creating the test automation.

1. Knowledge transfer	Is your team new to the Application under test or would they need training or time with the application to be comfortable with it? Is your team new to the chosen test automation tool(s)? Would they need training on the tool before they are productive with it?
2. Test automation environment	How long would it take to set up the test environment with the application and the tool for each team member in your test automation team?
3. The chosen tool's compatibility with your application	Do you need to perform a test tool evaluation before you begin automation? How compatible is the chosen test automation tool with your application's technologies (e.g. does the tool recognize each type of object in your application, does the tool work fast with your application and so on)?
4. Test automation framework	Would you need to create a test automation framework from scratch? Or Would you need to use an existing test automation framework? How simple or complex is it to learn to use this existing test automation framework?
5. Test cases to be automated	Would you have the test cases available for automation? Are these test cases automatable?
6. Size of the automation	Considering the size of the test case, the speed of your application and the speed of the chosen test automation tool, how long would it take to automate each test case? By how much would the usage of your test automation framework affect the effort of automating each test case?
7. Test data requirements	Would test data be available or would your team need to generate own test data? Where would the test data be located?
8. Types of testing required	What kind of unit tests would be performed on your test automation script? What kind of integration tests would be performed on the integrated automation scripts? What kind of tests would be performed to check the validity of test data? Do you need to create automation only for functional tests? Or for other tests as well e.g. performance tests?
9. Supporting automation	Would your team need to create automation for non-testing activities (e.g. creating a lot of dummy data within the application or emailing test logs to users)?
10. Version control and test automation builds	How slow or fast is the version control? How long would it take to integrate the test automation created by each team member? How frequently would the test automation build be created?
11. Reviews	What test automation items would be reviewed e.g. test scripts, test data? What will be the frequency of these reviews? How long would it take to complete the reviews and re-work?
12. Maintenance	How frequently would the base lined test cases be updated? How long would it take to update your test automation suite in line with these changes? How frequently would your team receive an updated application build? (Keeping your framework and your chosen tool's capabilities in mind), how long would it take to update your existing test automation?

What are the responsibilities of a Software Test Lead?

2010-03-31T15:44:00.001+05:30

This post is the second one in continuation to the earlier one, What are the responsibilities of a Software Test Engineer?

Common responsibilities of a Software Test Lead include the ones given below.

1. Be updated on the latest testing techniques, strategies, testing tools/ test frameworks and so on

2. Be aware of the current and upcoming projects in the organization

3. Review and analyze the project requirements

4. Plan and organize the knowledge transfer to the Software Test Engineers and self

5. Collect the queries related to the requirements and get them resolved by the business person (e.g. the client, business analyst, product manager or project manager) assigned to the project

6. Plan, organize and lead the testing kick-off meeting

7. Scope the required tests

8. Design the required test strategy in line with the scope and organization standards

9. Create the software test plan, get it reviewed and approved/ signed-off by the relevant stakeholders

10. Evaluate and identify the required test automation and test management tools

11. Estimate the test effort and team (size, skills, attitude and schedule)

12. Create the test schedule (tasks, dependencies and assigned team members)

13. Identify the training requirements of the Software Test Engineers

14. Identify any test metrics to be gathered

15. Communicate with the client or on site/ offshore team members, as required

16. Review the test cases and test data generated by the Software Test Engineers and get them to address the review comments

17. Track the new/ updated requirements in the project and modify testing artifacts accordingly

18. Determine, procure, control, maintain and optimize the test environment (hardware, software and network)

19. Get information on the latest releases/ builds from the development team/ the client

20. Create and maintain the required test automation framework(s)

21. Administer the project in the test management system

22. Administer the Application under test (e.g. add users for the tests), as required

23. Assign tasks to the Software Test Engineers based on the software test plan

24. Check the status of each assigned task daily and resolve any issues faced by the team members with their tasks

25. Ensure that each team member is optimally occupied with work (i.e. each Software Test Engineer should not be too overloaded or too idle)

26. Re-assign the testing tasks, as required

27. Track the assigned tasks with respect to the software test plan and the project schedule

28. Review the test automation created by the Software Test Engineers and get them to address the review comments

29. Own and maintain the test automation suite of the project

30. Schedule and execute the test automation on the project

31. Review defect reports and assign valid defects to the relevant developer/ development manager

32. Assign returned defect reports and assist the concerned Software Test Engineer, as required

33. Ensure the resolved defects are re-tested

34. Consolidate and report test results to the concerned stakeholders

35. Be approachable and available to the Software Test Engineers, as required by them

36. Update the software test plan, as required

37. Ensure that the test cases are updated by the Software Test Engineers, as required

38. Ensure that the test automation is updated based on the updated test cases

39. Gather the decided test metrics

40. Escalate and obtain resolution of the issues related to the test environment and team

41. Plan, organize and lead team meetings and ensure action is taken based on the team discussions

42. Plan and organize training for the Software Test Engineers

43. Review the status reports of the Software Test Engineers

44. Review the time logged by the Software Test Engineers for various activities

45. Report the status to the stakeholders (e.g. the client, project manager/ test manager and the management)

46. Keep the Software Test Engineers motivated

47. Improve the test process based on the suggestions by others and own judgment

48. Manage own energy level and time

As mentioned in the earlier post, these responsibilities may be tailored depending on the specific organization for which you are working. However, you should be aware of the responsibilities above so that you may perform well as a Test Lead. My best wishes are with you.