December 26, 2009

The story of Rich Tester and Poor Tester

Even though we may not realize it at the time, the way in which we approach our work sometimes has a profound impact on our future. Let me explain this with the help of a story that I have made up. This story is about two testers who faced identical circumstances. The two even had very similar knowledge about the craft of testing. The only difference was that one of them earnestly wanted an opportunity to make her mark in testing.

Once upon a time, there were two testers working in a company. The company started a project in which two testers were required. Our two testers were assigned to the project with hopes (as always) of successful testing. However, only the Rich Tester (though she did not think of herself as such at the time) had been eagerly waiting for such a chance.

As usual, the initial period of the project was used for the purpose of knowledge transfer and knowledge building about the application under test. Both the testers were serious about gaining in-depth functional knowledge about the application. Both of them asked a lot of questions to the trainers and obtained clarifications to their queries. The Poor Tester stopped at that but the Rich Tester did not. The Rich Tester took the time to build her own notes about the problem areas of the application, the problems that she observed and the areas of the application that she would like to explore in detail.

Soon enough, the period of knowledge transfer finished and the day came when the two testers were provided a recent build of the application. Their Test Lead divided the test items between the two. As expected of her, the Poor Tester executed the test cases on her assigned test items and logged the defects in the issue tracking system. But the Rich Tester did not immediately start the execution of her test cases. She performed a sanity test on the application. She found a few high-level problems. Then she executed the test cases on her assigned test items. She logged both the high-level defects as well as the defects in her assigned test items. The result was that the Rich Tester was able to find a few more and a few more important defects.

As subsequent builds of the application came up, the Poor Tester stuck to her assigned test items. Since the available test cases listed only positive tests, she limited her test execution to only positive testing. However, things went differently for the Rich Tester. On a new build of the application, she would start with the usual sanity test. Then she would execute her assigned test cases but did not limit herself to the test data listed in her test cases. She designed other interesting test data and executed her test cases with that as well. She spent the remaining test execution time to design and execute negative tests in her assigned areas. There was now an observable difference in the outputs of the two testers. Generally, the Rich Tester was able to find either defects with a bigger impact or simply more defects than the Poor Tester.

There was no stopping the Rich Tester. She utilized the time between the builds to automate the generation of certain test data as well automate the sanity tests. Automated generation of test data saved a substantial period of time that she could now devote to an even more thorough round of testing. She started her automated tests every evening before she left for the day and analyzed the test results the next morning. Due to the quality of the defects raised by her, the Rich Tester had established a better perception in the minds of the stakeholders of the project. For the Poor Tester, things had remained fairly static.

Fast forward a couple of years. The Poor Tester was still testing (efficiently, mind you but by the book). However, the Rich Tester had impressed the stakeholders in her previous projects. She was considered a role-model for other testers and was now leading a team of her own. Soon, she would be in consideration for a role as Test Manager.

We can realize from this short story that it is in our best interest to not wait to be asked but to push the boundary as far as possible. We should find the means to make a bigger impact and magnify our output.

May 17, 2009

Domain Knowledge: 5+ ways to build your domain knowledge

Professionals from many backgrounds work in the software testing field. Some professionals have vast experience in the testing field. They are aware of many ways to test, they might have good people skills and reporting skills and they might be aware of many resources and tools that they could leverage in their tests. Some other professionals are ex-developers and they bring their technical acumen to testing. They usually have rich experience in at least one development environment, they are quick to conceptualize the implementation of the software they test and they can learn any new test automation tools very quickly. There is a third category of professionals who work in the software testing field. They come from the business background. They have been in the industry for a long time and they are keenly aware of the general requirements of the business and the end-users. Of course, these professionals from the business background might not know a great deal about testing methodologies or test automation tools but they know the business domain.

If you work in the software-testing field and you do not belong to the third category of professionals, you should know that building up your domain knowledge would put you ahead of many colleagues and fellow testers who do not choose to build that knowledge actively. View my video on How to Get Domain Knowledge or read on.

Let us look at the various ways in which you can increase your domain knowledge related to your project. Many software testers who have not worked in prior projects related to an industry e.g. investment banking, retail and so on learn about the project from the available documentation and the knowledge given to them by their project team members. The documentation may exist in the form of a requirements specifications, design specifications or user stories. The project team members may provide them knowledge in meetings or casual interactions. However, the initial learning from the team and continuous learning from the feedback given by the business are not the only ways to get the domain knowledge. You should consider the other ways to build up your domain knowledge:

1. Just get interested in the domain.
Even acknowledging to yourself that you need to know about the domain is helpful in itself. You would see the opportunities to educate yourself in the domain e.g. if you watch business news on television, you would find yourself paying attention to a program related to your domain. If you come to know about a local exhibition related to your domain, you could go visit there.

2. Read the business publications (articles, news items, web casts and so on).
By doing this, you would quickly become aware of the vocabulary and key concepts used in the business.

3. Get your hands on the documentation of similar projects done in the past and read it.
This would make you aware of the past issues and if it is a long-running project, you might become privy to information that is known to few people in your team or company.

4. If possible, work only in projects related to a particular industry.
This way, you would be able to leverage your existing domain knowledge.

5. Network and interact with professionals in the business.
Once you have some knowledge in your domain, you could build it further by networking. Professionals like to hear from each other and share information.

You can safely assume that either some software testing professionals would not consider actively building up their domain knowledge or even if they do so, they might not make a concerted effort in that direction. Therefore, if you make a regular effort to build up your domain knowledge further, you would get the following benefits:

1. The requirements of your project would be clearer to you.

2. You would be able to focus on the key aspects of the project easily.

3. Your reports (test results and defect reports) would contain the language of the business. You would strike rapport with the business stakeholders in this way.

Strengthen your domain knowledge and excel in testing.

March 22, 2009

Bug report: How to write an effective bug report?

View the Bug Reporting video or read on. Each one of us in the software testing field has seen many bug reports. Bug reports are written in a vast variety of styles. Sometimes, we happen to see the bug reports created by experts and inspire ourselves by doing so. However, a test engineer may create a bug report in any manner as seen fit. If so, can one be sure that the bug report is best suited to the circumstances? Will the bug report do the job well? In this article, I wish to make my reader aware of various aspects of a bug report. I have attempted to integrate many methods used by experts. This will enable you, my reader, to create bug reports that are best suited to the conditions at hand. The next time you have to write a bug report, you will have many tips to choose. Alternately, you may implement certain parts of some tips to create a custom strategy that suits the purpose.

How to write an effective bug report?

The first thing that we need to do is to understand the meaning of a bug report. Put simply, a bug report is a problem statement. It describes a problem observed in the system under test. Along with the problem, a number of supporting data elements is usually present. A bug report is used to communicate important problem data between testing, development, support and management groups. It is possible that the bug report created by the reader is read and acted upon in a different country and/ or in a different organization. All the currently open bug reports collectively contribute to indicate the present quality of the system.

Next, let us look at the various data elements that can exist in a bug report. Depending on the reporting requirements or your wishes, you can use the fields available in the defect tracking system to convey the bug to the intended audience clearly.

1. Id: Each bug should have a unique identifier. If a defect tracking system is being used, this Id is usually generated automatically. If not, you can assign a unique Id to the bug report. Since managers use bug Ids to track the bugs and the bug Ids are present in past reports, you should never alter the bug Ids.

2. Summary: The summary is a very important part of the bug report. Ideally, the summary should capture the core of the problem in a few words. A summary can contain other supporting information e.g. project/ product version, build, component, platform/ environment, team name, name of the bug reporter etc. You should consider the norms of writing bug reports in your project/ organization and summarize appropriately.

3. Project/ product: The bug reporter usually files the bug report against a single project or product in the system under test.

4. Component/ Module: If the bug belongs to a particular part of the system, you should name that part of the system. On the other hand, if it were not possible to attribute the bug to a particular part of the system, you may not name the Component/ Module or may term the bug as a “system-wide” bug.

5. URL: While testing web applications, it is common to state the URL that is required to retrace the steps and observe the problem.

6. Build: The Build number is the build (major number and/ or minor number) in which you discovered the problem.

7. Type of bug: The Type of bug can be a functional problem, a data problem, a performance problem, a problem with the build/ installer or a problem with the user interface.

8. Priority: Though you can set up the priority of the bug report based on the anticipated impact of the problem on the users, development or management may set the priority value differently, if required.

9. Severity: The Severity value is set by the bug reporter based on the problem itself e.g. it can be showstopper or blocker or a high severity bug or a medium severity bug or a low severity bug.

10. OS/ Platform/ Environment: If testing was done in different environments e.g. different operating systems (Windows XP, Vista, Linux etc.) or different browsers (IE, Firefox, Safari etc.), it is useful to specify the particular environment, in case the bug is observed in that environment only. For bugs that exist in all environments, you may mention the extent of the problem in the Description field.

11. Description: This is perhaps the most important part of the bug report. It contains the basic steps required to reproduce the problem. Sometimes, it is useful to mention both the expected results as well as the actual results encountered on execution of the steps. The actual results are the problem statement. In order to obtain support to get the bug fixed, it may be useful to state the impact of the actual results on the users. You may mention any data element that does not fit in the fields available in the defect tracking system in the Description part of the bug report under appropriate heading(s).

12. Attachments: A screen capture or a short video (consider the size of the attachment) showing the problem is usually quite effective in explaining the problem to the recipients of the bug report. This file should complement the Description of the bug report. If the file shows a number of items, encircling or otherwise highlighting the problem area in the file would save the recipients time to understand the problem. In addition to screenshots/ videos, you may attach other items e.g. error log, a report, other documents to build support for bug fixing.

13. Reporter: In many defect-tracking systems, this field is automatically populated. In case it is not, you should mention your name and contact information here.

14. Assigned To: You could keep any incomplete (or “in progress”) bug report assigned to yourself. A completed bug report should be assigned to the correct person in development (a development manager/ lead or a developer). A fixed bug report should be assigned to the bug reporter so that it may be re-tested and marked as closed. In case the assignee requires information or a decision from another stakeholder e.g. management or marketing, the assignee may assign the bug report to the concerned person in the relevant group. Note that at any particular time, the bug report should stay assigned to exactly one person.

15. Cc: Often, some persons are interested in the bug reports logged against the system under test. However, they may not play any direct role in fixing the bug. Such persons can include leads and management. These persons should stay informed about any changes to the bug report.

16. Dates: A single bug report can include a number of dates. Common dates included in the bug report include Date Created, Date Assigned, Date Resolved, Date Reopened, and Date Closed. In case the date fields are not automatically maintained by the defect tracking system, care should be taken to log the correct dates. A number of testing metrics e.g. bugs opened/ closed, average age of an open bug and so on may depend on these dates.

17. Blocks/ depends On: It is possible that a particular bug has a dependency on another bug. If the reported bug blocks another bug from being tested, this can be mentioned in the bug report. Similarly, if the closure of another bug is a pre-condition to observe the reported problem, you can mention the Id of that bug.

The bug reporter should carefully review the completed bug report. Usually, it is possible to improve the content of the bug report. You may use the following guidelines to review your bug report:

1. Do not be in a hurry to submit the bug report. You can come up with several ideas to improve the bug report if you give yourself a little time to think about the bug report.

2. Ensure that the bug report is within the scope of the current test. Another thing to ensure is that the bug report is against a completed feature/ function/ module of the system under test.

3. Ensure that the bug report is as compact as feasible.

4. The bug report should not directly or indirectly point to any person or any role in the project or organization. The bug report should only provide factual data about the observed problem.

5. Sometimes a few bugs are discovered in a single area of the system under test. It may be feasible to group some such bugs in a single bug report. This is another benefit of taking some between writing the bug reports and submitting them.

6. Ensure that the bug report is not a duplicate of any other bug report. It makes sense to search the defect tracking system using different searches to find any existing duplicates.

The self-review and revision is an iterative process. You should iterate this process until you are very satisfied with the bug report. Now, it is time to request a colleague or a senior to review the bug report and point out problems or improvement areas. You should then address their review comments in your bug report. Note that sometimes it makes sense to bunch a group of bug reports and ask for their collective review.

I hope that my reader is now conversant with the proper use of different fields of the bug report and the guidelines for effective reviews and revisions of bug reports. Happy bug reporting!

Learn more by watching the video, How to report bugs effectively?

March 17, 2009

Welcome

A very warm welcome to this blog!In order to get things going, please note that in the month of January 2009, I wrote an article about security testing. This article focuses on the SQL injection technique. Please provide your inputs for this article.