Game Testing: How to test game software?

Games are available on a number of devices such as computers, mobile handsets, television sets and so on. In this post, I am going to focus on testing the software of a game that runs on a computer. See the video, Game Testing for Game Tester, or read on...

When you plan to test a software application, you tend to think about the types of testing required. Similarly, when you plan to test gaming software, you should identify the types of testing required. The common types of testing for game software are listed below. Keep in mind that the objectives of a game are quite different from the objectives of a business application. For example, the objectives of a business application could be to provide the given user the features to easily, reliably and accurately perform certain business tasks. In the case of a game, the main objective is having fun. There may be other objectives such as appealing to different types of player demographics, educating the players or developing team skills. Therefore, a game may be required to provide the following:
1. An intriguing or complex plot
2. Realistic/ delightful graphics (including characters, backgrounds and equipment) and sounds
3. Random events (to keep the player interested)
4. Little known facts (to educate the player)
5. Facilitate players to work as teams (In case of multi-player games)
Therefore, the following types of testing should be seen in the light of the given game's objectives.
1. Installation testing
Unless it is a (totally) web-based game, there would be some software that the player needs to install on their computer first. The installation process has a series of steps that should be tested on the common computer configurations (each configuration specifies data like the Processor, RAM, Hard Disk, Display, Display RAM, operating system etc.) used by players.
You should design test cases for the installation steps and then execute these test cases on each of the common configurations.
You should also check the completeness and accuracy of the content in the installation guide. This may be done just once (on the most common computer configuration).

2. Feature testing
This is the most important type of game testing. Typically, the game would have a number of features. You should aim to cover the game's features as exhaustively as feasible. An efficient way to accomplish this is to design test cases in a number of ways, such as:
a. Detailed test cases - These test cases are suitable to cover obvious features e.g. game options, progression from one level to the next, correct working of the controls, start/ suspend/ resume/ stop game etc.
b. Task based test cases - These test cases are at a higher level of detail. They take into account the objectives a player could have e.g. go through each level and win the game, choose and old saved game and continue it etc.
c. Test matrices - These test cases are useful when there are different features for different player statuses e.g. a particular weapon is available to the player only after clearing a particular level, the player can go through an obstacle only when they have collected a particular item before. A test matrix is drawn in the form of a table. The rows could then represent the features and the columns the player statuses (e.g. current level).
Executing the different types of test cases may still not be sufficient to discover a majority of the defects. You should perform exploratory testing. In simple words, this would mean play the game and perform interesting tasks that you think would reveal defects.

3. UI testing
Since games are typically based on user interfaces, this is also an important type of game testing. UI testing includes testing of the graphic elements (e.g. characters in the plot, backgrounds, objects in the foreground) as well as the content (both viewable and audible). An efficient way to test the UI is to first list the checks desired for every graphical element and content type. Then check each item in your list in every logical division of the game (e.g. each screen and each level).

There is an important thing that you should keep in mind regarding UI testing. It is possible that your game is released in multiple languages and cultures. If so, you should create a matrix (a table) listing all the supported languages. Then you should take the help of a speaker of a given supported language to check the content. You should get the content checked in each supported language at least once.

4. Performance testing
A novel story line, incredible graphics and sounds and numerous well-integrated features of a game may still fail to satisfy the player if the game becomes too slow or freezes up. You can check the speeds of operation of the game on common computer configurations. A good way to do this is to identify the common tasks a player is likely to perform. Then you should determine the acceptable times for these tasks. These times are the goals. You should then perform each task in the game and note the time it actually takes to do so. Since there are a large number of tasks to be timed, you should automate the performance tests.
Another flavor of performance testing comprises testing your game at extremes. For example, run a complete game without any pauses at all or run a game continuously for 24 hours or keep on increasing the number of players if your game is multi-player. Running such tests would give you data on how the performance of your game degrades on high loads. If the game shows performance problems at a realistic load, it means that you have discovered a problem that should be fixed.

You now know about the main types of tests that you can run on a game. Remember that feature tests and UI tests are especially important. Wouldn't you like to use your knowledge to test some game software?

Software Testing Books

Occasionally, people ask me about some good books on Software Testing. I refer them to this list. They say that though these books build up a subject from the beginning, everyone could learn something from them.

Knowledge Transfer: What is the best way to do it?

We know that a lot of knowledge is required by the tester before starting any test related tasks (even test planning). The knowledge is gathered and assimilated by the tester before the project starts or just after the project starts. The process by which this knowledge is gained is called Knowledge Transfer. Knowledge transfer is quite visible when a new team member joins the team and the existing team members (including the business people or the client) spend time in communicating with the new member to bring him or her up to speed. In an industry with a fondness for acronyms, knowledge transfer is also called KT.

Knowledge transfer is critical for testing. If your knowledge transfer has gone haywire, you could end up with people questioning your abilities (or worse, you could be out of the project). Remember that your successful performance depends on a good knowledge transfer to you. View my Domain Knowledge for Testers videos for industry domain knowledge in several industries. For example, view the video on Banking domain knowledge.

Sometimes the knowledge transfer happens in a chaotic, incomplete or hurried way due to any of the following problems:
a. Incorrect identification of knowledge givers
b. Lack of motivation on part of the knowledge givers
c. Communication challenges between you and the knowledge giver
d. Problems due to remoteness
e. Lack of trust (especially damaging to knowledge transfer)

The approach for knowledge transfer should take care of the above problems. Further, the approach should attempt to transfer knowledge that is located obviously in the documentation and processes and not so obviously in the team's behaviors and minds.

The knowledge transfer should have the following phases. Phases 3, 4 and 5 may be done in parallel. Otherwise, it is best to execute the phases in the given order. Even though the knowledge transfer may be initiated and closed by someone else (e.g. the client or the project manager), you are directly impacted by the success of the knowledge transfer. Therefore, you should take an active part in the process.

1. Prepare
The key people possessing the (most) knowledge in the various areas e.g. application functionality, application infrastructure etc. should be identified. It is not sufficient to inform the knowledge givers about the task. The knowledge givers should be motivated to impart the knowledge to you. This is usually best accomplished if the performance of the knowledge giver is somehow linked to the knowledge gained by you.

The one or more mechanisms of knowledge transfer should also be decided:
a. Demonstrations
b. Discussions
c. Providing documents as required by you
d. Replying to questions asked by you
e. Mentoring
f. Instant messaging
g. Wikis

2. Understand the client
It is important for you to understand the client organization (their business objectives, the purposes for which they use the application and so on). If the client cannot impart the knowledge to you, a business person (e.g. the business analyst) may impart the knowledge about the client.

3. Get familiar with the application
This is an important phase of the knowledge transfer since you would be spending a major part of her time with the application. A person very familiar with the application may demonstrate the functions and features of the application to you. Further, you should get a hold of a demo installation of the application in which you may explore the functions and features.
You should also get the (correct versions of) application documentation. Examples of application documentation include the system requirements, functional requirements, installation/ operating manuals, help files/ user guides, test cases, test plans and so on. This is the time to observe the application, read the documentation and ask as many questions are required to learn the nuances of the application.

4. Get familiar with the development, project management and test processes and systems
You should be familiar with the application development life cycle (e.g. agile, iterative or any other model), project management process (e.g. project planning, tracking, control, reporting and so on) and the test process (covering for example, build verification tests, new features tests, automated regression tests, defect life cycle and so on).
Further, you should understand (or learn and understand) the systems used for project management (e.g. Microsoft Project, TargetProcess) and any other systems used for test management (e.g. QC, Bugzilla).

5. Study the application environments
You should at least be familiar with the existing or proposed test environment (a.k.a. QA environment/ test bed/ test lab). It would be good for you to know about the other application environments for example, development environment, and staging environment and the production environment. The information about the application environments is usually available with the people playing the role of release/ deployment engineer or configuration manager or IT support (especially for production environments).

6. Provide feedback
You should provide feedback to the knowledge givers throughout the process of knowledge transfer. It is important to understand and internalize knowledge not only to respect the efforts of the knowledge giver but because this knowledge would be continuously used throughout the project.

Now that you are familiar with the various phases and aspects of the knowledge transfer, you should apply this knowledge in your next project. A good knowledge transfer would give you a definite advantage.

Well, you know how it is. You have been testing an application for some time now. One fine day, your manager walks over to you. He tells you that the management (or your client) is interested to get the functional tests automated. Since you are very familiar with the application, you should suggest the most suitable automated software testing tool for the purpose. You may know about your application’s technology and about a number of automated testing tools. However, you may not be clear about how to do justice to the evaluation exercise without burning time over this extra work. View the video, How to Select Automated Testing Tools. Then read on.

How to select the best automated software testing tool quickly

Evaluation of automated software testing tools can be a challenging and effort-prone task. Over the years, I have successfully analyzed, compared and evaluated a number of automated functional testing tools for applications written in various technologies. Below, I will describe the process for evaluation that I think is balanced in terms of effort and results.
1. Define your requirements

The primary requirement is that the automated testing tool should work (in other words, be compatible) with the application under test. There may be other requirements such as:
a. Ease of use
b. Available documentation
c. High quality customer support
d. Cost effectiveness (i.e. affordable price of license)

See the more comprehensive list of requirements towards the end of this post. Take your pick and/ or add requirements that are specific to your project, client or organization.

The requirements are rarely equally important. This means that some requirements are more important than others. After you choose your requirements, assign a weightage to each requirement using a scale, say 1 to 10 (1 being the worst and 10 being the best).

2. List the tools

There are many lists of automated functional testing tools on the web. One example is here. It is important to cast your net wide. Otherwise, you may miss a testing tool that beautifully suits your purposes. Later, someone (your manager, your colleague or your client) may ask why you did not consider another “obvious” tool.

3. Create a scorecard

Creating a scorecard need not be complex. Just list each of the requirements on the X-axis and the tools from your list on the Y-axis. See the sample. Add more columns and rows as required.









Look up the documentation on each tool's website. You could also browse the popular software testing/ QA forums e.g. www.sqaforums.com to get a feel of the kind of problems test automators generally face with a particular tool. Fill in the score (along with optional notes) in each cell. Use a consistent scale, say 1 to 5 (1 being the lowest and 5 being the highest), in each cell. You should have the raw data available at this time.

4. Analyze the scorecard

Next, you create the total scores for each tool. In order to get the total score, take the following steps for each row.
a. Multiply each score with the respective requirement weightage.
b. Add up all products in step a.

See the example scorecard with dummy scores. Note that the score for Tool1 is 4*8 + 5*5 + 2*10 = 77.







5. Try the short-listed tools

After you analyze your scorecard, you should see the tools that satisfy your requirements more than the others do. Short-list the top tools. You should select at least the top two tools (in order to give you some choice) e.g. Tool2 and Tool1 in the above table and at most the top three tools (in order to spare yourself too much effort).

It is the norm for tool vendors to offer trial or evaluation copies of the automated testing tools. Download and install the evaluation copy of each of the short-listed tools. Use each tool with your application. Create and run a few tests. Explore each of your requirements (given in your scorecard). Soon, you should have a good handle on:
a. the suitability of each short-listed tool with your application
b. your comfort level while using the tool

6. Present your results

You should create an automation demonstration with each of the short-listed tools. At least, you should create the demo for the top tool so far. Try to cover concepts related to as many requirements (from your scorecard) as you can. For example, if keyword driven testing were a requirement, it would be useful to have a keyword driven test within your demo.

With your scorecard data and your demo, you should be confident in presenting the results of your evaluation to any stakeholders.

Please fee free to use any information in this post for your purpose. Kindly let me know your thoughts on this evaluation process.

Possible requirements from the automated software test tool

• System requirements for tool installation
• System requirements for test creation
• Platforms supported for test creation (including platforms supported by using add-ins)
• Popularity (wide-spread use)
• Ease of use
• Object recognition
• Object management
• Integrated development environment
• Test debugging
• Integration with version control software
• Data driven testing
• Keyword driven testing
• System requirements for test execution
• Platforms supported for test execution
• Error recovery
• Reporting
• Documentation and training
• Customer support
• Load testing capability
• Test management capability
• Cost effectiveness (i.e. affordable price of tool and add-ins licenses)

Responsibilities and Roles of a Software Test Engineer

There are many talented and promising Software Test Engineers in the industry today. Their obvious responsibilities are testing software and reporting defects. However, different companies have different sets of additional responsibilities that are assigned to their respective Software Test Engineer roles. Therefore, I thought it would be a good idea to consolidate a list of the responsibilities of a Software Test Engineer. I hope that this list would help current Software Test Engineers to acquire any skills missing in their skill set. This list would also help any aspiring Software Test Engineers to get an understanding of the responsibilities of this role. View the video, Software Tester Job Role or read on.

Responsibilities of a Software Test Engineer

The responsibilities of a Software Test Engineer can be:

1. Go through the software requirements and get clarifications on one’s doubts (learn using my video on Requirement Analysis)

2. Become familiar with the software under test and any other software related to it

3. Understand the master test plan and/ or the project plan

4. Create or assist in creating own test plan

5. Generate test cases based on the requirements and other documents

6. Procure or create test data required for testing

7. Set up the required test beds (hardware, software and network)

8. Create or assist in creating assigned test automation

9. Test software releases by executing assigned tests (manual and/ or automated)

10. Report defects (usually in a defect database) to the stakeholders

11. Create test logs

12. Report test results to the stakeholders

13. Reply to returned bug reports (for example, when a bug report is returned as not reproducible)

14. Re-test resolved defects

15. Update test cases based on the discovered defects

16. Update test automation based on the updated test cases

17. Provide inputs to the team in order to improve the test process

18. Log own time in the project management software or time tracking system

19. Report work progress and any problems faced to the Test Lead or Project Manager as required

20. (If applicable) Support the team with testing tasks as required

21. Keep himself/ herself up-to-date on the overview of the development technology, the popular testing tools (e.g. automated testing tools and test management systems) and the overview of the business domain

As I mentioned before, one may expect the responsibilities tailored to the specific organization with which one is working. For example,

1. Small companies would have a less number of people and therefore would expect one to take care of most or all responsibilities.

2. Technology companies would focus more on the technical skills (e.g. design and development of the automation)

Overall, though the responsibilities of Software Test Engineer may remain static within an organization, it is only fair to suppose that the expectations from each responsibility would constantly increase.

Software Testing Jobs for freshers: How to get an entry level position?

In several forums, particularly Yahoo Answers, I have seen posts by users looking for entry level positions in the field of software testing. The problem is that most job posts for software testers seek candidates with not only specific skills but also substantial experience (in years) using those skills. This post should help the people freshly out of college and others who wish to begin their career in software testing. View the video on tips to get software testing jobs or read on.

First of all, a fresher or a beginner in software testing would do well to ensure that they have the basic knowledge that is going to be needed. In order to achieve this, one might use the following resources:

1. Courses
I have uploaded my entire manual testing tutorials set on YouTube. It is a set of 37 videos that covers all the basic software testing concepts in detail with multiple examples. These videos should be viewed in the order given. These are public videos so you can view them freely as many times as needed to understand them. See my Manual Testing Tutorials.
Also, there are numerous institutes that teach software testing. One may be able to find a good local institute. A good institute should offer placement assistance to its students and alumni.

2. Books

There are several good books that teach software testing. Here is one list of such books.

3. Tools

The developers of many popular automated testing tools e.g. QTP, Rational Functional Tester, and LoadRunner make evaluation copies of their tools available free of charge. You can learn LoadRunner using my free LoadRunner Tutorials.

Several popular testing tools e.g. Selenium, Bugzilla are open source. One may explore such tools to understand their features. Learn about Selenium using my free Selenium tutorials.

One should use all the relevant job web sites to search the entry level jobs. Some employers may be willing to relax the experience requirement if the candidate has a good knowledge of the testing work. If one is already employed, they may explore the possibility of an internal transfer to a software testing position.

There are a number of jobs posted on software testing forums. Examples of these forums are:

1. SQAForums

2. Software Testing Space

The additional advantage of joining such forums is that one is able to learn from and take advice from senior software testing professionals.

One should take others’ help in gaining the software testing position. In addition to asking for help from one’s friends and acquaintances, one should consider joining social networking sites like LinkedIn (which has an enormous number of professionals as its members). If one is a member of LinkedIn, one has the option available to join several groups which may have their own job discussions. One may also find work on tester communities like uTest.