The Ultimate Machine: How would you test it?

As many of you know, we now have our Software Testing Space group in LinkedIn. A lot of exciting discussions are taking place there. If you have not joined it, you should consider doing so.

My friend, Freddy Gustavsson from our STS group posted an interesting challenge on testing the Ultimate Machine. Well, the Ultimate Machine is attributed to Claude Shannon (1916 to 2001). It consists of a simple box with a single switch on its top surface and a closed lid. If you flip the switch on, the lid opens, a mechanical hand arises from it and the hand flips the switch off. Thereafter, the hand goes back into the box and the lid closes. Freddy's challenge was that if he asked me to test this Ultimate Machine, which tests would I perform? Here are the questions that I thought of. I would like to design tests based on these questions.

Positive tests

a. If the tester flips the switch on, does the hand (or an arm in certain implementations) switch it off?

b. If the tester does not touch the switch, does the machine just sit there and do nothing?

Negative tests

a. What happens when the tester flips the switch on and switches it off before the hand can reach it?

b. Is it possible to flip the switch somewhere between the on and off positions? What does the machine do in such a case?

Non functional tests

a. Can the tester operate the machine as it is or does it need to be installed? If installed, what does it take to install it?

b. Once the machine is switched on, how long does it take to switch itself off?

c. How many times can be machine be operated  before it stops working (due to battery discharge, wearing off of mechanical parts and so on)? Does it slow down after a while or show operational problems (make hitherto unknown noise for instance)?

d. Does the machine operate correctly if the ambient conditions are changed?

   i. The machine is vibrated e.g. during travel or a minor tremor.

   ii. The machine is operated outdoors (e.g. during a dust storm or in rain/ sleet/ snow).

   iii. The machine is operated in a strong magnetic field (e.g. certain places on a factory floor or in a hospital).

   iv. The machine is operated in different light conditions (e.g. direct sunlight, in the office daytime with blinds open/ blinds closed, at dusk or in total darkness).

   v. The machine is operated in different temperatures (e.g. at room temperature, in a freezer or near a furnace).

Usability tests

a. Is it easy for the tester to locate the switch?

b. Is the switch easy to flip (works smoothly, clicks to indicate it has flipped)?

c. Is the switch safe for the tester to operate (no rough surfaces or pointed end)?

d. Is the box pleasing to look at? How much space does it occupy on the tester's desk?

White box tests

We could open the box. Then, we can try to understand the mechanism inside and create more tests based on our understanding. For e.g. I would like to know if the time it takes to switch off is configurable and is the machine connected to any external system.

I am sure that you can think of more questions to help test the machine. Did you notice that I used the various types of testing to think about the above questions? What approach would you have used to design tests for the Ultimate Machine?

Do developers hate testing?

Having recently read a post on the UTest blog, I decided to think back about my developer days. What challenges did I remember from my experience as a Developer? Did I test my code or not?

1. Sometimes, it required a huge effort on my part just to know the correct set of requirements. Sometimes, the requirements would be given by a business person who would be focused on the promises already made out to the customers. The product manager/ business analyst would rarely verify the technical feasibility of implementing the requirements before committing to the customers. In such a situation, it became a matter of identifying the design alternatives available and exploring the promising ones in detail. As a developer, I used to take such situations as intellectual challenges. This resulted in me saying Yes, even when a simple No would save me a lot of work.

2. At other times, I was handed over source code that had been written, changed and enhanced (all these in different orders) by at least 3 or 4 developers before me. These developers were now either busy with other projects or no longer available in the company. I found this source code riddled with problems. There would be sections of the code that were incomplete, the sections would work with some input values but not with others, were partial duplicates of other sections or had logical problems. Getting such code to work was challenging. It involved plowing through the code (reading a section, unit testing it, resolving the problems, unit testing it again, refactoring it to make it more understandable, unit testing it and so on).

3. The other thing I recall is the sheer number of problems I had to face in order to implement a single requirement. Examples of these problems were:
a. The design was either non-existent or did not cover the particular requirement. This meant that my task doubled in scope. I first had to create a good design and then implement it.
b. The base components were not present or had defects. So, I had to decide either to re-write the base components from scratch or debug them before using them.
c. A similar requirement was already implemented in the application. The trouble was that I was either not able to understand that implementation or suspected hidden defects in that implementation.
d. The underlying development or run-time environment had issues (meaning defects or constraints) in certain conditions. This meant that I had to either somehow circumvent those limitations or break down my implementation into multiple parts in such a way that I did not run into these issues.

4. The next thing that I remember is how tight the schedules were. There were two reasons for this. The first reason was that the development effort was routinely under-estimated. Maybe my project manager (who estimated the development effort) wanted to increase my productivity constantly. Or maybe s/he estimated the effort considering the simplest development possible. The second reason was that I wanted my code to be perfect (in some cases, more than perfect). This meant that I would write the code implementing a requirement and run it repeatedly looking for problems. Once I had code running without problems, I would refactor it to make it leaner and strictly according to the coding standards (for example, naming the variables according to the specified nomenclature, adding comments every few lines and so on). If time permitted, I would even attempt to enhance the design or add extra functionality (though I stopped adding any extra functionality quite early on). My worries about how my code may not work under specific conditions led me to test it, modify it and again test it repeatedly. Given the tight schedule and my worries about the quality of my code meant that I was under stress.

It is not that as a Developer, I hated testing. Quite the reverse actually, I always wanted my code to be perfect and tested constantly. It was the sheer number of problems that I had to solve in a tight schedule that allowed some defects to fall through the cracks. And the defects discovered by the testers were only small fraction of all the defects that originally existed in the code.

Though I tested my code constantly looking for defects, there was only so much I could do given the tight schedule. In fact, I found that in internal projects (where the delivery deadlines were relaxed), I was able to submit code in which the testers could hardly find any defects.

Software testing involves a lot of work. The tight schedules do not help because it is very common for the development scope to increase during development. If the schedules are more realistic, I am confident that developers would test their code better and there would be a lot less defects in their applications.

Team Competition - How to compete against other software testing teams and win?

There may be occasions when you or your software testing team is pitted (compared from a business point of view) against another individual or another team. Examples include:

1. You are a senior tester and the management has just hired a junior person. Now, some tests have been assigned to the junior tester. The idea is to see if there is a discernible difference in the outputs of the two of you.

2. Your management or client has decided to off-shore (OR bring on-site) some of the testing work. Some work is now being done in parallel in both locations. The idea is to see which team provides more value with respect to the cost it incurs.

3. An experienced person with the business background has been added to the team. The idea is to see if business knowledge or software testing knowledge provides greater value in testing.

Before I go on, let me clarify that healthy competition is a fast way to grow your capabilities. Otherwise, things tend to stagnate. And quite often, the competing teams end up collaborating with and supporting each other the other side of the tunnel.

But, in the meantime, you have to think about at least maintaining your position. If you do NOT take any additional action, your role may be diminished or change into something else regardless of your preference. How can you maintain (and subsequently grow) your position?

1. Study the other team.
When sports team prepare for competitions, they actively study the other team. You should do the same. You should find out the following information about the other team:
 a. The number of members in the team
 b. Their names and location
 c. Their general background, experience, knowledge, achievements and struggles
 d. Their prior exposure to your company's specific applications, tools and processes
 e. Their working style/ culture
 f. Any natural advantages they have over you

2. Follow their work and deliveries.
You should actively follow their work and the deliveries they produce. Use all channels (official and informal) to keep you updated about their progress.

3. Analyze the other team's strengths and weaknesses.
Your team should now be able to have a fair idea of the other team's strengths and weaknesses. Be ready to update their image in your mind as you follow their progress regularly.

4. Identify your own strengths and weaknesses.
This is a good time to be more aware about your own team. What things is your team good at? What tasks is it not so good at? In what respects does your team absolutely struggle?

5. Change your approach appropriately.
If are keenly aware of the other team and your own team, it should be easy for you to identify the ways in which you can out-perform them. Some examples are given below but you should find out your own ways to out-perform the other team. And, implement the changes quickly.
 a. If the other team only executes the predefined test cases, your team may want to additionally invest time in enhancing and then executing the available test cases. Or perform exploratory testing in addition.
 b. If the other team focuses on functional bugs, your team may want additionally focus on performance bugs, usability bugs and look and feel bugs.
 c. Get work done faster. If their team members watch the clock (come in at time, work and leave on time), your team can shorten your breaks or concentrate more on the tasks or stay on the task longer.
 d. If the other team has individual performers, your team may want to harness the power of team work.
 e. If their team works in a silo, your team may have a good rapport with the other engineering teams.

6. Inform the decision makers yourself.

You can be sure that the decision makers who initiated this change would be actively following its effects. You should inform your own achievements in a positive light to them. Do NOT rely on someone else to do this for you. If so, they may tone down your achievements according to their own biases, opinions and agenda.

7. Finally, help your NEW team.
When the dust settles, your team will have a better assessment of their strengths and weaknesses. You will also be more mature. The other team will also share the same benefits. After all, everyone wants to grow.

If the decision makers take a professional decision, the work will be more aligned according to the strong areas of the either team. The teams may even be re-distributed or merged into a single unit. It is a WIN-WIN situation. Now is the time to help yourself and your new team to the best of your abilities.