!!*!! Software Quality Hub !!*!!: 2011

!! Smoke and Sanity Testing !!

There are two types of evidence which by nature are of very short duration and are usually designed so that major problems are found much earlier in the cycle. These two types of test are:

1. Testing for sanity
2. Smoke Testing

It seems that the smoke test of the word comes from the industrial test equipment. The smoke testing procedure is considered a safety test by a curtain of smoke into parts of the sewerage and drainage to detect sources of unwanted leaks and sources of sewer odors.

Proving for sanity is a surface test, but is performed whenever a cursory analysis is sufficient to prove the coating works consorting to specifications. This level of analysis is a subset of regress testing. Normally includes a set of fundamental tests of basic GUI functionality to demonstrate connectivity to the database, application servers, printers, etc.

However, in the field of trial software smoke test is a very important element. I often look for the two words have been used interchangeably. I will focus on the aspect of smoke test in this article.

Smoke Testing is applicable when adding new components and are integrated with existing code. Ensures that the building is not broken. The product in its current state is smoke tested daily. Merely ensures that the building is not broken and is ready to test further. Once an engineer certifies the smoke test is successful, the test team can take action to test deeper.

Typical features of Smoke Testing:

It exercises the entire system end to end.
It is not exhaustive, but should be capable of exposing major problems.
It ensures that the core functionality is the accumulation of work and is stable enough for further testing thoroughly.

Smoke Testing Advantages:

Reduction of Risk of Integration: From smoke testing is carried out integration problems are discovered at a much earlier stage than later in the cycle.
Find big problems: A good test designed smoke can increase the probability of finding a big problem when software is built early in the cycle. So catch bugs earlier in the cycle.
You can save time and money - If a major problem is detected at the stage when the software is ready built, it can save time and enormous costs if the mistake was discovered at the end of the cycle.

!! Smoke Testing !!

Introduction

Most projects have either schedule or effort overruns, or both, which leads to customer dissatisfaction. This is a widespread problem faced by most project managers. What usually happens is that issues that crop up during the integration testing, lead to delays in delivery of software.

We received a major project of 100 person years to be delivered in 9 months. Our goal was not only to meet the client’s expectations, but to exceed them. We started looking at various ways and means to meet this expectation. This included reviewing existing tools and techniques and also looking at best practices prevalent across the globe. The team decided that it would not be possible to achieve our goal, unless they did something radical.
We decided that one of the techniques the team would adopt was a weekly “Build and Smoke Test”.

Definitions:

Build: Build is defined as any of various versions of a software product that are being developed for release to users.

Smoke Test: Online Computing directory defines Smoke test as follows:

1. A rudimentary form of testing applied to electronic equipment following repair or reconfiguration, in which power is applied, and the tester checks for sparks, smoke, or other dramatic signs of fundamental failure.

2. By extension, the first run of a piece of software after construction or a critical change.

There is an interesting semi-parallel to this term among typographers and printers:
When new typefaces are being punch-cut by hand, a "smoke test" (hold the letter in candle smoke, then press it onto paper) is used to check out new dies.
Implementing for the first time is by no means an easy task. It requires a lot of discipline and changes to the way you do your work.

Frequency of Build

The principle behind weekly “Build and Smoke Testing” is to build the components and perform smoke testing every week, so as to reduce integration issues later. It could also be performed daily if several components are developed every day.

We tried implementing it on a daily basis but found that few components were getting developed on a daily frequency. We ended up building the same system two days in a row, and hence decided that the frequency of build could be weekly.

We first organised training sessions on weekly “Build and Smoke Testing”. The team felt that it was a good concept and it would definitely help them. They also suggested that it would be time consuming, if weekly testing was done manually and a “Build and Smoke” test tool should be developed.

Implementing Weekly Build

It was quite important that the “Build and Smoke Testing” activity be incorporated in the Project plan. This ensured that adequate resources are allocated for the build activity and we knew what components are undergoing weekly testing. The key is to know the progress of the project accurately.

The weekly build gave an enormous amount of visibility to the Project Manager as he could see the components being integrated, and issues being resolved each week.
Earlier the issues would be known only during integration testing. Some of the issues would be major and it would take up a lot of effort and consequently impact the schedule.

There was a Schedule drawn for weekly “Build and Smoke Testing” for each project, and a manager responsible for overseeing the test activity. It ensured that the complete activity was done in a clean environment (a build PC).

Broken Builds

It had to be ensured that the build was not broken. If the build breaks, then the following needs to be done:

• First check the build report for information, if that info is not sufficient, contact the manager to determine who is responsible. Contact that person and ask them to fix it.

• If the build does not compile or fails the smoke test, the manager should contact the developer responsible for that particular function and ask them to fix the code.

• Whenever the build breaks, it should be documented, and corrected in the version control system immediately. However, the weekly build that was broken should not be delayed or rebuilt. If a working weekly build is required for testing purposes, a second build may be created and posted with the weekly builds.

• All developers in the team should be instructed to check-in any code only after the entire module is tested to determine if their changes had an adverse effect.

Getting Customer to view the Product early

Adopting a weekly build process in the programme provided immense benefits to us. We could reduce time for integration testing. The integration issues were also reduced and the project team could be confident in delivering the product on time.

Usually in software development, “needs” and “wants” of a customer are different. Even if the system is designed with great care, using experienced analysts and the latest design techniques, it could fail to solve the customer's problem.

The product we were delivering may be as per the “wants”, but may not fulfill customer’s “needs”. The weekly “Build and Smoke Test” aims to address this issue effectively. The customer could see the product evolving mid-way and get a feeling of comfort. Any changes desired by customer could also be incorporated well in advance. The customer may view the product and let us know if any functionality had been missed out. Once the feedback from the customer is received, the software can be revised to meet their needs.

Conclusion:

We implemented weekly “Build and Smoke Testing” in addition to other tools and were able to deliver the projects with productivity improvement of 50%. The projects were also delivered with nil effort and schedule variance.