There are variuos names for the same thing. Some call it software inspection, which also could extend to the design and its documentation,
some call it code inspection which relates more to the source code. A third name would be Fagan Inspection, called after the
person who invented this quality assurance and testing method.
Code inspections are a highly efficient test method which can not be substituted by any other test methods. It is time consuming
but according to statistics it will find up to 90% of the contained errors, if done properly. However it all depends on the
methods and checks applied and on the diligence of the inspectors. It must not be confused with the so called "code review" or
"walk through" which is usually done in a single meeting lasting for a couple of hours. A proper code inspection may take
several days and needs the help of tools to browse the symbols in order to find the places where they are used. The code review
can be used in addition to e.g. to generated acceptance of a software package by the integrators, but it must not be a substitute for a proper inspection.
Proper inspections can be applied for almost all work products in the software life cycle. At the first glance they may look
very time consuming. But statistical evaluations have shown that over the whole life cycle of the software
development they even save resources and thus money and improve the quality of the product.
Source of the diagram: Michael Fagan
Most people are not aware that the manual static testing methods i.e. inspections, reviews and walk throughs are defined in the
"IEEE Standard for Software Reviews". This is the IEEE 1028-1997 standard. I want to give a short overview on the main
definitions in this standard, however I will not discuss the "Management Review" which is in the widest sense a check of a
project's performance and the related documents. I will also omit a discussion of the Audits described in the standard, which a more
relalted to having external checks on work products and processes. I will focus on the review techniques for technical work
products as they are typically used within a company. I also want to point out the problems involved with these methods and make an attempt to present some solutions for
A walk-through can have a twofold purpose. First of all is can be performed to evaluate a software product with the purpose of:
In summary you could say that this kind of walk-trough is a method which should be used throughout the design phase of a
software product to collect ideas and inputs from other team members which lead to an overall improvement of the product.
The second objective of a walk-through is to exchange techniques and perform training of the participants. It is a method
to raise all team members to the same level of knowledge regarding programming styles and details of the product. In a sense
it also generates agreement within the team about the object of the walk-through.
- Find anomalies
- Improve the software product
- Consider alternative implementations
- Evaluate the conformance to standards and specifications
The formal aspects of a walk-through have a low profile. There are only a few roles defined in the standard. There is a walk-through leader,
a recorder, the author of the work product and team members. The standard says that at least two members have to be assembled for
the walk-through and the roles can be shared among them. The walk-through has to be planned which means that the participants
have to be defined and the meeting has to be scheduled. Further the findings and outcomes of the meeting have to be recorded.
In total this is a nice and easy to use method for everyday technical work from which you can expect good benefit.
About the purpose of techincal reviews the IEEE standard says: "The purpose of a technical review is to evaluate a software
product by a team of qualified personnel to determine its suitability for its intended use and identify discrepancies from
specifications and standards." In other words the technical review is a meeting in which a team analyzes a work product to
see it its quality is as expected or if it needs some improvement. The standard further states that not necessarily all
aspects of the review object have to be examined and that it is a possible purpose of the meeting to come up with alternatives
for a better design. The list of work products for which the review can be applied is quite big: Software requirements specification ,
Software design description, Software test documentation, Software user documentation, Maintenance manuals,
System build procedures, Installation procedures and Release notes are possible candidates for the review. The review meetings should
be planned in the project plan or they can be held on request e.g. by the quality group.
The roles involved in a technical review are as follows:
- Decision maker
- Review leader
- Technical staff
- Management staff (optional)
- Other team members (optional)
- Customer or user representative (optional)
According to the IEEE standard the input to the technical review shall include the following:
- A statement of objectives for the technical review (mandatory)
- The software product being examined (mandatory)
- Software project management plan (mandatory)
- Current anomalies or issues list for the software product (mandatory)
- Documented review procedures (mandatory)
- Relevant review reports (should)
- Any regulations, standards, guidelines, plans, and procedures against which the software product is
to be examined (should)
- Anomaly categories (See IEEE Std 1044-1993 [B7]) (should)
Instead of describing the technical review procedure in words I put this into a diagram. In the following flow chart
you find the steps of the review, the expected inputs i.e. work product and other documents, and a rough description of each
process step in the comment boxes:
The inspection as decribed in the IEEE standard is basically the same as the Fagan Inspection, as invented and
described by Michael Fagan in 1976. Compared to the technical review there is more formalism and
more roles, etc. involved. Further down you will find a comparison and discussion of these review techniques so
I will not go into details here. The only thing which needs to be pointed out at this place, is that the
IEEE standard states that the inspection should be done according to the project plan. It is usually not just
done on demand. Further there is an explicit reference to the software verifcation and validation plan where the
inspections should be reflected. An inspection is therefore regarded as a proper testing activity rather than
an activity to evaluate a work product for suitability.
The IEEE standard says that the following roles shall be established for the inspection:
- Inspection leader
According to the IEEE standard the input to the inspection shall include the following:
- A statement of objectives for the inspection (mandatory)
- The software product to be inspected (mandatory)
- Documented inspection procedure (mandatory)
- Inspection reporting forms (mandatory)
- Current anomalies or issues list (mandatory)
- Inspection checklists (should)
- Any regulations, standards, guidelines, plans, and procedures against which the software product is
to be inspected (should)
- Hardware product specifications (should)
- Hardware performance data (should)
- Anomaly categories (see IEEE Std 1044-1993 [B7]) (should)
Again here I have a description of the inspection procedure in a diagram. In the following flow chart
you find the steps of the inspection, the expected inputs i.e. work product and other documents, and a rough description of each
process step in the comment boxes:
Comparison of Inspections and Reviews
As you can see from the diagrams a technical review in not the same as an inspection. There are big differences and I want to
summarize them in the following table:
||Determine the suitablity of a work product for it's intended use.
||Determine the suitablity of a work product for it's intended use, but beyond that search for anomalies by examination through educated inspectors.|
||A minimum of two persons is required. Each one of them can assume multiple roles. Since the scope is different
additional persons e.g. management or customer representative can participate, but this is not regarded as a role in the process.
||Additionally required roles are the Author and the Reader. The roles are explicitly separated and can not be assumed by one person.|
||The inputs are very similar than the ones of the inspection. It has to be observed that checklists are not mentioned i.e. not required.
||Additional inputs are: Inspection reporting forms, Inspection checklists, Hardware product specifications, Hardware performance data. Some
of these inputs are optional. The reporting form is mandatory as well as the inspection checklist. The wording of the
standard would suggest that checklists are not mandatory, however a table in the appendix of the standard makes them mandatory.|
||The only output is an Action Item List and a record (meeting minutes) of the technical review.
||The outputs are a formal inspection report, a formal defect summary and a defect list with classified defects. The emphasis is
on providing a standard output of found errors which also would allow statistical evaluations.|
|Entry Criteria for the Meeing
||No special criteria mentioned.
||It is explicitly stated that the meeting has to be re-scheduled if the inspection leader finds that the participants are
not well prepared.|
||Keep the described rules of a review meeting.
||The defined roles have to be explicitly kept. The reader and not the author will present the material. The other roles
also have to be formally kept.|
|Outcome of the Meeting
||Generate of the defined review report.
||At the end of the inspection the decision has to be made to accept the work product, close the inspection, but require a rework,
reject the work product and require a re-inspection after the rework is done.|
There are some additional things to be observed. According to the standard the inspection leader has to have a formal training
about the inspection process. Further from practical experience it can be determined that an inspection or review meeting
should not be longer than 2 hours. After that time the quality of the inspection or review decreases. Another rule is that in
an inspection you should cover a maximum of 100 lines of code per hour and a maximum of 400 lines of code per day.
Practical Problems with Inspections
An inspection requires a big effort and lots of resources. The number of participants in an inspection is usually around 5 to 6
persons. However in modern industry it is very difficult to get 6 people who fulfill the "requirements" to attend in numerous
inspection meetings. We do not live in 1976 any more. Time pressure is high and development teams are global. If you think of
a small piece of software, let's say 2000 lines of code and if you try to keep the inspection rules this would mean that you
have to schedule approximately 10 inspection meetings to do a complete inspection of the code. In case the leader finds that
preparation was not well done he has to re-schedule the meetings. In a modern company with high time pressure in development and
a global setup of the teams this seems to be almost impossible to accomplish, especially if you consider that a medium microcontroller
application consists of 20000 and more code lines!
The formalism in the inspection process is heavy. However, the formalism does not guarantee that bugs are detected! Formalism
in a way guarantees that the inspection meetings are performed in the intended way and by this it may be implied that the
quality of the inspection has a certain standard, but that's all. You guarantee that a number of meetings were held according
to the rules. But there is no real benefit in it and no guarantee that the real task i.e. TO FIND ANOMALIES is really accomplished
to an extend that justifies the heavy effort.
The preparation period as described in the standard and as performed in real life is very vague! The checklists may not cover
everything that is necessary. Usually they are too general. A typical question in such a checklist is: "Are overflows/underflows
in calculations considered?". Then it is left up to the inspector to find a way to check this. Some experienced inspectors may
have a standard method for themselves and they produce good results, but there is no clear method about the exact procedure that
has to be followed in the preparation period or in the inspection meeting. This is left completely open.
However, this activity IS THE BUG FINDING ACTIVITY! Especially during the preparation time the bugs are found. In the meeting
you may have the effect that things become more clear when they are presented in the words of the reader and additional problems are
found, but the main bug finding activity is the preparation time.
The "Specified Inspection" - a better Inspection Process
The following suggestion for a different inspection process was not developed on a theoretical basis. It is based on real life
observations and after an adaptation of the inspection process also a very successful usage for real projects. Some years ago I was part of a team performing inspections on
source code. As the years went on I had the increasing feeling during these meetings that what was done was not good enough. It left the impression on me to be
superficial. On top of that we increasingly had problems to run the inspections according to the process.
The inspections we did
have not been 100% according to the process anyway. Some of the roles were assigned to one person and the follow up process
was never part of the inspection but we always used our established standard problem resolution process. However, the inspections
still worked o.k. But as time went on we did not manage to set up the meetings any more. They were scheduled, but since the
team size decreased we ended up in having only half of the expected participants attending, because the other team members were
caught up in meetings with the customer, or other important activities. On the non-working level we realized that the times we
formerly had for a coffee and a chat had also decreased and it was not only in our department. It was a company wide and even
industry wide phenomenon. After suffering from a slow change of worklife we suddely realized that we were not living in 1976
any more. Therefore a change of the inspection process had to be considered as well.
The old process just did not match modern work life any more!
After one of the inspections it was
more out of curiosity that I asked a colleague from a different department to have another look on the source code to search for bugs. This colleague
had never done any reviews or inspections. So I had to give him more details on each question of the checklist,
explaining how to examine the code. The bad thing was, that this colleague found some bugs, really critical bugs which slipped
our previous inspection. This was a clear sign that we had to do something about our inspection process. It was impossible to
turn the clock back and get more manpower and a more relaxed planning, so we had to go a different way.
Basically this was the birth of the new Specified Inspection method. It is based on the inspection
we knew so far, but reduces formalism to a level that can be lived. Further from the experience of success I had with
a different kind of "checklist" I emphasized the "preparation period" to make it a
specified activity with clearly expected work steps, thus turning it into a real bug finding phase. I want to go into the details of this process now
and things can be best explained with this picture followed by an explanation of the details and differences:
Manpower became very expensive and teams are much smaller than they have been 30 years ago. Therefore one of the first optimizations
was to focus on the reduction of manpower in the inspection process. On the one hand a reduction is possible due to a different
procedure which does not need an inspection meeting, on the other hand a reduction is achieved by using already existing roles as
they are usually present in a software development process. Thus embedding the inspection process into testing rather than
keeping it as a separate process. The following roles are needed:
- The autor. The author is responsible to collect the work product and all necessary supporting documents for the
inspection. This is e.g. the source code and the related design document. Further the author shall initiate the inspection
by a "hand over" document in which the work product and supporting documents are clearly identified and other information is given
to the inspectors as e.g. known anomalies, the scope of the inspection or things to observe. Further the author has the
responsibility to support the inspectors during their inspection and answer questions regarding the work product or the intended
functionality. The author also has to participate in the problem resolution meeting and participate in the definition of action
items as a result of the inspection.
- The inspector(s). An inspector is a specially educated person who's job is to perform inspections and testing.
The inspectors of the normal inspection as described in the IEEE 1028 are also required to be educated. However this is
more an education about the inspection process itself. The inspector of the Specified Inspection however needs a special
education concerning the error finding techniques, and if the inspection is related to source code, he also needs an
education about the programming language and it's traps and pitfalls. Further it is recommendable to have two inspectors
examining the same work product at the same time.
- The test manager. The test manager in a software development process is responsible for the test planning.
He also shall perform the planning for inspections. As it was pointed out in the presentation of the IEEE 1028 inspection
process an inspection is regarded to be a testing activity. Therefore there is no reason to have a separate planning
for inspections. It just can be planned together with the other testing. Further the test manager has the responsibility
to perform the moderation and if necessary arbitration in the inspection process. Whenever inspector and autor enter into
unresolvable discussions the test manager has to intervene.
- Other team members. The involvement of other team members is optional. They are not regarded to be a proper
role in the Specified Inspection process. Other team members may be involved
in the problem resolution meeting when it comes to the definition of action items. The author may not be able to decide
about schedules and activities so that other colleagues or the project manager needs to be involved.
The planning of inspections has to be part of the overall test planning. The strengths and weaknesses of inspections have to be
considered and the planning has to reflect the use of the right testing methods at the right time. Testing is typically done
on source code, also the original Fagan Inspection was focusing on source code. However inspections may extend also to design documents,
requirements or other work products. In this case the test planning is not the right place for the inspection planning. It
should be rather part of the overall project planning in case inspection is performed on other work products. However, the
overall principles and process of the Specified Inspection can be also applied on other work products, if appropriate
inspection specifications are used.
The inspection process itself is different to the so far know inspection process. It can be summarized in the following features:
- Interaction with Static Code Analysis by Tools. During the past few years powerful static code analysis tools were developed. These tools have to be
used, since they cover a large amount of checks which 20 or 30 years ago had to be done by inspections! An example of this
may be the order of evaluation problem in C. A code line like: x[i] = i++; is dangerous.
It is not clear if the indexing is done first or the increment. Some years ago it was subject of an inspection to check
the code for problems like this. However now there are tools which find these kind of problems easily. Inspection nowadays
should only supplement these tools in areas where an automatic check is not possible. These areas are mainly the remaining
statically detectable coding problems like underflow and overflow at calculations and a check of the conformance of the
work product with it's specification. Below is an example of possible items that remain to be checked by an inspection after
the check by a static code analysis tool is done.
- Specification. The inspection steps have to be clearly specified. The usual checklists are not enough. They only
say "what to do". A proper specification goes into details about "how to do" an inspection step. An example of an inspection
specification is given below.
- Inspection Conduct. The inspector uses the inspection specification and follows it step by step and applies it to the
work product. This will usually take the inspector several times through the code examining it each time for a different
purpose. The inspection is performed at the inspector's desk as part of his usual woking hours. The inspector shall observe
some rules regarding the inspection. He should not be faster than a maximum of 150 lines of code per hour and shall perform
a maximum of 4 hours inspection a day. The lines of code have to be netto lines of code, i.e. without comments and blank lines.
This is to maintain the quality of the inspection. These rules have to be considered in the planning of the inspection.
The inspector will record any anomalies during his inspection. Although the inspection has to follow a specification the
inspector is obliged to apply his expert knowledge and also report anomalies which may not be covered by the specification.
In case a found defect is not covered by the specification it shall lead to an improvement of the specification.
- Multiple Inspectors. Four eyes seem more than two. This is an old principle which also applies to software
inspection. It is recommended that you have always two inspectors working on the same work product. During the inspection
they can compare their results and come to a combined output. But at least at the end of the inspection they have to
put their results together into one report. This will inevitably lead to a discussion about found or not found anomalies
and this way the inspection will be of better quality. The preferences of a single person will be reflected against the opinion of
the second person, thus avoiding extremes and false alarms on the one hand and confirming real problems on the other hand.
- Reporting. The inspection results shall be recorded in a proper test report. In case there were multiple inspectors
their results have to be arbitrated first and a common conclusion of the two inspectors has to be documented in the test report.
The contents of the report have to be the same as for a standard test report according to IEEE 829. These are e.g. The name of the
tester(s), date and duration of the inspection, an identification of the inspection object, the objective of the inspection,
the used specification, a list of the found defects including a classification of the defects, the overall result of the test i.e. passed or
failed, etc. The report has to be build up in a way that a defect data collection is supported.
Problem Resolution Meeting
This meeting has to be held after the inspection is finished and it must not influence the reported inspection result. The purpose of
this meeting is to explain in details to the author and the other project members what defects were found, why they were
regarded to be defects and if possible a recommendation should be given how the defect can be resolved. It may also have an
educational purpose to have this meeting, because usually the inspector is confronted with quite a number of the same or similar
bugs during his inspections of various source codes. Thus he tends to become a center of knowledge and in these meetings a feedback
to the developers will help to spread this knowledge into the developing community. The aim of this meeting is to generate an
action item list for the project where the project members and the project leader will define which defects shall be resolved and
in which release the change shall be implemented. It is recommended that the test manager is called into the meeting for arbitration if
necessary. Further it is recommended that the Quality Engineer will participate in the meeting or at least compare and check
the inspection result with the action item list, to make sure that no major defects are dropped. In a company with a good development process the action item list will be
in form of a problem resolution and tracking database.
Rework and Follow Up
In the normal inspection process the follow up i.e. the tracking of defects found in the inspection is part of the inspection
process and even the role of the inspection leader is forced to do the follow up. In an environment of an established software
development process there has to be a problem resolution process anyway. Therefore the inspection process should
refrain from interfering with it. The inspection process should attach to such an established process. Generally speaking
the two process items i.e. performing inspection and performing controlled problem resolution are expected to be present in
processes of a certain maturity. The absence of one and the presence of the other is unlikely. If they are not present the same time
it is likely that the problem resolution process is present first. Therefore I do not include the follow up as an own step being
part of the inspection but just attach to the existing tracking process.
Example of items that have to be covered by an Inspection
A standard set of rules and procedures for code inspection has to be defined for the company or organzational unit. This set
has to be applied for every project.
The areas of the inspection have to be aligned to the use of a static analysis tool. What ever is checked by the tool can be omitted in the
inspection. This leaves enough time to focus on the rest of the possible problems.
Calculation and Numeric Checks
I.e. checks for possible overflows or underflows, check for possible divisions
by zero, check that arrays are only accessed inside its range limits, etc.
Control Flow Check
Check the use of pointer accesses for misalignments, null assignments and lifetime, check the absence or safe use of GOTO
statements, check for possible endless loops,
check the absence or safety of recursive function calls, etc.
Data Check and Data Flow Check
Check if the sequence of data usage is correct, check if data is defined and initialized at the correct place, check the use of
local variables as prefered to static or global variables, check if variable names match the data type (in case the data type
is encrypted in the name), etc.
Check if the coding is matching the design as described in the design document. This will focus on the required functionality.
Additionally check if the general design rules were adhered to, e.g. : check if the nesting of "if" is too deep,
check if there are too many or too less sub-functions, check the levels of the include structure and possible recursive
includes, check if the interfaces are according to the design rules, etc.
Check the compliance with naming conventions, check if the coding guidelines were adhered to e.g. regarding the use of design
pattern, check if the defined templates were used, etc.
Example of a Specification for an Inspection
Negative Example of a Checklist:
The following question used to be in the checklist for an inspection: "Have overflow and underflow been considered in calculations?".
Positive Example of a Specification:
The following check applies to all code lines where you have a multiplication (including bit shifts to the left), an addition or a subtraction.
For every variable in a calculation obtain the information about the maximum value which it possibly can take on. Then use the worst
case scenario (e.g. for the multiplication of two variables take their highest possible value) and check if the result is
fitting in the result variable without overflow. Consider also intermediate results which may be stored in registers during
an operation. E.g. if you multiply two values and divide the result of the multiplication, the final result may fit into the
destination variable without problem. However the intermediate result of the multiplication may overflow.
Do the same check for negative values (underflow) in case the data types are signed. Use the following variable ranges:
unsigned char value range: 0......................255
signed char value range: -128...................127
unsigned short int value range: 0....................65535
short value range: -32768...............32767
unsigned long int value range: 0...............4294967294
long int value range: -2147483648.....2147483647