Code Inspections… Do they work?
Here is a link to an interesting article about Code Inspections.
http://en.wikipedia.org/wiki/Fagan_inspection
Basically, the idea behind it is that you would take a random subset of specifications (usually about 5%) and have a team of 2-5 inspectors that were qualified to review the article and that would give a figure of the overall errors in the document.
Agile SQC Process
Entry Conditions
• A group of two, or more, suitable people* to carry out Agile SQC is assembled in a meeting.
• The people have sufficient time to complete an Agile SQC. Total Elapsed Time: 30 to 60 minutes.
• There is a trained SQC team leader at the meeting to manage the process.
Procedure
P1: Identify Checkers: Two people, maybe more, should be identified to carry out the checking.
P2: Select Rules: The group identifies about three rules to use for checking the specification. (My favorites are clarity (‘clear enough to test’), unambiguous (‘to the intended readership’) and completeness (‘compared to sources’). For requirements, I also use ‘no optional design’.)
P3: Choose Sample(s): The group then selects sample(s) of about one ‘logical’ page in length (300 non-commentary words). Choosing such a page at random can add credibility – so long as it is representative of the content that is subject to quality control. The group should decide whether all the checkers should use the same sample, or whether different samples are more appropriate.
P4: Instruct Checkers: The SQC team leader briefly instructs the checkers about the rules, the checking time, and how to document any defects, and then determine if they are major defects (majors).
P5: Check Sample: The checkers use between 10 and 30 minutes to check their sample against the selected rules. Each checker should ‘mark up’ their copy of the document as they check (underlining issues, and classifying them as ‘major’ or not). At the end of checking, each checker should count the number of ‘possible majors’ (spec defects, rule violations) they have found in their page.
P6: Report Results: The checkers each report to the group their number of ‘possible majors.’ Each checker determines their number of majors, and reports it.
P7: Analyze Results: The SQC team leader extrapolates from the findings the number of majors in a single page (about 6 times** the most majors found by a single person, or alternatively 3 times the unique majors found by a 2 to 4 person team). This gives the major-defect density estimate. If using more than one sample, you should average the densities found by the group in different pages. The SQC team leader then multiplies the ‘average major defects per page density’ by the ‘total number of pages’ to get the ‘total number of major defects in the specification’ (for dramatic effect!).
P8: Decide Action: If the number of majors per page found is a large one (ten majors or more), then there is little point in the group doing anything, except determining how they are going to get someone to write the specification ‘properly’, meaning to an acceptable exit level. There is no economic point in looking at the other pages to find ‘all the defects’, or correcting the majors already found. There are simply too many majors not found.
P9: Suggest Cause: The team then chooses any major defect and thinks for a minute why it happened. Then the team agrees a short sentence, or better still a few words, to capture their verdict.
Exit Conditions
Exit if less than 5 majors per page extrapolated total density, or if an action plan to ‘rewrite’ the specification has been agreed.
Notes:
* A suitable person is anyone, who can correctly interpret the rules and the concept of ‘major’.
** Concerning the factor of multiplying by ‘6 ‘: We have found by experience (Gilb and Graham 1993: reported by Bernard) that the total unique defects found by a team is approximately twice that of the number found by the person who finds the most defects in the team. We also find that inexperienced teams using Agile SQC seem to have about one third effectiveness in identifying the major defects that are actually there. So 2 x 3 = 6 is the factor we use (Or 3 x the number of unique majors found by the entire team).
1 comments
Reply
This site uses Akismet to reduce spam. Learn how your comment data is processed.
In my opinion, there should be as many ways as possible to test software. Complete coverage is everything. Keeping that firmly in mind, here are my thoughts on this particular QA methodology (Fagan-Agile).
As the article says, this method uses little man hours to start, but before this can be done, the persons MUST be trained and understand the method. There may be a steep learning curve involved for QA and the developers must also learn new methods. In all cases, I believe that QA should be involved in the project at the start of the design phase to promote a better product, not just at the end of the coding phase.
This method requires a non-changing “product spec, design spec, and marketing requirements document”. Because of the nature of business, as well as human nature, these specifications are rarely, if ever, static. There is always marketing that says “We need this feature”, that leads to Feature Creep (a product that will never go to market, since the software will never be completed), there will always be the product manager that will not take the time to have his development team learn new methods that will initially cost productivity and time that will affect his quarterly, there will always be the developer that will not or cannot learn (I knew a developer that would Always Cut and Paste. This is a great “Time Saver” to get loads of productivity done, but 6 out of 10 times would introduce an error that would cost my time, the build team’s time, his time, and the resources that we used to get another version of the software built and tested. Strangely enough, this person was always “Busy”.), there is always the developer that will say “All I did was a spelling change, so no need to test it” where the spelling change involved 3 new variables, 2 new functions and an extra file created, and last but not least, “We cannot afford taking the entire team out of work for 4 months training.”
This method also relies on Random Chance. To take a 100 page document and only examine 2 pages to start, then multiplying the number of defects found by 3 (or 6 depending on the size of the team) is pure chance to start, and the team has to be qualified to do the correct work.
Estimations and odds aside, this method of QA is clean if done properly. It forces the designers to follow the rules to come up with cleaner, clearer designs.
The benefits of this method are very significant. If it holds true, you can estimate how long an SQA cycle will last and set up an accurate schedule. There will be no need to allow more time for one department to finish the work, decreasing time needed for another department to make the scheduled ship date.
Software is an amorphous cloud of code that is constantly changing and expanding to more diverse applications. New methods are always coming up, but the tried and true method of testing software must be to exercise every path, every variable, and every result. By creating a test plan, one can accomplish this and it will work 100% of the time, or with a changing spec, will work until it does not, but all other avenues have been seen and tested.
Here is a quote that I once read regarding SQA that I would like to share.
“QA is the Last Bastion of Defense against buggy code.
We protect the consumer from the machinations of development, and declare war on software defects.”
Militaristic, but true.