Here is a link to an interesting article about Code Inspections.
http://en.wikipedia.org/wiki/Fagan_inspection
Basically, the idea behind it is that you would take a random subset of specifications (usually about 5%) and have a team of 2-5 inspectors that were qualified to review the article and that would give a figure of the overall errors in the document.
Agile SQC Process
Entry Conditions
• A group of two, or more, suitable people* to carry out Agile SQC is assembled in a meeting.
• The people have sufficient time to complete an Agile SQC. Total Elapsed Time: 30 to 60 minutes.
• There is a trained SQC team leader at the meeting to manage the process.
Procedure
P1: Identify Checkers: Two people, maybe more, should be identified to carry out the checking.
P2: Select Rules: The group identifies about three rules to use for checking the specification. (My favorites are clarity (‘clear enough to test’), unambiguous (‘to the intended readership’) and completeness (‘compared to sources’). For requirements, I also use ‘no optional design’.)
P3: Choose Sample(s): The group then selects sample(s) of about one ‘logical’ page in length (300 non-commentary words). Choosing such a page at random can add credibility – so long as it is representative of the content that is subject to quality control. The group should decide whether all the checkers should use the same sample, or whether different samples are more appropriate.
P4: Instruct Checkers: The SQC team leader briefly instructs the checkers about the rules, the checking time, and how to document any defects, and then determine if they are major defects (majors).
P5: Check Sample: The checkers use between 10 and 30 minutes to check their sample against the selected rules. Each checker should ‘mark up’ their copy of the document as they check (underlining issues, and classifying them as ‘major’ or not). At the end of checking, each checker should count the number of ‘possible majors’ (spec defects, rule violations) they have found in their page.
P6: Report Results: The checkers each report to the group their number of ‘possible majors.’ Each checker determines their number of majors, and reports it.
P7: Analyze Results: The SQC team leader extrapolates from the findings the number of majors in a single page (about 6 times** the most majors found by a single person, or alternatively 3 times the unique majors found by a 2 to 4 person team). This gives the major-defect density estimate. If using more than one sample, you should average the densities found by the group in different pages. The SQC team leader then multiplies the ‘average major defects per page density’ by the ‘total number of pages’ to get the ‘total number of major defects in the specification’ (for dramatic effect!).
P8: Decide Action: If the number of majors per page found is a large one (ten majors or more), then there is little point in the group doing anything, except determining how they are going to get someone to write the specification ‘properly’, meaning to an acceptable exit level. There is no economic point in looking at the other pages to find ‘all the defects’, or correcting the majors already found. There are simply too many majors not found.
P9: Suggest Cause: The team then chooses any major defect and thinks for a minute why it happened. Then the team agrees a short sentence, or better still a few words, to capture their verdict.
Exit Conditions
Exit if less than 5 majors per page extrapolated total density, or if an action plan to ‘rewrite’ the specification has been agreed.
Notes:
* A suitable person is anyone, who can correctly interpret the rules and the concept of ‘major’.
** Concerning the factor of multiplying by ‘6 ‘: We have found by experience (Gilb and Graham 1993: reported by Bernard) that the total unique defects found by a team is approximately twice that of the number found by the person who finds the most defects in the team. We also find that inexperienced teams using Agile SQC seem to have about one third effectiveness in identifying the major defects that are actually there. So 2 x 3 = 6 is the factor we use (Or 3 x the number of unique majors found by the entire team).
Follow