How to gauge the quality of a product - quality metrices

When should we say the product is ready to be shipped? When can we really stop testing? How do we gauge the quality of a product? How do we measure the risk of a project? All in all, we need to have data for decision making. Then what data we need to have to support our decisions? What matrices exactly do we need?

Below are some useful matrices that we can refer to.

Accumulated Defect Find Rate against Accumulated Defect Closure Rate

The objective of this chart is to decide the stability and shippable of the project

• Two lines are deviated further indicates the project is unstable. You might need to invest more R&D resources on the defects. However, it's pretty common to have this pattern in the early phase of the project.
• Two lines are merging indicates the project is stabilizing. This represents a healthy project. This is often happen during the mid to late phase of the project. During this period, we can move some of the R&D resources on new project initiatives. However, if this happens too early, it indicates not enough QA definitely and QA resources need to be invested promptly to ensure the quality of the release.
• Two lines are merged indicates the project is shippable. Usually this would not happen automatically, but by intervention of project manager and leads to move the lower priority open bugs to next project.

While I worked in BigWorld, we used this method very often. I found this is very useful to keep all the leads on the same page. It will be easier to support our decisions which will involve more than a team, such as moving resources from one team to another. There won't have any emotional feelings on such decisions.

Defects Overview with Severity- a Pie chart

This is another popular pie chart for defects severity. Meaning, using a pie chart to show all the valid defects which has been filed in accordance to their specified severity might do the wonder. The severity of each defect can be classified as severity 1, severity 2 and severity 3 for example. Usual distribution of the severity might be 33%, 33%, 34% or about that. The high percentage in S1, for example as high as 70%, might indicate some underlying issues, such as testers are not testing enough simply.

Is this pie chart really useful? To me, it just gives you another piece of information. For example, what does that mean by 80%, 10%, 10% or 60%, 30%, 10%? Remember, testers might tweak the severity to get attentions on their defects. It may be early phase of the project that higher severity bugs are caught more and earlier. On the other hand, how about 10%, 30%, 60%? It may be from a very mature product that the product under-test has minor new features only in the release. Or, it may indicate you have a great development teams with quality in-mind and quality environment/systems (for example, prevent the developers from checking-in defective codes) in-placed?!

For whatever the reasons and outcome of this pie chart, it gives you a piece of information whereby you might have already know ahead. You know what kind of products you are testing, be it a prototype project, or minor features introduced in a mature product, be it you are in early phase or ending phase. Remember, you already schedule daily to scrub or triage the bugs so I am sure you have the high severity bugs (such as show stoppers or blockers) in mind to follow up.

Side note, I advocate "Just Enough Information" (or whatever the term) for project management to make decisions or trigger actions. For information or data, which will absolutely no chance to trigger actions or cause butterflying effects, to me is nice to have and for reading pleasures of everyone only.