Measuring Software Quality

Norm Goodkin

Quality Matrix International, Inc.

1D, 2D, 3D

1D = Total Number of Bugs

But some bugs are more equal than others

2D = Bugs by level of Severity

But one area can suffer while another is great

3D = Levels of Severity + Area Affected

Severity is not the same as Urgency for Fix.

Severity 0-9

9 = worst case scenario

8 = pretty bad

7 = service outage

6 = old feature fails

5 = new feature fails

…

0 = spelling error in an internal operations message

Accumulation of Low Level Bugs

Lots of Level 3 and Level 4 bugs can give you a tummy ache.

Set limits at all levels, for all areas.

Take appropriate action if it looks like you are going to exceed the limits.

Fast Track or Crash to allow more fixing

Prepare users to deal with the problems

Setting the Bar

Set realistic goals

Base goals on past performance

Enjoy the success of achieving goals

Gold Plating is as wrong for Quality as it is for Scope.

Zero Defects/Zero Downtime?

Is ZD² a realistic requirement?

Goals vs. Objectives

Continuous Improvement – a form of Gold Plating?

Set Quality Targets based on project history

Achieve Acceptable Quality

Quality Review Board

Represent Stakeholders

Not too senior

Subject Matter Experts

10-15 max

Quantify Problem Severity

Establish Quality Targets

Quality Review Board?

The Quality Matrix

3D Quality Matrix – Sample

Apply Feedback

Weigh the Risk

Which bug fixes will be the least risky?

If two have equal severity, fix the one with less risk.

Generally, the larger the change, the larger the risk.

Fixing a low severity problem can be risky.

There’s no such thing as zero risk with software.

Summary

Measure Quality

Set reasonable quality targets

Represent Stakeholders

Balance Schedule, Scope, Cost and Quality

Use feedback to get it right


	1D = Total Number of Bugs
		But some bugs are more equal than others
	2D = Bugs by level of Severity
		But one area can suffer while another is great
	3D = Levels of Severity + Area Affected
		Severity is not the same as Urgency for Fix.


	9 = worst case scenario
	8 = pretty bad
	7 = service outage
	6 = old feature fails
	5 = new feature fails
	…
	0 = spelling error in an internal operations message


	Lots of Level 3 and Level 4 bugs can give you a tummy ache.
	Set limits at all levels, for all areas.
	Take appropriate action if it looks like you are going to exceed the limits.
		Fast Track or Crash to allow more fixing
		Prepare users to deal with the problems


	Set realistic goals
	Base goals on past performance
	Enjoy the success of achieving goals
	Gold Plating is as wrong for Quality as it is for Scope.


	Is ZD² a realistic requirement?
	Goals vs. Objectives
	Continuous Improvement – a form of Gold Plating?
	Set Quality Targets based on project history
	Achieve Acceptable Quality


	Represent Stakeholders
	Not too senior
	Subject Matter Experts
	10-15 max
	Quantify Problem Severity
	Establish Quality Targets


	Which bug fixes will be the least risky?
	If two have equal severity, fix the one with less risk.
	Generally, the larger the change, the larger the risk.
	Fixing a low severity problem can be risky.
	There’s no such thing as zero risk with software.


	Measure Quality
	Set reasonable quality targets
	Represent Stakeholders
	Balance Schedule, Scope, Cost and Quality
	Use feedback to get it right