Schedule a Software Walkthrough Free Trial
Home / Product / Estimation
Product — Reliability Growth Module

Estimation

The Estimation module trends actual test data against classic software reliability growth models to estimate current MTBF and forecast into the future.

When To Use This Module

System-level testing only

The Estimation module is used during software system-level testing. It should not be used in developer unit-level testing, since the failure rate is typically increasing during unit-level testing.

Setup Page Inputs

Anchoring the fault rate to a start date

The estimation models require the fault rate to be computed with respect to software and system-level testing. The first day of testing and the current date are required inputs.

Input
Description
First day of testing
The first day of an operational software system test. You’re prompted once for this date; once entered, it can’t be changed — it’s the starting point for computing fault rate.
Most recent day of testing
Update this field every time you enter new defect and usage-time data, so results generate accurately. After each day of testing, adjust this field and enter faults and test hours for that day on the Data Input Page — or enter on a weekly basis instead.
Data Input Page

Entering defect & usage-time data

Input
Description
Day
Automatically generated from your setup inputs.
f
Non-cumulative number of software faults observed that day.
x (hrs)
Non-cumulative number of usage hours experienced that day.

For each day of testing starting with the first day (per your setup), enter how many software faults were observed and how many usage hours were experienced. These instructions must be followed for the estimation results to be normalized. If testing occurred on a day but no faults were observed, enter 0 for f and the usage hours in x(hrs).

Collecting data

You’ll need regular access to the software problem reports recorded during testing. Filter reports before entering any data:

  1. Include only problem reports for a particular product/release — don’t merge reports from multiple products or field releases.
  2. Do not include new feature requests, since these aren’t failures.
  3. Do not include hardware failures.
  4. Include only problem reports serious enough to warrant a corrective action.
  5. If one problem report covers multiple software defects, count each defect individually.
  6. Enter the number of software faults (per the above filtering) per day in the “f” column.

You’ll also need to track testing performed per time interval, across all individuals — calendar time is not a normalized measure, so don’t use it as the unit of time.

  1. Determine how many people are testing each day.
  2. Determine how many hours per day each person is testing.
  3. Determine how many hours of automated testing occur each day.
  4. Total test hours per day is the cumulative hours across each person and automated test suite.

Enter the total usage hours per day in the x(hrs) column.

Import data You can import data from a Comma Separated (CSV) file instead of entering it manually, using the “Data Input Import” template — copy and rename the template file. Large imports can take a while; minimize import time by importing only rows that haven’t been imported before.
Fault Rate Page

Reading the fault rate status

The status indicates whether there are enough data points to compute fault rate, and if so, whether it’s increasing or decreasing. Too few data points disables the remaining Estimation tabs. If the fault rate is increasing, most models are unable to generate a result.

Output
Description
n
Actual cumulative number of defects found so far in testing.
t
Actual cumulative number of test hours so far in testing.

Pressing “Defect and time primitives for all data points” displays the cumulative data below, used by every reliability model to predict MTBF and failure rate.

Primitive
Description
Day
Each day of testing, per your setup inputs.
t (hrs)
Cumulative usage hours through this day.
n
Cumulative number of software faults through this day.
n/t
Cumulative faults divided by cumulative usage hours — the fault rate used by every reliability model, and used to determine model applicability. Increasing ⇒ no models are usable. Decreasing linearly ⇒ Exponential models trend best. Decreasing non-linearly ⇒ Logarithmic models may trend best.

Exponential models group

Output
Description
N₀ — Estimated inherent defects
If fault rate is decreasing, this is the Y-intercept of the fault rate graph — an estimate of total defects in the software at this point in time. Refines as you add data. Used by the Exponential models.
λ₀ — Estimated initial fault rate
If fault rate is decreasing, this is the X-intercept of the fault rate graph — an estimate of the first fault rate. Refines as you add data. Used by the Exponential models.
Estimated K value
A parameter used by the Exponential model: K = −estimated inherent defects / estimated initial fault rate.

Beneath this grouping is a graph of fault rate (n/t, x-axis) versus cumulative faults (n, y-axis) for every day of testing. A negative slope means fault rate is decreasing; positive means increasing. The Y-intercept (inherent defects), X-intercept (initial failure rate), and slope (−1/K) feed the Exponential models. Press “Exponential model parameters for all data points” to see the estimated inherent defects, initial failure rate, and K for each day of testing.

Logarithmic model group

Output
Description
Slope
The slope of ln(n/t) versus n, as graphed.
θ — Theta
The inverse of the slope of ln(n/t) versus n.

Beneath this grouping is a graph of n (y-axis) versus ln(n/t) (x-axis) for every day of testing; Theta = 1/slope. Press “Logarithmic model parameters for all data points” to see the estimated theta and slope for each day of testing.

Model Results

Six estimated failure rates per day of testing

This page shows six results for each day of testing, starting with the most recent.

Time Based Exponential Model
Estimated failure rate = N₀ k e−kt
Defect Based Exponential Model
Estimated failure rate = k (N₀ − n)
Logarithmic Time Based Model
Estimated failure rate = λ₀ / ((λ₀ θ t) + 1)
Logarithmic Defect Based Model
Estimated failure rate = (1/θ) × (ln(λ₀ θ t) + 1)
Cumulative
Estimated failure rate = t / n
Actual Time to Next Failure
= fi / xi
Where i is an interval of testing, f is the number of failures in that period, and x is the test hours in that period.

The tabular display shows estimates for the most recent test day, plus prior days; the graphical button plots the table.

Relative Accuracy

Which model is trending closest

Shows the relative accuracy of each model, averaged over all data points — the model with the smallest relative error is trending closest to actual time between failures. A second table shows relative accuracy for the most recent N data points, where you choose N — entering 1 shows the accuracy of the very last estimate; entering 5 shows the last five. Since the fault rate trend can change direction during testing, it’s useful to know which model tracks best against the most recent data.

Confidence Bounds

Upper & lower bounds per model

Illustrates upper and lower bounds for each model at a selectable confidence value. 95% confidence gives the widest bound range; 50% gives the narrowest. A confidence interval is a range x ± bounds around the mean: for any population mean outside that range, the probability of a sample mean further from it than x is less than alpha (e.g. alpha = .05 ⇒ 95% confidence). The bound is computed from the selected alpha, the standard deviation of the N₀ estimates, and the total number of estimates.

Forecast

Projecting MTBF forward

One of the most common reasons to use software reliability growth models: forecasting what MTBF will be after Δt more hours of testing.

Time or Defect Based Exponential Models
Forecasted failure rate = λp e−kΔt
Logarithmic Time or Defect Based Models
Forecasted failure rate = (θ × Δt) + MTBFp
Δt = additional test hours planned. MTTFp = current calculated MTBF using the model. λ₀ = current calculated failure rate using the model.
Residual

How much more testing is needed

The other common reason to use these models: forecasting how many more test hours or defects are needed to reach a specific objective, using the exponential models.

Remaining test hours
Δt = (N₀ / λ₀) × ln(λ₀ / λf)
Δt = test hours required to meet the objective. N₀ = estimated inherent defects. λ₀ = initial failure rate (the actual first observed rate from day one). λp = the objective, or desired, failure rate.
Remaining defects
Δn = N₀ / (λ₀ × (λp − λf))
Δn = remaining defects, assuming an Exponential model. λ₀ = initial failure rate (first data point, day one). λf = the objective (desired) failure rate. λp = the present failure rate.

Compare against your predictions.

Trended estimation data pairs with the Dashboard’s predicted-versus-actual comparison.

Go to the Dashboard ← Back to Home