Guidelines for quantitative analysis

The Commission's Guidelines for Quantitative Analysis outline the principles we follow when undertaking quantitative analysis as part of our decision-making.

The principles apply to any quantitative analysis used in our decision-making.

However, the guidelines do not provide guidance or recommendations on whether quantitative analysis is required as part of the Commission's decision-making. These guidelines only apply to those circumstances where the Commission considers quantitative analysis is appropriate.

Guidelines for quantitative analysis

March 2012

Summary

We have identified five key principles:

  • Principle 1: the analysis has a clear objective.
  • Principle 2: the analysis uses appropriate approaches and techniques.
  • Principle 3: the analysis is repeatable and replicable.
  • Principle 4: the results of the analysis are robust and explainable.
  • Principle 5: all quantitative analysis is reviewed.

The principles set out in these guidelines apply to any quantitative analysis used in our decision-making. However, the actions we take to implement these principles may vary across projects.

 

Guidelines for Quantative Analys

Principle 1: The analysis has a clear objective

Principle objective

The Commission agrees and specifies the objective of the quantitative analysis upfront to help ensure the analysis provides value and that it achieves its objective efficiently.

What does this mean in practice?

  • We have agreed understanding of what the analysis is to achieve between the person(s) tasked with the analysis, the project manager and Commissioners (where relevant).
  • The audience for which the final outputs of the analysis are intended is defined.
  • A clear purpose statement is provided at the start of the analysis which states what the analysis is trying to achieve and how it achieves this.

Principle 2: The analysis uses appropriate approaches and techniques

Principle objective

The Commission uses appropriate approaches and techniques to help ensure the objective is achieved efficiently and that the analysis is fit for purpose.

What does this mean in practice?

  • We use approaches and techniques that are as simple as possible but sufficiently robust.
  • We ensure the approach and techniques used are proportional to the value added by the analysis.
  • We identify the limitations of the approaches and techniques used.

Principle 3: The analysis is repeatable and replicable

Principle objective

The Commission ensures that the analysis can be repeated at a later date (eg, to incorporate new data or assumptions, or to replicate the original results) even if person who provided the analysis originally no longer works at the Commission.

What does this mean in practice?

  • We design and structure the process, approach and outputs before doing the quantitative analysis.
  • We provide supporting documentation or include a commentary within the analysis that make it possible for the analysis to be updated or replicated at a later date by another person that is unfamiliar with the analysis. For example, we may include a specification which sets out the different steps of the analysis, the calculations included and the underlying assumptions.
  • The inputs and outputs of the analysis are clearly identified.
  • We ensure the process by which inputs are transformed into outputs is transparent.
  • We state any assumptions used in the analysis.
  • The analysis follows a clear structure.
  • We include the raw data, sources for the raw data, or links to the raw data, and clearly state any manipulations to this data.
  • We avoid the use of password protection where possible (and consider using alternative means for preserving the analysis or data).
  • We ensure the version of the analysis which produces the outputs used is clearly identifiable.  

Principle 4: The results of the analysis are robust and explainable

Principle objective

The Commission aims to ensure the results of the analysis stand up to internal and external scrutiny.

What does this mean in practice?

  • We check that the results of the analysis are intuitive and can be explained. For example, we check that the results are consistent with theory and previous work in this area (eg, academic studies).
  • We verify that the underlying data is from a reliable source and that, where appropriate, the data has been validated and cleaned.[1]
  • We test the analysis for robustness. This may include examining the sensitivity of the results to the key assumptions made.

Principle 5: All quantitative analysis is reviewed

Principle objective

The Commission reviews the quantitative analysis to identify its limitations and to ensure the analysis meets its stated objectives and is error-free.

What does this mean in practice?

  • We identify an appropriate stage to carry out a review.
  • The analysis is reviewed to an agreed and appropriate standard.
  • The review includes verification (ie, was the analysis done correctly?) and validation (ie, was the correct analysis done?).
  • We ensure that the review is done by a person with the appropriate skills. Ideally, the reviewer is someone who has not been involved in specifying or doing any of the analysis.
  • In some cases, the analysis will also require a more detailed or external review (for example, where the results have a substantial impact or are particularly sensitive). When providing analysis for external review, we ensure that any confidential or commercially sensitive data/information is protected against unauthorised external disclosure.
  • We provide the analysis to interested parties for consultation and review, where appropriate. When doing so, we ensure that any confidential or commercially sensitive data/information is protected.
  • We incorporate the review into the final analysis in an appropriate manner and where relevant.

 


[1]             This may include inspecting the data to identify missing and extreme observations, and correcting or removing erroneous data.