Curves Are Better Than the Matrix

Cyber risk usually gets a colour, not a number: red, amber, green, maybe a score bolted on for good measure. Quantitative risk analysis swaps the colour for a range in pounds, and it produces a sharper, more useful picture than any matrix, especially once that range gets tested against the judgement of people who actually know the risk.

The Case for Measurement

Picture the workshop. Someone points at a spreadsheet and asks whether a finding is a 3 or a 4 on the impact scale. Someone else says it depends on the day. Twenty minutes later, the room has settled on a number that means something different to everyone in it, and that number is what ends up in the board pack.

That is the risk matrix in action: two questions, how likely and how severe, plotted onto a grid of low, medium and high. It is fast, it fits on one slide, and management loves it for exactly those reasons. It is also, according to research published in Risk Analysis, unreliable in ways that matter more than the slide suggests. Matrices can hand two very different risks the same rating, a flaw researchers call range compression, and they can even rate the smaller risk higher than the bigger one, because the boxes on the grid are themselves a judgement call.

Quantitative risk analysis skips the argument about boxes altogether. Douglas Hubbard and Richard Seiersen make the case at length in How to Measure Anything in Cybersecurity Risk: price the risk instead. An expected annual loss of £180,000 sits on the same scale as one of £40,000, the same way two price tags on a shelf do. Red and amber do not sit on any scale at all.

The Case Against Pure Numbers

None of that makes quantification easy, and it is worth admitting why. Ask five experienced security leads for the probability of a serious breach next year and you will likely get five different numbers, none of which anyone would want to put real money on. The events that matter most, a serious breach or a major outage, are rare enough that most organisations have barely any loss history to build a model from. A study on cyber-vulnerability prioritisation puts it plainly in this statistical framework paper: partial and incomplete threat data is a genuine obstacle to quantitative analysis, not a rounding error.

There is a sharper danger hiding in the numbers too: false precision. A figure with a pound sign in front of it looks more rigorous than a colour, whether or not the guess behind it has earned that confidence. Hand a board a bad quantitative model dressed up as a good one, and it can do more damage than a matrix ever could, purely because it is more convincing.

Where Expert Judgement Fits

The fix for a shaky number is not to give up and go back to the grid. It is to make the number less shaky.

This is where structured expert judgement earns its keep. Cooke’s classical model, described in Expert Judgement in Risk and Decision Analysis, first tests specialists against questions with known answers, the kind you could check with a calculator, and scores each one on how accurate and how informative their estimates turn out to be. Only then does it ask them the real question, and it weights their answer by how well they did on the practice round. Hubbard and Seiersen build a version of the same idea directly into cybersecurity, running practitioners through calibration training until their 90 percent confidence intervals actually contain the right answer nine times out of ten, not the wildly overconfident guess most people start with.

Put five calibrated security practitioners in a room instead of five uncalibrated ones, and the argument about whether a finding is a 3 or a 4 turns into something closer to a genuine estimate: a wide range, honestly stated, that gets narrower as real evidence comes in. That range is not perfect. It is, however, considerably better than a guess dressed up as a category, and it is the only version of the answer that ever gets checked against reality and improved.

That is the synthesis worth taking away. Expert opinion does not replace the number, and it should not be squeezed into a grid either. It sharpens the number, and the number is what should be sharpened. In practice, the evidence for both usually comes from the same place: a structured look at controls across people, process and technology, of the kind you get from a Cyber Maturity Assessment. What changes afterwards is what happens to that evidence, whether it lands on a grid or feeds a range that can be tested, compared and made better.

Sources

• What’s Wrong with Risk Matrices? — Risk Analysis, Wiley Online Library
• How to Measure Anything in Cybersecurity Risk — Wiley
• A robust statistical framework for cyber-vulnerability prioritisation under partial information in threat intelligence — arXiv
• Expert Judgement in Risk and Decision Analysis — Springer Nature Link