There is (another) silent peril undermining our decision making. Unlike bias, which by now, most of us recognise, it goes largely undetected. “Noise” has recently emerged as being equally disruptive to good decision making and is particularly important in professional judgments. Daniel Kahneman has made this the subject of his latest book, titled “Noise”, with co-authors Olivier Sibony and Cass Sunstein.
When you get a quote for insurance, you assume that the quote is based on a standardised calculation within any given insurance company. The authors research included an insurer who thought that too – the board realised there might be a small variation in the premium quoted by different individual underwriters – maybe up to 10% variation?? They were shocked to discover that the variation was as much as 55%. This was based on an exercise where a number of experts assessed the same criteria for an insurance quote – with very different results.
This is an example of System Noise; the unwanted variation in judgments within a system. It’s a much bigger problem than most of us are aware of, in part because it’s so difficult to measure.
The book breaks down system noise to a few component types:
1) Level Noise – one person’s average (level) is assessed at a different level than the average person’s average. A roundabout way of saying that people have different ‘base levels’ -eg one judge may set harsher sentences than the average judge or one valuer may set valuations lower than the average valuer.
2) Occasion Noise – the same person may make a different decision at a different time – this can be influenced by seemingly irrelevant factors like time of day, their mood, the weather or the order in which events occur.
3) Pattern Noise – certain situations cause decision makers to make a different assessment than in most of the cases they consider. This can be caused by personal experiences, for example a particular case unduly influences future assessments for that individual. The ‘availability heuristic’ mentioned in my last article can cause someone to alter their normal view (ie more recent events might have a greater impact on their judgments, causing ‘transient pattern noise’).
If you employ professionals to deal with complex issues and make judgments (let’s face it, that covers many activities within many organisations) you are probably suffering from noise.
Noise therefore causes a lack of consistency of outcomes, which can have a large impact. Another example quoted in the book is an investment firm, which gave identical P&L and cash flow data for investment prospects to its analysts. They found the valuations varied by 44% amongst its analysts, based on identical sets of data.
So if noise is such a big problem, how come we don’t know about it? Well, apart from the timing of the research – Noise was published in 2021, a decade after Thinking Fast and Slow, which opened our eyes to bias; noise is much harder to measure than bias. Bias can be quantified by measuring the deviation from an objective standard, whereas noise involves variability in judgments that may not be easily comparable.
One of the key findings of the research is that you are unlikely to recognise that noise is a problem in your system; the authors carried out a number of noise audits with their research subjects to identify the differences in judgments of individuals. This requires an exercise where multiple decision makers are given an identical set of information, the nature of which they might encounter in their regular work. They assess the information and make a decision including a quantitative output. For judges, this might be length of prison sentence, for Estate Agents, value of a property, for M&A advisors, the value of a company. These are then compared and the variability quantified.
The next obvious question is can we do something to reduce noise? The authors provide some suggestions to improve our “Decision hygiene”. I talk about that in another article that you can read here.
