LKNA13: How to measure anything – intro from Douglas Hubbard

What’s the most important project in the business? How to know what project is the most important project IS the most important project. (making decisions based on priorities)

Decision making techniques have been studied extensively, so we know some things about how to get this most important project right.

In my book how to measure anything I explain why the three reasons people claim something is immeasurable are actually illusions.

How do you evaluate decision making methods – how do you do meta-decision making?

If you let people use a structured, formal method for making decisions, they feel better about their decisions, even if they are worse.

If you use more info, or you collaborate with others beyond a certain amount, or you spend too much time looking at your investments, you do worse, not better. Confidence goes up, but info advantage levels off. (and poss complexity conceals more than reveals)

Experts vs Quantitative Methods (math’s wins)

Paul Meehl compared 150 historical statistical models with human expert intuition. There were barely 6 where humans were better. Math’s is better than intuition. Philip Tetlock tracked over 82k political forecasts from 284 experts and found no domain in which humans clearly outperformed crude extrapolation algorithms, less still sophisticated statistical one’s.

The Monte Carlo model method for estimating cost (link to uncertainty) has proved superior to human judgement.

Reasons people don’t use quantitative methods:

  • ‘we don’t have sufficient data’
  • ‘each situation is so unique and complex that we can’t rely on historical data’ (experience is only historical data with flawed recall)
  • There is too much error and bias in the data
  • There are so many factors, the measurement alone tells us nothing

THEREFORE:

‘we are better off relying on your experience instead’.

This is a mathematical claim. You are saying ‘the cost of using the data doesn’t justify the results we’d get from using it’. This can be calculated and proven wrong.

Our Ability as Estimators and Decision Makers

  • Overconfidence (people tend to think they’re more likely to be right than their past decisions suggest), inconsistency from moment to moment due to different emotions and attitudes to risk, decision fatigue etc – people say different things depending on arbitrary external factors – even experts! We’re bad at complexity (identifying the relevant info and ignoring irrelevant) and at probability (90% confidence is actually 65% chance of being right)

People can be trained to put odds on things well. Bookies can do it well, physicians badly. Most people do it badly, including Harvard MBA’s.

Experience is inevitable, learning is not.

Having skin in the game is a factor, but even imagining you have skin in the game helps.

Practice helps – lots of iterations, fast feedback helps.

Creating a computer that predicts the answers you give to problems – the comp is better than you. How? It isn’t effected by tiredness etc, it’s a reflection of you at your best, but you aren’t always at your best. It wins because it’s consistent.

Interpreting Limited Data & Probabilities 

A sample of 5 people out of 10,000 is accurate enough to work out roughly with 92.5% accuracy – limited data can be pretty useful.

Intuitions fails with some math’s stuff.

Now I’ve picked on the non-quants. I’ll pick on the quants for a bit. I studied quants using Monte Carlo simulation, I found they were systematically overconfident. None of the 35 compared their predictions with the actual outcomes. Does anybody build a model, then say ‘I’m uncertain about this variable or that one’ and then create a control model to figure it out? People don’t and they should.

Misunderestimating: 

Experts and some popular quant methods underestimate catastrophic risk. People think catastrophic risk is less common when there hasn’t been one in a while. They really, really underestimate big risks.

Making the best decisions: 

‘How do I measure ABC?’ 

ALWAYS begin by asking ‘why? what decision are you hoping this data to serve you in making?’ 

It’s a decision modelling problem. Then you need to establish your current uncertainty, if you’re using subjective factors that are based on decent measures then this is good. 

Then you work out how much value would be added by knowing that data. The value of a measure might be high, or the cost of figuring it out might not justify trying. 

The more uncertainty you have, the bigger the uncertainty reduction you get by a small amount of data. The highest pay off from measuring tends to be from a small amount of data. 

Everybody is systematically measuring the wrong stuff. They measure costs more than benefits, even when benefits are less certain than costs. They ignore the highest value stuff. 

The two biggest measures in IT were ‘adoption rates’ and ‘cancellation rates’. These don’t get measured as much as costs stuff or whatever.

Question: He’s givnig examples of what his clients wanted to measure vs what they should actually measure. It seems to me this isn’t about ‘how to measure anything’ it’s about ‘how to know what to measure’. Is that correct? If so, how do you do that? What’s that got to do with how you measure it?

Given that one of the things leading to inconsistency in human judgement is changing levels of risk aversion, you should work out mathematically investor risk aversion level (how much risk they perceive as being acceptable relative to return)

SIDENOTE:  I’d like to figure out my own risk aversion in relation to stock market investing, starting a business, mortgages and spending money on leisure. ‘Scared money never wins’ so those than invest beyond their comfort level or means are in trouble as well as those that under-invest and miss out on reward. Where’s the balance for my temperament? 

He’s created an AIE model for people to use. Every assumption or decision making criteria should be tested.

SIDE NOTE: This is similar to how Ray Dalio says he makes decisions in his book principles. Except Douglas Hubbard is relying mostly on data, whereas Dalio is talking more about other people’s opinions, which they must articulate and defend using reason. 

Application of these ideas to my current work

State simply the key idea(s) in a few sentences’:

Human intuition is inferior to math’s done properly, therefore data should be relied on for reducing uncertainty in decision making. Even a small amount of data is powerful when uncertainty is high. 

Begin by asking what decisions you’re expecting the measure to help you with, then tailor it to that. 

Most people measure the wrong things. You should measure to reduce uncertainty and take advantage of big opportunities. Costs are actually quite predictable, so why do so many companies spend more time measuring them than benefits? 

‘Apply this idea to an aspect of your life of your choosing’:

Watched this vid as part of work. Trying to measure the value added by my department.

Ok, so ‘what decision are you expecting the measure to help you with?’

(went away to do a mind map)

Decisions:

  1. Which of our projects and forms of support are the highest value
  2. ‘Value’ as according to perception of seniors, perceptions of customers and empirical evidence of actual impact.

Therefore, what we are trying to do is identify the ‘sweet spot’ where what we are doing is satisfying management, good for our reputation with customers and crucially, objectively having a large impact.

The way to figure this out is to investigate the following:

  1. Perceptions of management as to what value we offer and what would make them satisfied
  2. Perceptions of customers (what kind of value do they associate us with, what kind of value would they appreciate the most).
  3. Objective comparison of the relative impact of our activities and projects, so that we know what kinds of things add the most value. (applying LEAN to ourselves). This means applying 80/20 rule perhaps to identify the things we do that are more effective.

‘Apply this idea to your current goals and longer term goals. To what extent does it apply, and what’s the significance of it’s application?’