You Get What You Measure

Listening to the news today and there is a story about someone (an expert) advising language teachers to coach their students for the oral exam by practicing the conversation as scripts. The conversations are apparently recorded and an examiner listens to them to award a mark in an exam.

Is this cheating? If you don’t live in a country or community that regularly uses the language, then to some extent you probably are learning the language by rote. The exam is to test the student’s ability to be able to hold a conversation in a foreign language. In this case, the student can hold a specific conversation, but not an impromptu conversation. So I’d say it’s cheating! 

Why are they (the teachers) doing this? Well exams are not just about how good the student is. The exams are used to determine the teachers and schools performance. By having more students pass or at higher grades they (teachers and schools) improve there position relative to their peers. It probably means more money or at least less interference.

I was reminded of the need to cut down times on hospital waiting list. In this case time on the waiting list were cut but having “pre-waiting lists” or if the patient was going to be on the list too long referring them back to the GP and taking them off the list. All sorts of innovative ways to ensure those that were measuring this got what they wanted.

The police crime rate clear-up (the percentage of crimes solved) is measured and at one time they were doing deals with criminals that if they confessed to more crimes. So the crime clear-up rate went up but the amount of crime also went up! Why? Well once a crime had been “solved” there was no further investigation, so the real culprit was never found and was free to continue to commit crimes. And do we believe the statistic anyway?

A better measure would be to add up the amount of time officers spent in the community. To both help solve crime and to reassure the public.

Again those that were doing the measuring got what they wanted but like so many measures the effect and affect (or the action and results) are not linked as tightly as first thought.

So sales targets and bonuses result are difficult in these times. Innovative sales people insure they are associated with any deal that is about to be one, especially if its big. They spend more time monitoring their industrious colleagues then creating and closing their own deals. Lo and behold all the sales targets are met but the overall sales/sales person are falling. The company is blind to the true market position.

Boards of companies often have a measure on their performance that is related to the share price. If the price per share goes up they must be doing a good job! But being an innovative lot they organise the company to use its profit (or maybe take out a loan) to buy back shares from the market. Simple example: company worth £100m and 200m shares, value per share 50p. Company spends £50m and buys 100m shares now the company is still worth £100m but only 100m shares so the value of each share is £1. The board pat themselves on the back for their ability to double the share value and award themselves large bonuses on the back of their success.

So be careful what you measure as you will probably get what you measure, but not necessarily what you want.