Here's the actual limerick:
There was a young man from place B
Who satisfied Predicate P,
He performed action A,
In adjective way,
Resulting in Consequence C.
Thursday, 20 May 2010
Tuesday, 18 May 2010
Wikipedia can't add up
Have a look at this list of best-selling books. In particular, look at the Harry Potter books. There is one Harry Potter book in the list of best-selling books. Harry Potter and the Deathly Hallows, which has sold approximately 44 million copies. Harry Potter is also listed as the best-selling series of all time, with 400 million total copies sold.
In order to see why there is a problem with this, you just need to know that there are 7 Harry Potter books, and understand the Pigeonhole Principle.
There is a (sort of) explanation of the problem in the 'talk' page of the wikipedia article, but the inconsistency remains on the initial page. This actually highlights a fairly major problem with wikipedia's epistemology. The inconsistency has to remain because there's no 'credible source' for the figures for the other individual books. So even though everyone knows that the article is wrong, it can't be corrected because this would constitute 'original research'. Hmm... just how credible does a 'credible source' have to be? Will a mathematical proof do?
In order to see why there is a problem with this, you just need to know that there are 7 Harry Potter books, and understand the Pigeonhole Principle.
There is a (sort of) explanation of the problem in the 'talk' page of the wikipedia article, but the inconsistency remains on the initial page. This actually highlights a fairly major problem with wikipedia's epistemology. The inconsistency has to remain because there's no 'credible source' for the figures for the other individual books. So even though everyone knows that the article is wrong, it can't be corrected because this would constitute 'original research'. Hmm... just how credible does a 'credible source' have to be? Will a mathematical proof do?
How to sample randomly
I was once told that 'duration of unemployment' figures were collected in the following way: people were telephoned at random during the day. If they answered, they were asked if they were unemployed. If they said yes, they were asked for how long they had been unemployed. Before you read any further, can you see what is horribly, horribly wrong with this method of data collection? (There are several things wrong with it, but one of them renders it entirely useless)
Before I give you the answer, a brief detour. When I was at school, I did a piece of statistics coursework (I think it was for GCSE's) in which I compared the average sentence length in a French text to an English text. I can't remember exactly which texts I chose, I think it was newspapers of 'equivalent' quality, but that's largely irrelevant. In order to estimate the average length of sentences in each text, I adopted the following method: pick a word uniformly at random from the text and count the number of words in the sentence containing it.
I had collected around 100 sentence lengths before I noticed the utter ridiculousness of this method. In case anyone hasn't spotted it yet, this 'random sampling' is guaranteed to massively overestimate the average sentence length in any given document, as the probability of any given sentence being chosen is in direct proportion to its length.
Consider the following passage:
"The quick brown fox jumps over the lazy dog whilst the five boxing wizards jump quickly over my lovely sphinx of quartz. Jesus wept"
If we pick a few random words from this and compute the 'average' sentence length of the sentences that contain them, we're going to come up with something very close to 20 (if we pick every single word, we'll get 20.3333) The actual average sentence length is 12.
Now, if you didn't immediately spot that this was the key problem with the method of collecting unemployment data I mentioned in the first paragraph (there are problems with telephone polls in general, of course, but they are essentially insignificant compared to the problem with the sampling method), this should make you worry about how easy it is to slip *exceedingly* dodgy statistics past people who aren't paying attention. I'll post a few examples of my favourite 'correlated for spurious reasons' statistics in another post later this week.
As an aside - if you do actually collect the data in the way suggested, you can presumably still get some information about the distribution you're studying - what's your best estimator for the mean? And what assumptions do you have to make about how the data are distributed?
Before I give you the answer, a brief detour. When I was at school, I did a piece of statistics coursework (I think it was for GCSE's) in which I compared the average sentence length in a French text to an English text. I can't remember exactly which texts I chose, I think it was newspapers of 'equivalent' quality, but that's largely irrelevant. In order to estimate the average length of sentences in each text, I adopted the following method: pick a word uniformly at random from the text and count the number of words in the sentence containing it.
I had collected around 100 sentence lengths before I noticed the utter ridiculousness of this method. In case anyone hasn't spotted it yet, this 'random sampling' is guaranteed to massively overestimate the average sentence length in any given document, as the probability of any given sentence being chosen is in direct proportion to its length.
Consider the following passage:
"The quick brown fox jumps over the lazy dog whilst the five boxing wizards jump quickly over my lovely sphinx of quartz. Jesus wept"
If we pick a few random words from this and compute the 'average' sentence length of the sentences that contain them, we're going to come up with something very close to 20 (if we pick every single word, we'll get 20.3333) The actual average sentence length is 12.
Now, if you didn't immediately spot that this was the key problem with the method of collecting unemployment data I mentioned in the first paragraph (there are problems with telephone polls in general, of course, but they are essentially insignificant compared to the problem with the sampling method), this should make you worry about how easy it is to slip *exceedingly* dodgy statistics past people who aren't paying attention. I'll post a few examples of my favourite 'correlated for spurious reasons' statistics in another post later this week.
As an aside - if you do actually collect the data in the way suggested, you can presumably still get some information about the distribution you're studying - what's your best estimator for the mean? And what assumptions do you have to make about how the data are distributed?
Saturday, 15 May 2010
Why don't we sample more?
Steve Landsburg recently blogged about a maths professor who weeds out 'unlucky' applicants by randomly rejecting half of the resumes he gets sent. Now, this is unusual, in that it is a random sampling method which significantly *reduces* the average quality of the applicant that gets hired.
There are a *lot* of situations in which random sampling would reduce workload whilst having no effect whatsoever on effectiveness. I'll start with one of the simplest and least controversial (and one that I have the most personal experience with). Students regularly submit 10 or more pieces of coursework for each course in a university semester. Every question is then marked, and the papers returned to the students. Assuming (which is probably not entirely accurate) that the courseworks are solely intended as a normative assessment of student performance, surely it would be massively more efficient to sample questions at random and mark those, rather than marking the entire paper. The expected mark for any given student is the same - only the variance goes up.
There are a few situations in which students suffer as a result of this. Say there's a pass mark of 40, and you have to pass every coursework, now someone who answers exactly 40% of the questions right in each coursework expects to fail (although they do expect to get an average mark of 40). Similarly, there are situations in which students benefit from this (pass mark of 40, answer exactly 39% of the questions correctly, you now have a non-zero chance of passing). On the whole, I would expect these things to cancel out, and that no one student knows their mark accurately enough to know whether they would benefit or lose out from this policy being enacted.
So why isn't this done more? I've heard from a few lecturers who've tried it, and it went down horribly with the students, who perceive it as 'unfair'. Apparently there were several comments along the lines of 'what if you only mark the questions I did badly?'. I guess this is some sort of loss aversion - it is quite obviously equally likely that we only mark the questions you did well!
Yvain has an article about a similar example from education - in which students are reluctant to guess answers to true/false questions with a penalty of 50% of a point for a wrong answer for some inexplicable reason. Again, random sampling is a massive net win.
Another example is public transport. No-one every pays to get on the 25 bus. This is because it is extremely rare for anyone to check whether you've paid or not and the penalties just aren't high enough to make it worthwhile paying given how rare the checks are. There are two obvious solutions to this problem: you could either do twice as many checks (thus requiring you to hire twice as many people to do the checking, and inconvenience twice as many people whilst checking) or you could double the fine. I've no idea why they don't take the second option.
How about voting? Instead of counting all of the votes in a general election, why not shake the votes up in a big bowl and count, say, the first 10,000 for any given seat? I can't be bothered to crunch the numbers, but I'm pretty sure the probability of error would be down below 1% - and errors would only occur in seats which were closely contested - where errors are not so important anyway, as the people obviously don't have a clear preference between the candidates.
Most of the examples I can think of exploit the same principle as the public transport idea above - when committing some transgression, your expected utility is the utility of cheating minus the disutility of punishment times the chance of getting caught. Since it's expensive to increase the chance of getting caught, there are a lot of situations in which I think it would be a net win to decrease this and increase the size of the punishment. Why not check half as many tax returns and double the fine for misfiling? Have half as many speed cameras and double the fine for speeding (speed cameras aren't expensive, so this might not be a net win)?
There are dozens of examples - and I don't think that the people in charge have sat down and done the relevant calculation in all cases. Are people just afraid of randomness? Afraid of seeming 'arbitrary'? Afraid of letting people 'get away with' committing crimes - assuming the only legitimate purpose of the criminal justice system is deterrence, this shouldn't be an issue. Maybe there's legitimate concerns that a 'random sampling' approach to some of these problems would be more subject to corruption - but we can just check a few of the samplers at random, and have massive fines for people doing it corruptly!
The law of large numbers is a powerful and important mathematical theorem. Why don't we exploit it better?
There are a *lot* of situations in which random sampling would reduce workload whilst having no effect whatsoever on effectiveness. I'll start with one of the simplest and least controversial (and one that I have the most personal experience with). Students regularly submit 10 or more pieces of coursework for each course in a university semester. Every question is then marked, and the papers returned to the students. Assuming (which is probably not entirely accurate) that the courseworks are solely intended as a normative assessment of student performance, surely it would be massively more efficient to sample questions at random and mark those, rather than marking the entire paper. The expected mark for any given student is the same - only the variance goes up.
There are a few situations in which students suffer as a result of this. Say there's a pass mark of 40, and you have to pass every coursework, now someone who answers exactly 40% of the questions right in each coursework expects to fail (although they do expect to get an average mark of 40). Similarly, there are situations in which students benefit from this (pass mark of 40, answer exactly 39% of the questions correctly, you now have a non-zero chance of passing). On the whole, I would expect these things to cancel out, and that no one student knows their mark accurately enough to know whether they would benefit or lose out from this policy being enacted.
So why isn't this done more? I've heard from a few lecturers who've tried it, and it went down horribly with the students, who perceive it as 'unfair'. Apparently there were several comments along the lines of 'what if you only mark the questions I did badly?'. I guess this is some sort of loss aversion - it is quite obviously equally likely that we only mark the questions you did well!
Yvain has an article about a similar example from education - in which students are reluctant to guess answers to true/false questions with a penalty of 50% of a point for a wrong answer for some inexplicable reason. Again, random sampling is a massive net win.
Another example is public transport. No-one every pays to get on the 25 bus. This is because it is extremely rare for anyone to check whether you've paid or not and the penalties just aren't high enough to make it worthwhile paying given how rare the checks are. There are two obvious solutions to this problem: you could either do twice as many checks (thus requiring you to hire twice as many people to do the checking, and inconvenience twice as many people whilst checking) or you could double the fine. I've no idea why they don't take the second option.
How about voting? Instead of counting all of the votes in a general election, why not shake the votes up in a big bowl and count, say, the first 10,000 for any given seat? I can't be bothered to crunch the numbers, but I'm pretty sure the probability of error would be down below 1% - and errors would only occur in seats which were closely contested - where errors are not so important anyway, as the people obviously don't have a clear preference between the candidates.
Most of the examples I can think of exploit the same principle as the public transport idea above - when committing some transgression, your expected utility is the utility of cheating minus the disutility of punishment times the chance of getting caught. Since it's expensive to increase the chance of getting caught, there are a lot of situations in which I think it would be a net win to decrease this and increase the size of the punishment. Why not check half as many tax returns and double the fine for misfiling? Have half as many speed cameras and double the fine for speeding (speed cameras aren't expensive, so this might not be a net win)?
There are dozens of examples - and I don't think that the people in charge have sat down and done the relevant calculation in all cases. Are people just afraid of randomness? Afraid of seeming 'arbitrary'? Afraid of letting people 'get away with' committing crimes - assuming the only legitimate purpose of the criminal justice system is deterrence, this shouldn't be an issue. Maybe there's legitimate concerns that a 'random sampling' approach to some of these problems would be more subject to corruption - but we can just check a few of the samplers at random, and have massive fines for people doing it corruptly!
The law of large numbers is a powerful and important mathematical theorem. Why don't we exploit it better?
Friday, 14 May 2010
Lucozade Sport Lite: the Low Energy Energy Drink!
I just saw an advert for Lucozade Sport Lite. Yes. Seriously. This is an energy drink which contains only 50 calories. But don't worry:
The website really is excellent though - it actually include a section entitled 'how do I use it?'. Erm... put it in your mouth and swallow?
Lucozade Sport Lite contains electrolytes and fluid which help to keep you hydratedHmm... a fluid which helps keep you hydrated, I wonder what other things there might be that fit that description....
The website really is excellent though - it actually include a section entitled 'how do I use it?'. Erm... put it in your mouth and swallow?
Wednesday, 12 May 2010
Gladwell on probability
There's quite a nice list of random quotes from Malcolm Gladwell in an interview for this Sunday's Observer.
However, one of them seems to show some misunderstanding of probability:
Eg, let's say we're trying to find out where a particular terrorist group has their headquarters. To start with, our probabilities are essentially uniformly distributed across the whole of the world. Our spy comes up to us and says 'the HQ is at number 32 Barkston Gardens, Earl's Court, London'. This information is far from useless - in fact, if we have more than one spy coincide on the same piece of information then we're in business, and can find the location pretty quickly.
Of course, I think Gladwell's '50%' is actually just a proxy for 'exactly as true as you'd expect if they were generating their statements at random', but that's not *quite* the same thing
However, one of them seems to show some misunderstanding of probability:
History suggests that there is almost exactly a 50% chance that any piece of information a spy gives you is true. We would be as well off getting rid of the secret service and flipping coins.Now if the first part of this sentence is true (which I have no reason to doubt) the second part most definitely does not follow. This is (tangentially) related to a discussion that's been going on at Peter Cameron's blog about probability. Unless spies only ever make statements about things where your prior was already 50%, a 50% accuracy rate could be incredibly useful.
Eg, let's say we're trying to find out where a particular terrorist group has their headquarters. To start with, our probabilities are essentially uniformly distributed across the whole of the world. Our spy comes up to us and says 'the HQ is at number 32 Barkston Gardens, Earl's Court, London'. This information is far from useless - in fact, if we have more than one spy coincide on the same piece of information then we're in business, and can find the location pretty quickly.
Of course, I think Gladwell's '50%' is actually just a proxy for 'exactly as true as you'd expect if they were generating their statements at random', but that's not *quite* the same thing
Tuesday, 11 May 2010
Whacky World Cup Formula: Germany will win
According to this article from the Telegraph (which, incidentally, is a carbon copy of the article in various other news sources from around the world - I assume it's lifted directly from a wire service, but who knows?) Germany are definitely going to win this year's world cup. How do we know this? Trigonometry!
We do get some indication of his track record:
"Nobody can beat us this year and you can already put the champagne on ice."
However, my favourite part of the article is the end, in which Tolan demonstrates that he doesn't understand basic game theory or probability, with regard to penalty shoot-outs:
I'm not quite sure why people who seem to have perfectly respectable research careers get inolved in this sort of thing. I would suggest it was for the money, but he did the same thing 4 years ago... is it just that some people can't resist having their name in the paper?
The scientist has written a formula based on trigonometry which analyses all Germany's results from previous World Cups and predicts a winner for this year's tournament.Unfortunately, Metin Tolan doesn't show his work, so we can't see quite how he came up with this ridiculous conclusion. It's hard to see why any team with an average finish of 3.7 (whatever that means) would expect to finish in position 1. It's also really, really hard to see how a formula which appears to be essentially some sort of regression/reformulation of the law of averages could be 'based on trigonometry'.
Having won the World Cup three times, in 1954, 1974 and 1990, Germany's average finishing place at previous tournaments is 3.7 and Prof Tolan says his formula shows this will be Germany's year to lift the trophy.
We do get some indication of his track record:
Prof Tolan already predicted Germany would win the last World Cup, which they hosted in 2006,So he may be displaying a tiny bit of overconfidence in his predicition to say:
"Nobody can beat us this year and you can already put the champagne on ice."
However, my favourite part of the article is the end, in which Tolan demonstrates that he doesn't understand basic game theory or probability, with regard to penalty shoot-outs:
"The weakest kicker should take the first penalty, then the second-weakest and so on," he said. "Then you have the greatest chance of scoring as many goals as possible."Now this is quite clearly the exact opposite of the truth. I can't even be bothered to crunch any numbers, because it only takes a few seconds of thought to see that if one side follows the good professor's advice whilst the other uses the more sensible plan of doing the exact opposite, the latter side will win before things have even gotten started - the better players on Tolan's team literally won't get a kick.
I'm not quite sure why people who seem to have perfectly respectable research careers get inolved in this sort of thing. I would suggest it was for the money, but he did the same thing 4 years ago... is it just that some people can't resist having their name in the paper?
Subscribe to:
Posts (Atom)