(Scottish) opinion polls – a reminder

Standard
English: Northern Ireland opinion poll on Prot...

English: Northern Ireland opinion poll on Protestant identity in Northern Ireland. Data from the Northern Ireland Life and Times Survey http://www.ark.ac.uk/nilt/. (Photo credit: Wikipedia)

This is not about which way you should use your vote if you have one – that is here.

Instead it’s reminder of the maths of opinion polling, because I suspect we are going to see great numbers of polls in the next few weeks.

So here are some things to remember:

1. The best an opinion poll can do is tell you what a good opinion poll would show. In other words, opinion polls cannot be thought of as reliable predictors of results. Simply put – if people systematically fail to tell the truth to opinion pollsters then no opinion poll is going to perfectly correct for that (the pollsters try but their work here is merely based on informed guessing). So when pollsters talk about “margins of error” they don’t mean in comparison to a real election result, but to what another – well taken – poll would show.

2. One in twenty opinion polls – no matter how well conducted – will be very wrong. This is the so-called “rogue” poll and it’s incorrectness is not because it has been conducted improperly but because sampling a small subset of a large population is inherently statistically risky.

3. Doubling the sample size does not mean your poll more twice as accurate. In fact it only makes it \sqrt2 more accurate. The important point here is that when you look at small samples – such as Scottish regions – you are looking through the other end of this telescope – so a smaple that contained \frac{1}{5}th of the poll sample would actually have a margin of error that was \sqrt 5 (or about 2.2) times bigger (and that assumes the sampling in that region actually matches the population in that region as opposed to Scotland as a whole – if it doesn’t, and the chances are that it won’t, then you are better off just ignoring the subsample).

4. “Margin of error” is really a measure of how likely other polls will give the similar results. We have already covered this – but here’s a longer explanation. If we say that the margin of error on a poll is plus or minus three per cent, then typically what we mean is that 95% (i.e., 19 out of 20) polls will give results where the figures do not differ by more than three per cent. This also means if you describe a 1 per cent change in a rating as in some way significant then you are very wrong – because actually your poll does not give you enough information to make that claim. To go from a 3% margin of error to a 1% margin requires you to increase the sample size by a factor of 9. To go to a margin of plus or minus 0.5% would require an increase in sample size by a factor of 36.

5. The margin of error actually depends on the score polled. The highest margin of error is at 50% – where for a 1000 sample poll it is:

2 \times \sqrt \frac{0.5 \times 0.5}{1000} = \pm3.4\%

For 40% the margin becomes 2 \times \sqrt \frac{0.6 \times 0.4}{1000} = \pm 3.1\%

(And these figures are for the numbers before the don’t knows are discounted.)