Friday, June 27, 2014

Why are YouGov so obsessively secretive about the most important detail of their referendum polling?

I'm indebted once again to Calum Findlay for pointing out something that I hadn't noticed - YouGov are repeatedly showing a higher No vote among people who voted SNP in 2011 than any other pollster.  As an illustration, here are the figures from the most recent poll released by each firm...

No vote among 2011 SNP voters -

ICM : 15%

TNS-BMRB : 13%

Survation : 17%

Panelbase : 14%

YouGov : 24%

The pattern is the same even after Don't Knows are stripped out...

ICM : 16%

TNS-BMRB : 17%

Survation : 20%

Panelbase : 16%

YouGov : 26%

Figures aren't available for Ipsos-Mori, because they don't weight by 2011 vote.  But it's significant that TNS-BMRB are well in line with the other pollsters on this point, because like Ipsos-Mori and YouGov, they're one of the most No-friendly firms at the moment.  So whatever the reason for YouGov producing such a high No vote among SNP voters from 2011, it can't be a factor that is generic to all No-friendly pollsters.  Indeed, it may be a sign that YouGov are No-friendly for a completely different reason to Ipsos-Mori and TNS - which has always seemed intuitively likely, given that YouGov are so far out of step with all of the other firms that use online fieldwork.

As Calum suggests, the most likely explanation is the unique and rather eccentric weighting procedure that YouGov employ.  Like all other BPC pollsters apart from Ipsos-Mori, they weight by recalled vote from 2011, but they make one crucial exception - they separate out the people (or at least some of the people) who voted Labour in 2010 and then switched to the SNP in 2011.  Although SNP voters as a whole are often upweighted from the raw data, the upweighting usually occurs almost exclusively among the group that voted Labour in 2010.  In the most recent poll, that group was upweighted from 55 real respondents to 105 'virtual' respondents, meaning that the referendum voting intention of every person within the group was effectively counted twice.  We can reasonably infer a few things from this -

1) This group are probably producing much more No-friendly figures than SNP voters at large.

2) YouGov are struggling to find enough people in this group, so if the tiny sample they do have are in any way unrepresentative of Labour-to-SNP switchers, that error will be dramatically magnified after the weighting, and the same problem will occur in each and every poll.  It's quite likely that exactly the same people are being interviewed over and over again.

3) If YouGov didn't split SNP voters into two separate groups, the Yes vote would probably be higher in the headline figures, and the No vote would probably be lower.

This is all guesswork, of course, because for some reason YouGov only ever reveal the voting intentions of SNP voters as a whole, after weighting has been applied - there's never a breakdown for the two distinct groups they use for weighting.  It's very hard to understand why they would keep that information a secret, unless it shows such an improbable disparity between the two groups that eyebrows would be raised about the wisdom of the methodology.

To be fair, it's not completely impossible that there may be method in YouGov's madness - they were more accurate than ICM and Survation in the European elections, after all.  But if, for example, the purpose of this weighting procedure is to correct for a Yes-friendly bias that is inherent in volunteer online polling panels, it's very hard to understand why a non-online pollster like TNS would be producing a No vote among 2011 SNP voters that is broadly in line with the other online pollsters.

In any case, what is so special about Labour-to-SNP switchers?  There were two other important groups of switchers in 2011, namely Lib Dem-to-SNP, and Lib Dem-to-Labour (the size of the latter group was masked by the direct Labour-to-SNP swing).  Surely if there's wisdom in separately weighting one of those groups, there must be wisdom in separately weighting the other two groups as well?  Why the inconsistency?

Given Peter Kellner's well-known agenda on the subject of independence, it's sometimes very hard to escape the conclusion that he starts by working out what sort of headline numbers would "feel right" to him, and then works backwards to devise a methodology that will generate those numbers.

13 comments:

  1. Very good and well balanced argument and conclusions.

    ReplyDelete
  2. Good analyisis James and thanks to Callum Findlay for his contribution too.

    As you point out Kellner has a great deal of form already with his agenda. He hasn't even tried to hide his extraordinary bias when it comes to Independence which is why I never put much credibility in YouGov Independence Referendum polling. They have a small utility in spotting trends but not much else. Putting much faith in their headline figures would be extraordinarily foolish.

    Meanwhile the incompetent fop Cameron continues to try and lose the next election despite being up against one of the most feeble Labour leaders we've ever seen.

    Not content with his EU elections 'triumph' the fop thought it was a good idea to bang on about the EU again and have yet another flounce. This when he had no realistic prospect whatsoever of getting the rest of the EU on-board with him.

    Inevitably Flounce 2.0 has blown up in Cammie's face revealing that gullible tory eurosceptics haven't learned a thing and are as dumb as ever. It also leaves Cammie looking weaker than ever with Farage beaming happily all over the News. Not much prospect of the tories getting back all their kipper voters right now when the incompetent fop keeps making the exact same dimwitted mistakes.

    Lest you think that's the only example of Cameron's unbelievably appalling judgement, mere days after his own spindoctor Coulson is found Guilty, Cameron's senior aide and friend Patrick Rock has just been charged with making and possessing child abuse images.

    Make no mistake if there was an even halfway competent Labour leader right now (instead of the pathetic little Ed) then the tories would be getting beaten easily by double figures consistently instead of being so close in the polls.

    Though of course that goes both ways and if the tories didn't have an incompetent fool like Cameron at the helm (following Osbrowne's imbecilic 'master strategies' like EU Flounce 2.0) then they would be easily beating the ineffectual little Ed by double figures as well.

    As for the toxic Calamity Clegg, well, who takes him seriously any more? The yellow tories are quite obviously heading for a 2011 style annihilation in 2015 and Clegg's ostrich faction don't even seem to care.

    With three out of touch incompetent leaders like that as Westminster's 'finest' for 'better together' it's hardly a surprise the results of the mass canvass were what they were.

    ReplyDelete
  3. MORI had 15% of current SNP VIers for No in Feb down to 11% in their recent poll.

    Same as the others. Yougov problem most definitely linked to what you are saying.

    Also way to many English people etc.

    The problem with a polarising binary question like this is that any mistake that favours one group which is strongly biased to one side is that it can shift your result massively as what you take from one often goes straight to the other.

    If you are VI polling, then an method issue which takes 3 points of the other parties doesn't have much impact.

    However, if you take 3 points from Yes and hand them to No, you create a decent six point gap when e.g. there was actually parity. 40/40 becomes 37 / 43 just like that...

    ReplyDelete
  4. Sorry, that should be:

    If you are VI polling, then a method issue which e.g. takes 3 points off the SNP and gives one each to the other parties doesn't have much impact.

    ReplyDelete
  5. What you say is also backed up by the weirdly high No amongst Labour 2011 voters in Yougov.

    71% Yougov
    65% ICM
    57% Panelbase
    54% Survation

    ReplyDelete
  6. Oops

    64% TNS

    ReplyDelete
  7. As a postscript to the jaw-dropping 'judgement' of the incompetent fop and his chumocracy...

    Michael Crick reports that Cameron was going to make his friend and senior aide Patrick Rock a Lord just before his arrest!

    You couldn't make it up, as Coulson and his red top chums also used to say.

    ReplyDelete
  8. YouGov being willing to fudge the figures shouldn't come as a surprise. The current head of the company used to be a Conservative MP and the former is a Conservative MP. YouGov being in effect an extension of the Party.

    ReplyDelete
  9. Daily Mail claiming that the SNP are holding crisis talks, as yougov polls showing No ahead of yes by 20%

    They really do think our heads are zipped up the back!

    ReplyDelete
  10. Over on Stormfront Lite the politicalbetting twits are trying to decide just how big a triumph Cammie's Flounce 2.0 is while the usual racist lunatics foam at the mouth.

    They seemed to have inexplicably missed the part where Cammie was utterly humiliated along with the stupendously gullible (but always amusing) tory eurosceptics.

    The Mail have about as much insight into the referendum and scottish politics as the out of touch PB tory twits do. Which would be zero.

    ReplyDelete
  11. Sunshine on CrieffJune 28, 2014 at 8:28 PM

    Patrick Roden

    A very misleading article in the Mail. Even with YouGov's questionable methods their latest poll actually showed a fall in the No lead in those certain or very likely to vote. Don't forget that YG don't take likelihood to vote into account in their headline figures, unlike other polling companies.

    And if the No campaign was in panic in March, then they have no reason to relax now. I make the average lead amongst the 80% or so of people most likely to vote is now below 10%. Game on!

    ReplyDelete
  12. Lovely friendly comment supporting Andy Murray on the Telegraph. He's done everything to get their support apart from get a union jack swastika tattoo (afaik) and they still hate him because he's Scottish.

    jelliedeels • 9 hours ago
    No English people should support this sweaty unless he takes an oath to put England first .

    its an englishmans birthright to look on the sweaties with contempt

    ReplyDelete