Skip to content

When (French) Polls Go Wrong

Ballots and Bullets is delighted to be publishing several guest blogs from the two minds behind 500 signatures. Professor Jocelyn Evans and Dr Gilles Ivaldi offer expert insight into the forthcoming Presidential elections in France. Prof. Evans is currently working with Matt Goodwin from Nottingham on a study of far right supporters, launching on March 8th.

Since the infamous 1936 Literary Digest poll misforecasting a famous victory for US Republican Alf Landon over the Democrat incumbent Franklin Roosevelt, people have had good reason to be skeptical about pre-election polls. Even by 1948, a more methodologically refined approach to survey research championed by Angus Campbell and Robert Kahn had proved its worth in correctly backing Harry Truman (unlike pollsters, who had all their money on Thomas Dewey). In France, the more conceptual critique of mass opinion polling as a “science without a scientist” by sociologist Pierre Bourdieu has nurtured immense distrust in what American political scientist Benjamin Ginsberg alternatively referred to as a “subtle instrument of power”.

Even below the heady heights of critical sociology, pre-election polls have constantly been the subject of sharp criticism in French politics for simple empirical failure. In 1995, pollsters came under fire for pre-emptively crowning centre-right candidate and outgoing Prime Minister Edouard Balladur as the new Head of State. Seven years later, scathing criticism was directed at polls once again for not having anticipated Lionel Jospin’s elimination from the 2002 presidential race. It is only because things went “according to plan”, i.e. a straight Left-Right run-off in 2007 that the enduring lack of accuracy in the polls – most notably, the over-estimation of Jean-Marie Le Pen’s eventual score – went almost unnoticed.

The current presidential campaign is not immune to this type of debate. We have already addressed some of these issues in two previous posts on ‘house effects’ and the discrepancies that can be observed across pollsters. Two recent developments in the campaign have turned our attention back to the significance of pre-election polls and the need for some transparency in the methods employed by polling institutes.

First, the Council of State (Conseil d’Etat) has confirmed the right of French pollsters to keep their methods for poll adjustment and data weighting under wraps. In their decision, the administrative jurisdiction deemed such practices equivalent to business ‘trade secrets’. This has infuriated left-wing harbinger Jean-Luc Mélenchon, who initially brought the case to the Council following a controversial Harris Interactive poll allocating him a measly 3 per cent of the vote, and after urging the institute to disclose their calculation methods. Mélenchon has indicated he will lodge his case at the European Court of Human Rights.

The absence of transparency has not escaped the politicians. This very question was central to the provision of a cross-partisan bill proposed in February 2011 to amend the 1977 law on polling, but to date is still pending a vote by the National Assembly. Whilst understandable for commercial reasons, the reluctance of French pollsters to disclose their in-house processes of post-survey weighting contrasts sharply with best practice elsewhere.

For example, the British Polling Council – the counterpart to France’s Commission des Sondagesrequires that pollsters make available all weighted and unweighted figures. A half-way house is Italy, which legally requires polling institutes to reveal their sampling methods, non-response rates and post-collection adjustments – but in a survey by Dr Graziella Castro of the University of Salford, only around 7 per cent of institutes actually did so.

Another controversy has been triggered by the publication of a series of polls testing vote intentions on the basis that Marine Le Pen does not appear on the ballot paper. Each of these polls provided a different picture of what the first round of the presidentials would look like should the FN candidate fall short of the precious 500 signatures (see chart below).

IFOP, for instance, anticipates a tie in the first round between Hollande and Sarkozy, should Le Pen not be present. BVA and IPSOS see a rise in Sarkozy’s vote, but not sufficient to reach his Socialist rival. As Le Monde’s polling blog notes, one reason for IFOP’s more favourable score for Sarkozy is its omission of the smaller candidates, many of them on the Right. But that still does not explain a number of other disparities.


Summary of Polls “without Marine Le Pen”*


 *Candidates’ gains under the hypothesis that Marine Le Pen would not be able to run in the first-round

What emerges from this latest discussion is disagreement among pollsters over the use of internet panel surveys (the so-called ‘CAWI method’) in electoral polls. Interestingly, a similar debate had already taken place in March 2011 after Harris Interactive released a much commented internet-based poll showing Marine Le Pen leading the first round of the presidentials with 23 per cent of the vote.

The argument made by detractors of internet samples is that of the specific mode of recruiting panel members via commercial sites, often in exchange for blatant economic incentives such as shopping vouchers and the like. Whilst they might at a pinch be representative of the social structure of the French population – although again, without transparent reporting of sampling frames, quotas and non-response, even this has to be taken on trust – the attitudinal profile of those respondents might differ.

It might also be the case that highly motivated far right sympathizers will seize the opportunity to express their views – a phenomenon that is not limited to the Internet. Witness, for instance, the attention-grabbing headline from Le Pen’s campaign site that Marine leads the polls amongst iPhone owners. Lastly, it is often argued that internet polls might help overcome the traditional ‘shameful vote’ problem, voters being more prone to reveal their true preferences when seated in front of their computer screen, rather than engaged in conversation with a human being in a CATI setting. One cannot help but draw parallels with individuals’ apparent willingness to engage in the sort of vicious outbursts and slander common to all web discussions that they would shy well away from in less anonymous interactions.

There is empirical evidence that the internet polls consistently bias scores relative to telephone polls. Taking all election surveys since June 2011 asking for vote intentions, and controlling for the time-period in which they took place, our calculations based on 56 polls published since June 2011 indicate that Hollande tends to score around 2.5 per cent less in internet polls, and Marine Le Pen two-and-a-half points higher. Similarly, the gap between Le Pen and Sarkozy – that crucial score that defines whether the run-off becomes another referendum on democracy, as in 2002 – is around 2.5 points narrower amongst internet polls. Going on internet polls, then, the race is tighter, with Marine Le Pen threatening to overtake the incumbent president. Telephone polls suggest at this stage a more likely two-candidate run-off with Le Pen trailing.

Of course, these differences are relative. It is possible that the internet polls represent the ‘true’ (pace Bourdieu) system of social forces at play in the French electorate, and the telephone polls are overselling Hollande at the expense of Le Pen. Furthermore, such a net trade-off is simplistic – the shifts by candidate aggregating to this position are likely more complex, and again due to differences in sampling rates and latent bias.

But this again highlights the current problem. If we cannot know exactly who is answering the question, “If the election were held tomorrow, how would you vote?”, and with what weighting, it is impossible to begin even to assess which scores are factual, which are artefactual, and which are outright fiction.

France may not yet be characterised as a sondocrazia like its Southern neighbour, where polls are seen as explicit tools for electoral manipulation and a “guide” to voters; but to the extent that polls do influence voters in their suggestion of the likely winners and losers, the political information which pollsters provide as a public service should at least come with their own instruction manual.

Jocelyn Evans and Gilles Ivaldi

Published inElectionsEuropean PoliticsFrance 2012

Be First to Comment

Leave a Reply

Your email address will not be published.