I wish this were humor, I really do.

On the very day I describe how questions can be a source of methodological error in my regular series post, I find out about the new Zogby poll commissioned by John Ziegler.  Nate Silver at 538 has been all over this one, and good on Nate.  But I want to do my own bit of analysis here, just to illustrate my point about questions and methodological error.

There's more...

----------------------------------

First, some background on Ziegler's poll.  He commissioned a Zogby poll of Obama voters (and Obama voters only) to see how ill-informed they were.  Along with your regular demographics, he had them polled on a series of 12 questions - the first, purely factual; the other eleven to be attributed by interviewees to one of the four presidential or vice presidential candidates.  The questions are below, with "correct answers" provided in brackets:

1. Before this past election, which political party controlled both houses of congress? [DEMOCRATS]
2. Which candidate could not say how many houses they own? [JOHN MCCAIN]
3. Which candidate said they could see Russia from their house? [NONE - TINA FEY]
4. Which candidate had to quit a previous political campaign because they were found to have plagiarized a speech? [JOE BIDEN]
5. Which candidate won their first election by getting all of their opponents kicked off the ballot? [BARACK OBAMA]
6. Which candidate wore clothes that their political party reportedly spent $150,000 on? [SARAH PALIN]
7. Which candidate currently has a pregnant teenage daughter? [SARAH PALIN]
8. Which candidate said that Obama would be tested in his first six months as president by a generated international crisis? [JOE BIDEN]
9. Which candidate claimed to have campaigned in 57 states? [BARACK OBAMA]
10. Which candidate said their policies would likely bankrupt the coal industry and make energy rates skyrocket? [BARACK OBAMA]
11. Which candidate said that the government should redistribute the wealth? [BARACK OBAMA]
12. Which candidate started their political career at the home of two former members of the Weather Underground? [BARACK OBAMA]

Full results are available in .pdf form here.  (The link is on Zogby, thank God, so I don't feel bad about generating any hits with it)

Now, Nate has taken issue with the characterization of these statements as "factual".  His point is well-taken, but for the most part these statemens ARE factual, if in some cases very badly worded.  I find Q1 to be a particularly telling example, coming as it does close to a national election.  It could have easily been worded more clearly: "Before the 2008 election, which party controlled both houses of congress."  Just reading this poll, and knowing that some individuals had been asked these questions individually at polling places on Nov. 4th, I assumed the answer was Republicans.  I had to check the date the phone poll was conducted to verify the correct answer.

Similarly, some of the Obama questions in particular are notable mischaracterizations.  Q10 asks about a statement Obama made earlier in the year, that people would be free to open new coal plants, but would bankrupt themselves doing it (unless they used clean coal).  The conversation was, of course, about ways to reduce pollution and consequences of different policy provisions.  The question, however, is worded to suggest an almost malicious intent by the originator of the quote.  Q11 is, I believe, not a direct quote.  It applies about equally to both Obama and McCain in 2000, however, when McCain argued for the benefits of progressive income tax.  Any proponent of progressive taxation, to some extent, supports the redistribution of wealth.  Q11, however, characterizes the issue much more starkly, as something only one candidate could have said (in the context of the poll framework).

Ziegler takes the answers to his poll questions and uses them to argue that Obama supporters are overwhelmingly ignorant and ill-informed, especially about issues relating to their own candidate.  Unfortunately, his conclusions are clearly undermined by the selection and distribution of his own questions.  Of his eleven attribution questions:

• Five refer to Obama
• Two refer to Biden
• Two refer to Palin
• One refers to McCain
• One is a trick question soliciting a Palin response

So clearly the balance of collected data leans towards questions referring to Obama or Biden.  Of course, poll participants are much more likely to get a greater number of Obama questions wrong, simply because there are more Obama questions.

A further confound, very important in this instance, is the media saturation of the stories in question.  Ziegler picks some of the most widely reported negative stories on McCain and Palin as his question material, but some of his Obama/Biden questions refer to issues that received very little airtime in the mainstream media (excluding , perhaps, Fox News, which hardly counts as mainstream - or even credible - media).  Biden's "will be tested" comment and Obama's "redistribution of wealth" position certainly got their fair share of airtime, but Obama's comment on coal power and Biden's earlier abortive presidential campaign were largely passed over.  Q12 about Obama "start[ing] his political career" in the Ayers' living room is also likely to be passed over by many interviewees who find this characterization incorrect.  No parallel questions exist on the Republican side.

Fairness here would probably involve a further study of media saturation of these memes to give comparative analysis.  Somehow, I doubt that's on John Ziegler's agenda.

Fairness would also involve polling McCain/Palin supporters in similar fashion, and Ziegler has challenged democrats to do so, stating that he'll pay for the polling if McCain supporters prove less informed.  Of course, it's hardly a level playing field if anyone sticks to Ziegler's questions for the aforementioned reason.  Unless the questions are balanced in quantity and in terms of media saturation, it's hard to say much of any meaning about the results.  This is not unlike Jack Thompson's "A Modest Video Game Proposal".  And similarly, I doubt very much that Ziegler would be willing to pay up if anyone did the survey without his biased question set.

All in all, I find this a fascinating example of the problems I was just discussing earlier this morning.  John Ziegler's clearly biased efforts in this area serve as an excellent example of the fact that no poll is ever better than the methodology it's built with.