I've been meaning to write about this all day, but I'm only just now getting around to it.  Apparently, the Battle For EFCA has escalated to the point that different groups are going to Ambinder and leaking their polling. Of course, it would be nice if they would leak the datasets, sampling information, etc., but what can you really expect?  Anyway, it looks like this post has some nuggets in here that make for interesting commentary on how to poll.

At first glance, it looked like a case of the Coalition For A Democratic Workplace using loaded language in order to get the result that they wanted. I mean, come on, look at the wording:

There is a bill in Congress called the Employee Free Choice Act which would effectively replace a federally supervised secret ballot election with a process that requires a majority of workers to simply sign a card to authorize organizing a union and the workers' signatures would be made public to their employer, the union organizers and their co-workers. Do you support or oppose Congress passing this legislation?"

Not long ago, I wrote about the importance of using neutral language in poll when you're trying to get a good measurement. This is why the AFL's version of the question reads:

"[a]llows employees to have a union once a majority of employees in a workplace sign authorization cards indicating they want to form a union."

Now, at first glance, it seemed to me like the CDW had to be using loaded language to get the results that they wanted simply because no respectable pollster would ever, ever ask the question that way because of the incredible bias that it would introduce into the response.  No one would ever use the answers to that question to measure what kind of support that EFCA has. The language is just too inflammatory and over the top.  No respondent who heard it could fail to tell exactly how the interviewer wanted him to respond.   But then I started thinking about it, and I realized that McLaughlin is not a complete idiot. There was absolutely no way that he was using that question to measure support.

So what was he doing? Why was that question in there? This would be easier to confirm if we had the entire questionnaires from each poll to compare, but I'll bet dollars to donuts that this question was asked for purposes of message-testing.  My colleague Student Redux described this as "parse polling":

Parse polling is intimately related to policy polling, but with a twist. Where policy polls focus on communicating popular opinion, parse polls focus on the most popular ways to communicate an opinion. Parse polling is about crafting a message, not crafting policy

There is absolutely no way in hell that an astroturf group like CDW was actually polling to find out what kinds of policies people actually support and how to builid their agenda.  What they were doing was testing a version of the message so that they could find the most effective rhetoric, themes, etc. for their ad campaigns.

So, how do they do they do message testing? It's actually pretty easy, but it reveals another nasty practice that pollsters do: splitting the sample. Let's assume that that you have a sample of 500, and that you want to test two different messages.  You randomly assign each member of the sample to be in either Split A or Split B.  People in A will get message A, those in B will get message B.  Who sees the problem?   In a separate and unrelated post, I wrote about why small sample sizes are more consequential than people think:

...The point of a poll is not to ask some set of arbitrary questions to some arbitrary group of people and then look at them as a lump.  The point of a poll is to ask a set of carefully considered questions of a carefully constructed group of people in order to allow you to make projections about the whole universe.  In other words, think of your n as not being one big lump, but the aggregate of lots and lots of little, overlapping lumps, and inter-connecting to build your n.    So, if you have a poll of 500 people that's broken out as follows:

 

• 50% Male, 50% Female
• 35% African-American, 40% White, 15% Latino, 10% Other/Don't Know/Declined to State
• 15% Some College, 50% Bachelors Degree, 20% Some Post-Grad, 15% Other/Don't Know/Declined to State

That means that you'll have an n of 250 for all men and an of 250 for all women. But you're not just going to look at men and women as men and women: you're going to look at them against all the other relevant variables.  So, for example, you're going to want to see those who are Male, African-American and has a Post-Grad degree answered your questions.   So, let's do a little arithmetic.

500 Respondents * 50% of being male * 35% of being African-American and 20% chance of being Post Grad = 17.5 cases, realistically, 18.

Suddenly, that rationalization of going for 500 completes no longer seems like a good idea. The confidence that you have in those observations is so low that it's not even worth looking at - it's probably not worth the paper it's printed on. (I'm sure that every firm out there has a floor for how low some sub-universe has to be before they'll examine it. I've seen numbers ranging from 50 to 100.)

Now, everybody knows this, but it usually winds up getting dealt with in one of two ways:

• People ignore it, and act as if the only relevant MoE is the MoE of the whole poll instead of the Margins of Error of all the sub-universes or
• People acknowledge the problem, wring their hands, and say "What are you going to do? That's the nature of the biz."

You've just made it a whole lot worse by adding another factor in there that splits your number of observations!  If you were looking at African-American males with post-graduate education to see how they reacted to the messages, you would have 9 cases for split A and split B!  How's that Margin of Error looking now? HINT: It's not pretty.

Well, this has been fun.  It goes to show you what you can learn about a poll just by looking at the wording of the questions!

DD