Hi, everyone. I woke up with a bit of insomnia, so I thought that I would post something that had occurred to me in my sleep. (Yes, I know that it makes me an inveterate dork that I think about these things in my sleep. I don't care.) Recently, my colleague, Student Redux, has been exploring methodological problems in sampling and weighting, and I had an idea that I wanted to explore as a possible solution.  Now, granted, I'm not committed to this, and I'd love to have your comments as further exploration of the topic.

There's more...

To do a little injustice to Student Redux's argument, the problem with stratified sampling is that getting counts based on the universe alone, without considering vote history, gives us an incaccurate projection of the subset of that universe who will actually turn out to vote on Election Day.  Weighting isn't a good solution because it can only correct universe paremeters that are already represented and because we would still ultimately be weighting to counts determined by the initial sample universe or to whatever intuitively determined set of weights we imagine the likely voter universe to look like.   The bottom line is that because we don't have a verifiable, testable means of determining voter turnout, we don't have a good way to sample or weight.

Part of the problem is that pollsters often wind up constructing models that are based on intuitions and gut checks.  They carefully study a district, and then try and predict what the turnout is likely to be, and then, sample and call those people.  Unfortunately, this is rarely done in a quantitative manner. My solution is to try and quantify this method by use of modeling.

Using a well built list, you can look at vote history to predict likelihood of turnout based on previous elections for individuals, and then start doing the same for groups. So, for example, you can look at a voter file for a Congressional District in Texas, and see that white people who've voted in general elections prior to the current are 75% likely to vote in the current election, black people who've voted in the last  two general elections prior to the current are 62% likely to vote in the current, etc.   You can start doing the same by geographical districts, gender, age, etc., and really get into some good details. (Because you're still working with the entire voter file at large, and not a sample thereof, you still have enough records that the margin of error isn't likely to become a real problem.)

You can input all these factors into a model, as well as creating independent side rules, such as "regardless of race, voting for any three consecutive elections gives you a +5% chance to vote", to try and generate a turnout score. Obviously, you would have to evaluate things like the relationship between date of registration and first election voted in, as well as other factors, and put it up against race, gender, etc.  Vote history isn't enough for what I'm talking about here. You want to look at almost every variable that you can, at this stage.   You may also want to evaluate the recency of the data - for example, we know that in 2008, massive registration and subsequent turnout are distorting the turnout models.

Having built this gigantic model, we can now compute some vote likelihood score, VLS.  Having computed VLS, perpaps we can use either a large, large completely randomly sampled list with quotas set to universe counts or we can use a smaller list that's stratified and randomly sampled, using VLS to correct for over and under-representations in the sample.

It's important to note a few things, here.

1. What I have attempted to make quantifiable and external is the internal and qualitative sense that a pollster brings to approaching turnout questions.  The problem is that the current method culls the sample universe down, rather than correcting a sample of the entire universe after the fact.
2. I have deliberately left the modeling method open to discussion as I am not entirely sure what the best method would actually be.  My immedaite thought is that logistic regression would be your best bet, but I'm open to discussion.
3. This isn't intended as a silver bullet.

One of the problems that this does solve is that you no longer wind up with sub-samples that are way too small to have any confidence in the observation, and you also don't wind up weighting to either universe counts or weighting to some some sub-universe counts - you wind up weighting to a predicatable Vote Likelihood Score.  And now, because the VLS is external, quantified and available for everyone to see, you have an objective basis for criticising the weights of a poll.

I'll await my late night sleepy thought evisceration in the comments. Thoughts?

Dirty D