Those of you following along at home may notice that this post has introduced some new categories to the list, and that's because there's no really easy way to categorise this.  Basically, not too long ago, two of the entities listed on this site as Inspirations, RealClearPolitics and FiveThirtyEight, got into a major slapfight over different methodologies, transparency and whether or not one of them was committing major fraud in an effort to drive the media.

Here's the context: Nate Silver wanted to know why RCP wasn't including the Research 2000 polls commissioned by Daily Kos, but would include the polls by the Associated Press.  Silver argued that it was because the R2K polls were showing massively favorable Democratic results, while the AP was finding better results for the Republicans.  Since the editorial position of RCP is Republican, he argued that they had a vested interest in promoting better numbers for McCain, and that he had caught them doing it.

As much as I like Nate, though, I think that he's wrong, but he has managed to touch on one of the most fascinating things about the internet: the way that professionals and pitted against knowledgable impassioned people, and how this results in different models of information dissemination.

There's more...

There are two parts to this discussion:

1. Whether RCP was willingly engaging in methodological fraud to change a narrative
2. Why this debate even came up in the first place.

The first part is seemingly the crux of the matter, so let's get that out of the way first.  Yes, to anyone who reads RealClearPolitics, it's patently obvious that they have a center-right editorial position. I first started reading them in 2004, when they were mostly just doing content-aggregation, and you could tell that they were skewed center-right because of what they would select to feature.  Nonetheless, they were very,very good at finding content, and so I read them daily - merely being on  the other side of the partisan divide didn't mean that they were stupid, nor did it mean that they were incompetent.

Nate's problem with RCP actually reminds me a lot of the casual ease with which accusations of media bias are thrown around.  If the quality of political reporting were necessarily a function of one's editorial beliefs, then the contents of the Wall St. Journal's news reporting would probably resemble the crazed utterings of the mouth-breathers at K-Lo's House of Crazy.  Oh, wait, it doesn't.  The Wall St. Journal has consistently had some of the best news reporting out there, winning Pulitzer after Pulitzer, regardles of how crazy the knuckledraggers on their editorial page are.

Why do I bring this up? There's a really good reason that legitimate journalistic entities maintain a firm wall between news and editorial, and that's because people who engage in journalism are driven by a fundamental commitment to discovering and communicating the truth. I've met some journalists who believe that all sorts of erroneous things are the case, and some things that I believe to be morally repugnant, but all of them have been driven by a commitment to truth.  I have no reason to believe that the new media journalists at RCP are any different from any other real journalist, and I don't see why anyone would.

Furthermore, when you talk about anything relating to analytics, data or quantitative research, you're always going to have methodological arguments.  It's inevitable, and the simple fact that one guy who's made some pretty startlingly accurate predictions accuses another group who's done the same of methodological fault is unsurprising. Pollsters and political scientists are always arguing with each other about how they ask questions, model the data, etc.  Why should be surprised that two people engaging in a poll of polls would argue with each other on methods?

More importantly, on this discussion of methodology, is the comparison between the two.  RCP uses a method which involves choosing polls that are commissioned by major media entities, conducted by large firms with a strong history of polling, weighting the data according to traditional schemata and then taking an aggregate.  Silver's method involves choosing all sorts of polls from a certain time period and then applying step-wise regression to the results based on the populations of the states being polled.  Let's note that step-wise regression is called revolutionary by its advocates and data-dredging by its opponents.  In other words, when it comes to methodology, Silver's house is pretty vitreous.  This is not to say that I think that his conclusions are wrong.  That's hardly the case.  What I'm pointing is that Silver's methodology is more flummoxing to pollsters and political scientists than the RCP method. Also, the fact that Silver gets better measurements than RCP doesn't mean that they're lying. It just means that he has a better method.

Journalistic entities, new or old, live and die by their credibility.  I can't imagine a circumstance in which RCP would jeopardize their credibility to support a political candidate.  Think about what would be involved in this.  They would have to very cleverly feature content that skewers McCain/Palin, create their own content skewing McCain/Palin,all in order to conceal a very minor tweak to their polling average that would in theory affect the result by fewer than two points.  This would be in order to try and fool their readership, which consists of journalists, bloggers, activists and campaigns, all of whom are savvy and knowledgeable enough to know if they're being hosed.  Their credibility would go out the figurative window.

If that doesn't satisfy the paranoiac in you, then think about this. RCP is not Fox.  Their content does not consist of a bunch of screaming lunatics trying to whip up the base with propaganda.  Just as no slimy push poller hires a real pollster to conduct a push poll, you don't hire a liberal political scientist like Brendan Nyhan  to write about your polling if you're just going to much it up anyway. You hire Dick Morris. RCP is building their brand entirely on their credibility.

Now, if the paranoiac inside you still thinks that they would be willing to engage in such an elaborate subterfuge in order to provide an assist to the GOP candidate in a banner year for Democrats in which every little bit helps, imagine that the businessmen at RCP are being asked to to throw away all the credibility that they've built over the last few years for John McCain.  McCain. Really? The only Republicans who hate John McCain more than the great unwashed denizens of the Republican base's revival tent are the plutocrats against whom he rails when he's doing his Teddy Roosevelt impression.  While I could maybe believe that they'd have done for it a true believer, like Reagan or Bush 43, I just don't see them doing it for McCain.

In short, I think that Nate's method is better and more transparent, and I think that RCP's measurements are nowhere near as accurate as they should be, but hey, it's journalism. People are wrong all the time.  Being wrong doesn't make you a fraud - it just makes you wrong. RCP will be forever susceptible to this kind of attack till they make their methodology open, all of which leads us to...

The second bullet item that I brought up paragraphs and paragraphs ago: why this debate even came up in the first place.  I'm sure that HummuSoft can talk about this far more knowledgeably than I can, but something that I've noticed about the internet is the degree to which it's empowered passion-driven people like Nate (and us, I hope) to  take on the professionally appointed gatekeepers.  My observation on the way that this works is that the open source movement, when confronted with proprietary data or techniques, does one of two things. Either they:

1. Clone it, like GIMP does to PhotoShop or
2. they surpass it and come up with something entirely out of left field and way better, like the way Debian came up with apt-get or the way that Pidgin  is way better than AIM.

When it comes to politics, you can see evidence of the former all over Daily Kos and The Next Right, where all kinds of new voicies are popping up and reporting on things uncovered by the traditional media, using traditional polling analysis methods, etc.  You can see evidence of the latter in things like Silver's site.  As we all know, Nate's a statistician who's been working on baseball for a long time now, heck, I have several of his books, and he's managed to bring in new methods to bear on solving old problems.

It's no wonder that those of us trying to make better products would clash with the people who've been  out there and been doing it for a while. Implicit in our new methods is a rejection of what they were trying to do. Moreover, the fact that we're doing it because of our passions provides us with a completely different set of motivations than those people who are doing it for the money. For us, transparency and sharing of methods provides a demonstration of the value of our work - with enough eyes looking at us, we're l33t. We come from an environment that fosters the sharing of ideas in order to come up with a group-authored ultimate product.

The other side, however, comes from the position that they're trying to make a buck by doing what no one else can. They're trying to provide a unique product on the market that people will want so badly that they'll just have to have it, and so will buy it from them.  Have you ever tried asking a pollster about how she built the sample universe? From where they got the sample? Why something was clustered instead of replicated?  Hell, Mark Penn wouldn't even show the Clinton campaign a filled in questionnaire.  This is not to let RCP off the hook for being proprietary about their methodology - I'm a huge open source advocate - but it's typical of the closed source dinosaur model of informaton.  For that matter, Nate doesn't actually show us his model, either.  He just tells us that he has one, and how he built it.

This all leads me to something that I hope separates us from the rest of the people out there.  We're doing this site for the love, yes, but also because we want to use our collective years of learning and expertise to make things easier for the smaller campaigns and non-profits who can't afford to hire a pollster or a data consultant.  If you can find a whole bunch of resources that show you how to do things on your own, even if you hire a pollster or a data consultant, you're still more empowered in your dealings with her.  We don't write our ongoing series just to flaunt  our knowledge, and  we're certainly not using this blog to market our services.  We're trying to give you the tools and knowledge you need to make your group more data-driven. Any tools we put up here are the tools that we ourselves use.

Should RCP disclose their methodologies? Of course. It not only makes the debate better, but increases confidence in their methods. Should Nate do the same? Of course. Should everyone? Yes. We believe in free and open debate with fully free flow of information.  Does it make anyone who doesn't a liar? Absolutely not.

Dirty D