Nate Cohn

Nate Cohn @Nate_Cohn

The catch, of course, is that although their polls would have been *worse*--and shown, say, Trump+5 in FL or whatever v. Clinton+1--the average would have been better. — PolitiTweet.org

Nate Cohn @Nate_Cohn

We can see this with Marist, for example, who notes that they did just in *their* final polls in 2016 without education, or others on this website who have said some version of the same argument. In fact, there are folks whose polls were *worse* with education weighting! — PolitiTweet.org

Nate Cohn @Nate_Cohn

As a result, I don't really think that this generally reasonable point by Nate here holds up exactly. The incentive structure for individual pollsters is different than for the polling aggregator. Education weighting threatens the aggregator far more. https://t.co/NVJkBCMrH7 — PolitiTweet.org

Nate Cohn @Nate_Cohn

The challenge for education is that the effect on *error* is potentially quite small (oh I was only off by 3 points, in the MoE, I'm fine!), even as the effect on bias is fairly large (oh i took 10 polls and they all leaned Democratic, even if off by 3 on average...) — PolitiTweet.org

Nate Cohn @Nate_Cohn

These choices sometimes put pollsters at odds with poll aggregators. A poll aggregator prefers an unbiased poll, even if it's noisy. A pollster wants to reduce error, and might make choices that induce bias to poll it off--weighting by party identification, to take an easy one — PolitiTweet.org

Nate Cohn @Nate_Cohn

These choices sometimes put pollsters at odds with poll aggregators. A poll aggregator prefers an unbiased poll, even if it's noisy. A pollster wants to reduce error, and might make choices that induce error to poll it off-- weighting by party identification, to take an easy one — PolitiTweet.org

Nate Cohn @Nate_Cohn

That's definitely not the way things work now, if it ever was. Whether due to low response, noncoverage or opt-in sampling, pollsters don't have unbiased samples. Instead, they do a bunch of increasingly heavy handed statistical adjustments to compensate. — PolitiTweet.org

Nate Cohn @Nate_Cohn

In this view, polls would only need to be weighted by sample size with a control for house effects, provided that the polls met the theoretical conditions for unbiased sampling — PolitiTweet.org

Nate Cohn @Nate_Cohn

In theory, a bigger poll/polling average would shrink error until it reached the practical limits of nonsampling error, like undecided voters / turnout / interviewer effect, etc. — PolitiTweet.org

Nate Cohn @Nate_Cohn

Back in the early days of polling averages, you could argue that polling averages worked because the polls were fundamentally unbiased samples, controlling for house effects. Aggregating polls was like getting one really big and good poll, with a steadily shrinking MoE. — PolitiTweet.org

Nate Cohn @Nate_Cohn

Leaving aside this specific example aside, there is an interesting conversation to be had here about why polling averages work nowadays, and what that means for how you want to think about constructing a polling average. — PolitiTweet.org

Nate Cohn @Nate_Cohn

Ultimately, the range of options under-consideration by pollsters today aren't centered on the population mean--no education weights means a long tail toward a Silicon Valley electorate. That's why weighting is necessary and why it would risk *bias* in the average. — PolitiTweet.org

Nate Cohn @Nate_Cohn

I agree with the default assumption that diversity is good for averages, but this is taking it too far for me. Do we think averages would be better if half of pollsters stopped weighting on race and age, just to get some extra methodological diversity? https://t.co/RgDguQOWKu — PolitiTweet.org

Nate Cohn @Nate_Cohn

@ElectProject @jon_m_rob i don't think that's at all obvious tbh, though it is possible — PolitiTweet.org

Nate Cohn @Nate_Cohn

The effect of partisanship, controlling for everything else you're weighting on, is still fairly consequential of course. But it is less than just moving everything 4 points. — PolitiTweet.org

Nate Cohn @Nate_Cohn

Well, depends on the other weights, but if you just give more weight to the GOP in the sample, you just made your sample whiter than it should be, or maybe now you have too few young voters, or too few people outside the South, and so on. https://t.co/iTkhSGmFUn — PolitiTweet.org

Nate Cohn @Nate_Cohn

As an example, many noticed our AZ poll had like Trump>10 on '16 recalled vote. But you can't just unskew it 5 points to the left. Our poll, as it was, had the right number of reg Ds/Rs; unskewing on the '16 vote tab would have meant giving Dems a reg advantage in AZ — PolitiTweet.org

Nate Cohn @Nate_Cohn

One minor thing, for the crosstab divers: you can't just reweight a poll by changing the proportions in the crosstabs, since that can throw off other variables. You've got to weight on everything at once. — PolitiTweet.org

Nate Cohn @Nate_Cohn

@jbarro 1.56 — PolitiTweet.org

Nate Cohn @Nate_Cohn

This meant 51-42, sorry. — PolitiTweet.org

Nate Cohn @Nate_Cohn

There are any number of other things one could conceivably weight for--density, marriage, and so on. I'm not going to go through all that. But I think this gives you a general sense of what's going on here, and the consequences of some of these choices. — PolitiTweet.org

Nate Cohn @Nate_Cohn

This won't usually have a huge effect, but here it's worth a point. If we take our previous Biden+8 (assuming two-year = "some college", but using our weighting targets) and now weight by race x edu as well, we now move to Biden+9, 51 to 43. — PolitiTweet.org

Nate Cohn @Nate_Cohn

Thus, our last adjustment: race by education. As you can see, having the right number of college grads is good, but having the right number of whites with a degree v. nonwhites with a degree is better. — PolitiTweet.org

Nate Cohn @Nate_Cohn

This basic pattern helps explain some common patterns we've seen in state polling, where the polls that aren't weighted by education still seem to kinda fair ok or even underestimate Dems in the diverse southwestern states, even while understating Dems badly in the Midwest — PolitiTweet.org

Nate Cohn @Nate_Cohn

In our version (Biden+8, assuming two-year = somecol): White: Trump 51, Biden 44 Black: Biden 88, Tump 9 Hispanic: Biden 64, Trump 21 So you can see the often overlooked effect of not weighting by education on your nonwhite sample, too — PolitiTweet.org

Nate Cohn @Nate_Cohn

One interesting thing here is the effect all this has on the crosstabs, too. In *their* version, here's the result by race: White: Biden 49, Trump 46 Black: Biden 82, Trump 13 Hispanic: Biden 59, Trump 25 — PolitiTweet.org

Nate Cohn @Nate_Cohn

If we take our weights across all five of these categories: With two-year as 'college grad': Biden+10, 52-42 With two-year as 'some college': Biden+8, 51-43 — PolitiTweet.org

Nate Cohn @Nate_Cohn

On closer examination though, one thing that's clear is that this poll is pretty different from our estimates across a number of different variables. Non-Hispanic whites represent 76% of RVs who offer a race/ethnicity! Both seniors (20%) and 18-29 year olds (9%) too low — PolitiTweet.org

Nate Cohn @Nate_Cohn

But we'll try it both ways. If we use *their* weighted figures for race, age, region, gender and *our* targets for education, then here's what you get: With two-year as 'college grad': Biden+9, 52-43 With two-year as 'some college': Biden+6, 50-44 — PolitiTweet.org

Nate Cohn @Nate_Cohn

And many of you have noted in the replies that the parenthetical note on years of education may help clarify to the interviewer that the two-year degrees belong in the some college category. — PolitiTweet.org